Overview

Some issues that you may observe after upgrading your Kubernetes® environment to version >= 1.25 are caused by the OpenTelemetry™ Java Agent failing to properly discover your container ID from within the container you are running in. The container ID you discover has the correct form and syntax, but is different from the true container ID as used by Kubernetes. This breaks correlation between Application Performance Monitoring (APM) data and infrastructure.

The solution to extraction of correct container ID in Kubernetes version >= 1.25 is to query the K8 API Server for the container ID.

This document contains references to third-party documentation. Splunk AppDynamics does not own any rights and assumes no responsibility for the accuracy or completeness of such third-party documentation.

Before You Begin

The CGroupV2 container ID extraction is provided as an OpenTelemetry Java Agent extension and is available for download from the Splunk AppDynamics portal. To enable this, complete the following:

Before installing, ensure that you meet the following requirements:

  • You are using OpenTelemetry Java Agent version >=1.19  
  • You have downloaded the Splunk AppDynamics Extension for OSS OpenTelemetry Java Agent extension zip from the AppDynamics Download Portal under AppDynamics OpenTelemetry Extensions or from the public JFrog artifactory (group: com.appdynamics.opentelemetry.extension, name: appd_oss_java_agent_extension). Minimum supported version for the CGroupV2 container ID extraction is = 24.4.

If your application is deployed in a container environment, you can download the AppDynamics Extension for OSS OpenTelemetry Java Agent docker image from Docker Hub into a shared volume utilizing an init container.

Using the Extension

The extension cannot be used on its own and must always be used with a supported version of the OSS OpenTelemetry Java Agent. The agent can be downloaded from the OTel Java Agent releases.

1. Copy the appd_oss_java_agent_extension.jar file to the host that is running the application which you are monitoring with the OpenTelemetry Java Agent.
2. Modify the application startup command to add the full path to the extension jarfile. For example:

 java -javaagent:path/to/opentelemetry-javaagent.jar \
          -Dotel.javaagent.extensions=path/to/appd_oss_java_agent_extension.jar \
          -jar myapp.jar
CODE

Enabling Extraction of Container ID for CGroupV2 Based Containers

In addition to loading the above extension, to get the container ID from Kubernetes, the following configuration is required:

  • Application Pod should be associated to a service account (with at least get permission to kubernetes resource pods)
  • Provide target container name as environment variable (APPDYNAMICS_CONTAINER_NAME) or using the following system property:
    -Dappdynamics.container.name=<target-container-name>
    CODE


The extension sets the resource attribute, container.id, after it detects a valid container ID. If it is unable to fetch the container ID, it sets an all-zeros container ID and adds a warning/error log for same. Check the container logs or stderr in such a case and address the issue.


Sample Service Account, Role Binding and Role resource yamls:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: pods-reader
  namespace: javaspace
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get"]
-----------------------------------------------------------------------
  
apiVersion: v1
kind: ServiceAccount
metadata:
  name: view-serviceaccount
  namespace: javaspace
  
-----------------------------------------------
  
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: read-pods-namespace
  namespace: javaspace
subjects:
- kind: ServiceAccount
  name: view-serviceaccount
  namespace: javaspace
roleRef:
  kind: Role
  name: pods-reader
  apiGroup: rbac.authorization.k8s.io
CODE

OpenTelemetry™ and Kubernetes® (as applicable) are trademarks of The Linux Foundation®.