Infrastructure Correlation

Kubernetes and App Service Monitoring provides the ability to correlate OpenTelemetry™ traces to Kubernetes® and cloud infrastructure metrics, events, and logs. The Service UI below shows an example where infrastructure correlation is able to correlate a specific application Service and Service Instance with the associated Kubernetes Cluster, Namespace, Workload, and Pod.

Cisco Cloud Observability automatically establishes correlation which removes the need to manually filter the MELT (Metrics, Events, Logs, and Trace) data that span OpenTelemetry, Kubernetes, and Cloud data sources.

When infrastructure correlation is not enabled, the Service and Service Instance entities will not display the associated Kubernetes and cloud infrastructure in the RELATIONSHIPS view, and the Service Instance will display Unknown ID. See Enable Infrastructure Correlation for Non-Java OpenTelemetry Instrumented Services.

Infrastructure Correlation Requirements

In order for Cisco Cloud Observability to establish infrastructure correlation, the OpenTelemetry traces sent to Cisco Cloud Observability must include container.id as a resource attribute.
The instrumented service sending trace data to Cisco Cloud Observability must be running in a supported Kubernetes cluster where Kubernetes and App Service Monitoring is installed. See Software Requirements on Install Kubernetes and App Service Monitoring.

Steps to Enable Infrastructure Correlation

Not every language-specific OpenTelemetry SDK provides the container.id required to enable correlation, so you may need to take additional steps to include it in an OpenTelemetry trace. See the language-specific OpenTelemetry docs.

Infrastructure correlation is automatically supported by Java services that use automatic instrumentation, which includes the container.id by default. See https://opentelemetry.io/docs/instrumentation/java/.

For non-Java services or Java services using manual instrumentation, it's possible to add container.id via an environment variable, which avoids the need to add SDK calls to a service. See Enable Infrastructure Correlation for Non-Java OpenTelemetry Instrumented Services.

Kubernetes Attribute Processor Configured on the OpenTelemetry Collector

Kubernetes Pod Container ID

The following instructions are used for collecting container ID attributes for containers in K8s clusters using Kubernetes API. It utilizes a Kubernetes Attribute Processor configured on OpenTelemetry Collector.

Kubernetes RBAC

The Kubernetes Attribute Processor requires special permissions to access Kubernetes API. Required permissions can be provided creating ClusterRole, ClusterRoleBinding, and ServiceAccount. For more information, see the Kubernetes Attribute Processor documentation. More information on proper RBAC permissions and proper permissions example in the following:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: collector
  namespace: <OTEL_COL_NAMESPACE>
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: otel-collector-role
rules:
  - apiGroups: [""]
    resources: ["pods", "namespaces"]
    verbs: ["list", "watch", "get"]
  - apiGroups: ["apps"]
    resources: ["replicasets"]
    verbs: ["list", "watch", "get"]
  - apiGroups: ["extensions"]
    resources: ["replicasets"]
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: otel-collector-binding
subjects:
  - kind: ServiceAccount
    name: collector
    namespace: <OTEL_COL_NAMESPACE>
roleRef:
  kind: ClusterRole
  name: otel-collector-role
  apiGroup: rbac.authorization.k8s.io

YML

OpenTelemetry Collector Configuration

The deployment of the OpenTelemetry Collector through a Kubernetes Pod will be split into a service, deployment, and ConfigMap that provides otel-collector-config. The service and deployment will be used to manage your pod, while the config sets up the service pipeline for the collector. In this example the otel/opentelemetry-collector-contrib image. is used as it supports the Kubernetes Attribute Processor, and gRPC to connect your application and the collector. Collector config should include k8sattributes processor configuration with extraction of container.id metadata. A configuration of the service, deployment, and configure within a singular yaml can be seen in the following example:

apiVersion: v1
kind: Service
metadata:
  name: otel-collector
spec:
  ports:
  - port: 4317
    targetPort: 4317
  selector:
    app: otel-collector
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: otel-collector
spec:
  replicas: 1
  selector:
    matchLabels:
      app: otel-collector
  template:
    metadata:
      labels:
        app: otel-collector
    spec:
      containers:
        - args:
            - --config=/etc/otel-collector-config.yml
          env:
          image: otel/opentelemetry-collector-contrib:0.105.0
          name: otel-collector
          ports:
            - containerPort: 4317
              protocol: TCP
          volumeMounts:
            - mountPath: /etc/otel-collector-config.yml
              name: otel-collector-config
              subPath: otel-collector-config.yml
      volumes:
        - configMap:
            items:
              - key: otel-collector-config.yml
                path: otel-collector-config.yml
            name: otel-collector-config
          name: otel-collector-config
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: otel-collector-config
data:
  otel-collector-config.yml: |
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317

    processors:
      k8sattributes:
        extract:
          metadata:
            - container.id

    exporters:
      debug:
        verbosity: detailed
      logging:
        verbosity: detailed

    service:
      pipelines:
        traces:
          receivers: [otlp]
          processors: [k8sattributes]
          exporters: [debug, logging]

YML

Application Configuration OpenTelemetry Collector Specifics

The Application should send to the k8s.container.name attribute. It can be done by setting OTEL_RESOURCE_ATTRIBUTES to k8s.container.name=<container_name>. For example, if your application container name was "aspnetcore-sample-container," you would set OTEL_RESOURCE_ATTRIBUTES=k8s.container.name=aspnetcore-sample-container. That environment variable may contain more value pairs, using a comma as separator (<key1>=<value1>,<key2>=<value2>). OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_EXPORTER_OTLP_PROTOCOL environment variables can be used in order to configure your application to send data to the collector. The collector from the previous step configured to accept gRPC data at port 4317. For example, if in the service.yaml file of your collector you name the service "otel-collector" within name within metadata, the correct address would be http://otel-collector:4317, or http://"your-service-name-in-metadata":4317. The following example displays how to correctly configure the environment variables in your app deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: aspnetcore-sample-deployment
  labels:
    app: aspnetcore-sample
spec:
  replicas: 1
  selector:
    matchLabels:
      app: aspnetcore-sample
  template:
    metadata:
      labels:
        app: aspnetcore-sample
    spec:
      containers:
        - env:
            - name: OTEL_EXPORTER_OTLP_ENDPOINT
              value: http://otel-collector:4317
            - name: OTEL_RESOURCE_ATTRIBUTES
              value: k8s.container.name=aspnetcore-sample-container
            - name: OTEL_EXPORTER_OTLP_PROTOCOL
              value: grpc
          name: aspnetcore-sample-container
          image: mcr.microsoft.com/dotnet/samples:aspnetapp
          ports:
            - containerPort: 8080
              protocol: TCP

YML

Use Language-specific SDKs to Enable Infrastructure Correlation

The following details provide guidance for adding the container.id resource attribute for Java (manual instrumentation), .NET, Node.js, and Go.

OpenTelemetry for Java

When you auto-instrument the Java Agent, the Container resource is enabled by default, and the hostname is captured, on any SDK that uses auto-configure.

label	value
container.id	<container id>

AWS EC2

Not bundled with the Java Agent by default. See Ec2Resource.java.

label	value
`cloud.platform`	AWS
`host.id`	<host id>
`cloud.availability_zone`	<availability zone>
`cloud.region`	<region>
`cloud.account.id`	<account id>

AWS EKS

Not bundled with the Java Agent by default. Requires configuring an EKS token. See EksResourceProvider.java.

label	value
`cloud.platform`	AWS
`container.id`	<container id>
`cluster.name`	<cluster name>

For AWS Elastic Beanstalk, AWS Lambda, and Amazon ECS support, check each class in OpenTelemetry AWS Resource Support.

OpenTelemetry for Node.js

OpenTelemetry detectors are available to capture container metadata based on the container type. See npm package resource-detector-aws. These can be added to the tracing code:

ECS

import { detectResources } from '@opentelemetry/resources';
import { awsEcsDetector } from '@opentelemetry/resource-detector-aws'
const resource = await detectResources({
   detectors: [awsEcsDetector],
})

const tracerProvider = new NodeTracerProvider({ resource });

JS

EKS

import { detectResources } from '@opentelemetry/resources';
import { awsEksDetector } from '@opentelemetry/resource-detector-aws'
const resource = await detectResources({
   detectors: [awsEksDetector],
})

const tracerProvider = new NodeTracerProvider({ resource });

JS

Additional Detectors are available awsBeanstalkDetector, awsEc2Detector, awsLambdaDetector

OpenTelemetry for Go

The Go OpenTelemetry SDK provides detectors for collecting container.id information for containers in ECS and EKS environments. These detectors can be enabled in code by placing these lines in the application during SDK initialization.

ECS

// Instantiate a new ECS Resource detector
ecsResourceDetector := ecs.NewResourceDetector()
resource, err := ecsResourceDetector.Detect(context.Background())

// container.name container.id are collected and annotated to resource metadata

YML

EKS

// Instantiate a new EKS Resource detector
eksResourceDetector := eks.NewResourceDetector()
resource, err := eksResourceDetector.Detect(context.Background())

// k8s.cluster.name container.id are collected and annotated to resource metadata

YML

OpenTelemetry for .NET

OpenTelemetry detectors are available to capture the container.id based on the environment type. It is not bundled with opentelemetry-dotnet sdk by default. These detectors can be enabled in code by adding the required package and below lines in the sample application.

Amazon ECS
An additional package needs to be added to extract the resources from an ECS environment. See opentelemetry-dotnet-contrib project.

using OpenTelemetry;
using OpenTelemetry.Contrib.Extensions.AWSXRay.Resources;
using OpenTelemetry.Contrib.Extensions.AWSXRay.Trace;
 
Sdk.CreateTracerProviderBuilder()
    .SetResourceBuilder(ResourceBuilder.CreateDefault().AddService("aws-otel-integ-test").AddTelemetrySdk().AddDetector(new AWSECSResourceDetector()))
    .AddXRayTraceId()
    ...
    .Build();
 
 Sdk.SetDefaultTextMapPropagator(new AWSXRayPropagator());

CODE

Amazon EKS
An additional package needs to be added to extract the resources from an EKS environment. See opentelemetry-dotnet-contrib project.

using OpenTelemetry;
using OpenTelemetry.Contrib.Extensions.AWSXRay.Resources;
using OpenTelemetry.Contrib.Extensions.AWSXRay.Trace;
 
Sdk.CreateTracerProviderBuilder()
    .SetResourceBuilder(ResourceBuilder.CreateDefault().AddService("aws-otel-integ-test").AddTelemetrySdk().AddDetector(new AWSEKSResourceDetector()))
    .AddXRayTraceId()
    ...
    .Build();
 
 Sdk.SetDefaultTextMapPropagator(new AWSXRayPropagator());

BASH

AKS
An additional package needs to be added to extract the resources from a Docker environment. See opentelemetry-dotnet-contrib project.

using OpenTelemetry;
using OpenTelemetry.Extensions.Docker.Resources;
var tracerProvider = Sdk.CreateTracerProviderBuilder()
                        // other configurations
                        .SetResourceBuilder(ResourceBuilder
                            .CreateEmpty()
                            .AddDetector(new DockerResourceDetector()))
                        .Build();

BASH

All these resources can be extracted only if the containers are running in a Linux environment, not in a Windows environment.