Auto-Instrument Your Applications using OpenTelemetry Operator for Kubernetes

This page describes how to auto-instrument your applications using the OpenTelemetry Operator for Kubernetes, which is installed by the Kubernetes and App Service Monitoring Helm charts.

Before You Begin

The steps described on this page assume that:

• Your applications are running in a supported cluster. See Software Requirements in Install Kubernetes and App Service Monitoring.
• You have installed Kubernetes and App Service Monitoring using Helm charts. See Install Kubernetes and App Service Monitoring.
• Your applications are not instrumented with OpenTelemetry™, using an agent or SDK.
• Your applications are implemented using Java, Node.js, Python, and .NET, which are languages supported by OpenTelemetry Operator for Kubernetes auto-instrumentation. See https://opentelemetry.io/docs/instrumentation.

The Helm chart for Kubernetes and App Service Monitoring installs the OpenTelemetry Operator for Kubernetes, which supports auto-instrumentation. This feature makes it simple to add the language-specific OpenTelemetry agent to a Kubernetes® deployment without requiring code changes or changes to the Kubernetes deployment spec. Auto-instrumentation can be applied to one or more namespaces and any of the deployments within a namespace that have the required annotation. 

The OpenTelemetry code examples are sourced from https://github.com/open-telemetry/opentelemetry-operator/blob/main/README.md.

This document contains references to OpenTelemetry™ and Amazon Web Services (AWS) documentation. Splunk AppDynamics does not own any rights and assumes no responsibility for the accuracy or completeness of such third-party documentation.

Auto-Instrument Your Applications

This section describes the methods that you can use to auto-instrument your applications. If you want more control over your configuration, see Advanced Configuration.

Basic Configuration

1. (Optional) If you need to collect logs using the OpenTelemetry Java SDK, ensure you Send Logs from OpenTelemetry Java SDK to Cisco AppDynamics Distribution of OpenTelemetry Collector.

2. Query the Kubernetes Custom Resource and check that appd-instrumentation exists in the appdynamics namespace:

   kubectl get otelinst -n appdynamics 

   BASH
3. Add an annotation to a pod to enable injection. The annotation can be added to:

• A namespace so that all pods within that namespace get instrumentation, or

• Individual PodSpec objects available as part of Deployment, Statefulset, and other resources.

  Annotation

  instrumentation.opentelemetry.io/inject-java: appdynamics/appd-instrumentation

  BASH

  Example

    template:
      metadata:
        annotations:
          instrumentation.opentelemetry.io/inject-java: appdynamics/appd-instrumentation 

  JAVA

  Annotation

  instrumentation.opentelemetry.io/inject-python: appdynamics/appd-instrumentation

  PY

  Example

    template:
      metadata:
        annotations:
          instrumentation.opentelemetry.io/inject-python: appdynamics/appd-instrumentation 

  PY

  Annotation

  instrumentation.opentelemetry.io/inject-nodejs: appdynamics/appd-instrumentation

  BASH

  Example

    template:
      metadata:
        annotations:
          instrumentation.opentelemetry.io/inject-nodejs: appdynamics/appd-instrumentation 

  BASH

  Annotation

  instrumentation.opentelemetry.io/inject-dotnet: appdynamics/appd-instrumentation

  BASH

  Example

    template:
      metadata:
        annotations:
          instrumentation.opentelemetry.io/inject-dotnet: appdynamics/appd-instrumentation 

  BASH
  If you prefer not to inject, set the annotation to "false".

Advanced Configuration

This method is intended for users who want to create advanced configurations for their auto-instrumentation.

1. (Optional) If you need to collect logs using the OpenTelemetry Java SDK, ensure you Send Logs from OpenTelemetry Java SDK to Cisco AppDynamics Distribution of OpenTelemetry Collector.

2. Configure a Kubernetes Custom Resource called Instrumentation. This Custom Resource is installed by the OpenTelemetry Operator For Kubernetes and can be installed in any namespace. For the configuration settings, see Configure APM Auto-Instrumentation. The value OTEL_EXPORTER_OTLP_ENDPOINT under the Python section is different from Java and Node.js because Python auto-instrumentation uses HTTP to communicate to the otel-collector not gRPC.

   apiVersion: opentelemetry.io/v1alpha1
   kind: Instrumentation
   metadata:
     name: auto-instrumentation
   spec:
     exporter:
       endpoint: http://appdynamics-otel-collector-service.appdynamics.svc.cluster.local:4317
     propagators:
       - tracecontext
       - baggage
       - b3
     env:
       - name: OTEL_EXPORTER_OTLP_INSECURE
         value: "true"
       - name: OTEL_LOG_LEVEL
         value: "debug"
       - name: OTEL_TRACES_EXPORTER
         value: "otlp,logging"
     java:
       image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:latest
       env:
         - name: OTEL_EXPORTER_OTLP_ENDPOINT
           value: "http://appdynamics-otel-collector-service.appdynamics.svc.cluster.local:4317"
         - name: OTEL_JAVAAGENT_DEBUG
           value: "true"
     nodejs:
       image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-nodejs:latest
       env:
         - name: OTEL_EXPORTER_OTLP_ENDPOINT
           value: "http://appdynamics-otel-collector-service.appdynamics.svc.cluster.local:4317"
     python:
       image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-python:0.28b1
       env:
         - name: OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
           value: "http://appdynamics-otel-collector-service.appdynamics.svc.cluster.local:4318/v1/traces"
         - name: OTEL_TRACES_EXPORTER
           value: otlp_proto_http
         - name: OTEL_PYTHON_LOG_LEVEL
           value: "debug"   
     dotnet:
       env:
         # Required if endpoint is set to 4317.
         # Dotnet autoinstrumentation uses http/proto by default
         # See https://github.com/open-telemetry/opentelemetry-dotnet-instrumentation/blob/888e2cd216c77d12e56b54ee91dafbc4e7452a52/docs/config.md#otlp
         - name: OTEL_EXPORTER_OTLP_ENDPOINT
           value: http://appdynamics-otel-collector-service.appdynamics.svc.cluster.local:4318

   YML

3. Run the following command:

   kubectl apply -f - <<EOF
   <sample-auto-instrumentation-config>
   EOF

   CODE

   Alternatively, you can save the sample config to a YAML file and run the following command:

   kubectl apply -f <path-to-yaml-file>

   CODE

4. Query the Kubernetes Custom Resource:

   kubectl get otelinst

   YML
5. Add an annotation to a pod to enable injection. The annotation can be added to:
   
   • A namespace so that all pods within that namespace get instrumentation, or
   • Individual PodSpec objects available as part of Deployment, Statefulset, and other resources.
   instrumentation.opentelemetry.io/inject-java: "true"

   BASH

   instrumentation.opentelemetry.io/inject-python: "true"

   BASH

   instrumentation.opentelemetry.io/inject-nodejs: "true"

   BASH

   instrumentation.opentelemetry.io/inject-dotnet: "true"

   BASH

   Available values for the annotation are:

   • "true" - To inject the "Instrumentation" resource with the default name from the current namespace.
   • "my-instrumentation" - To inject the "Instrumentation" CR instance with the name "my-instrumentation" in the current namespace.
   • "my-other-namespace/my-instrumentation" - To inject the "Instrumentation" CR instance with the name "my-instrumentation" from another namespace "my-other-namespace".
   • "false" - Do not inject.

Multi-container Pods

If nothing else is specified, the instrumentation is performed on the first container available in the pod spec. In some cases, for example in the case of the injection of an Istio sidecar, you must specify the container(s) that are to receive this injection.

To fine-tune the pod(s) on which the injection will be carried out, use the instrumentation.opentelemetry.io/container-names annotation. Indicate one or more pod names (.spec.containers.name) to inject. In the example, myapp and myapp2 containers are instrumented, myapp3 is not.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-deployment-with-multiple-containers
spec:
  selector:
    matchLabels:
      app: my-pod-with-multiple-containers
  replicas: 1
  template:
    metadata:
      labels:
        app: my-pod-with-multiple-containers
      annotations:                  
        instrumentation.opentelemetry.io/inject-java: appdynamics/appd-instrumentation         
        instrumentation.opentelemetry.io/container-names: "myapp,myapp2"
    spec:
      containers:
      - name: myapp
        image: myImage1
      - name: myapp2
        image: myImage2
      - name: myapp3
        image: myImage3

Send Logs from OpenTelemetry Java SDK to Cisco AppDynamics Distribution of OpenTelemetry Collector 

To collect the logs using the the OpenTelemetry SDK, you are required to:

1. Configure the OpenTelemetry Log Appender for your Logger. 
2. Enable log exporter for OpenTelemetry Java SDK or Agent.

Configure OpenTelemetry Log Appender for Your Logger

You can configure the log appenders based on the logging library that is used in your application.  

• Log4j
• Logback

• SLF4J

Collecting the logs using the OpenTelemetry SDK is currently supported only for Java applications.

Enable Log Exporter for OpenTelemetry Java SDK or Agent

The log exporter sends your Java application logs to the Cisco AppDynamics Distribution of OpenTelemetry Collector. To enable the log exporter, add the env variable, OTEL_LOGS_EXPORTER with the value "otlp,logging" in collectors-values.yaml file. For information about the YAML file, see Auto-Instrument Your Applications. 

appdynamics-otel-instrumentation:
  spec:
    java:
      image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:latest
       env:
         - name: OTEL_EXPORTER_OTLP_ENDPOINT
            value: "http://appdynamics-otel-collector-service.appdynamics.svc.cluster.local:4317"
         - name: OTEL_JAVAAGENT_DEBUG
           value: "true"
         - name: OTEL_LOGS_EXPORTER
            value: "otlp,logging" 

Continue with step 2 to Auto-Instrument Your Applications.

The histogram may not display the details when the logs are collected through the OpenTelemetry Java SDK. However, you can click View all logs to monitor the logs.

OpenTelemetry™ and Kubernetes® (as applicable) are trademarks of The Linux Foundation®.

Amazon Web Services, the AWS logo, AWS, and any other AWS Marks used in these materials are trademarks of Amazon.com, Inc. or its affiliates in the United States and/or other countries.