This page is intended for users who have already deployed open-source OpenTelemetry™ Collectors and want to configure them to send OpenTelemetry protocol (OTLP) trace and log data to the Cisco Cloud Observability backend. If you have not deployed any OpenTelemetry Collectors, follow one of the workflows on Monitor Applications in Kubernetes.

Open-source OpenTelemetry Collectors can only be used to send OTLP trace and log data. To send Cisco Cloud Observability Collectors data to Cisco Cloud Observability and enable infrastructure correlation, you need to Install Kubernetes and App Service Monitoring.

Although open-source OpenTelemetry Collectors can send any MELT data, only metrics collected from Cisco Cloud Observability Collectors can be sent to the backend. 

OpenTelemetry Collectors act as the service pipeline by gathering telemetry data from your applications and exporting it to a backend. OpenTelemetry Collectors can be configured to collect and send distributed traces to the Cisco Cloud Observability backend.

The Collector consists of the following components:

  • Receivers: Receive data from OpenTelemetry sources (tracers and SDKs)
  • Processors: Allow additional processing of data, such as data enrichment and batching
  • Exporters: Convert the data into a specified format and sends the converted data to a chosen destination
  • Extensions: Enable tasks that do not involve processing telemetry data

See the following sections to configure these components in your OpenTelemetry Collector. Once configured, the components must be enabled via a service pipeline. Splunk AppDynamics recommends that you edit your open-source OpenTelemetry configuration file to add the components for Cisco Cloud Observability. 

This document contains references to OpenTelemetry™ documentation. Cisco Cloud Observability does not own any rights and assumes no responsibility for the accuracy or completeness of such third-party documentation.

Before You Begin

  1. Deploy the OpenTelemetry Collector version >= 0.36.0 in your computing environment.
  2. Instrument your applications.
    1. For applications running in AWS Lambda functions, see Monitor AWS Lambda Using OpenTelemetry Instrumentation.
    2. For applications running in other AWS services, see the OpenTelemetry documentation to instrument your applications.

Configure Receivers

Receivers specify how data gets into the OpenTelemetry Collector. Receivers use OTLP to support transaction-oriented application data processing.

The OTLP receiver is bundled with the open-source OpenTelemetry Collector by default. This setup requires OTLP receiver version >= 0.33.0, which uses OTLP version >= 0.9.0. For a list of verified OTLP open-source versions, see Verified Open Source Versions.

  1. In the receivers: OTLP: section, specify the Collector endpoints:

    receivers:  
  otlp:    
    protocols:      
      grpc:        
        endpoint:      
      http:        
        endpoint:
    YML


    By default, OTLP uses the following endpoints:

    • 4317 for GPRC
    • 4318 for HTTP
  2. (Optional) Configure the Filelog Receiver within your OpenTelemetry Collector to send logs to  Cisco Cloud Observability. You do this configuration in the receivers:filelog and service:pipelines sections of your collector's override configuration file. See Configure the Filelog Receiver.

Configure Processors

Processors allow additional processing of data, such as data enrichment and batching.

Kubernetes® Attributes Processor

To add the Kubernetes attribute and enable Kubernetes cluster correlation, you must add the following attribute in the processors section:

Kubernetes Attribute Processor

processors:     
  k8sattributes: 
      passthrough: false
YML

Batch Processor

The batch processor accepts spans, metrics, or logs and places them into batches in the OpenTelemetry Collector pipeline.

Configure the following batch: attributes to help improve performance:

  • timeout: Time after which a batch is sent, regardless of size.

  • send_batch_size: Number of spans or metrics after which a batch is sent.

Batch Processor

processors:
  batch:
      send_batch_size: 1000
      timeout: 10s
      send_batch_max_size: 1000
YML

Configure Exporters

The exporter is how data is sent to the Cisco Cloud Observability backend. Only HTTP is supported to transmit OTLP trace export calls. Your tracer/SDKs exporter should be based on OTLP version >= 0.9.0.

In the exporters: otlphttp: section, you must configure the following attributes with the Cisco Cloud Observability endpoint and the OAuth2 Client. The <cisco-cloud-observability-endpoint> is generated from the Cisco Cloud Observability UI during step 4 of Install Kubernetes and App Service Monitoring Using Helm Charts. The value will be https://<cisco-cloud-observability-url>/data.

exporters:   
  otlphttp:     
    metrics_endpoint: <cisco-cloud-observability-endpoint>/v1/metrics
    traces_endpoint: <cisco-cloud-observability-endpoint>/v1/trace
    logs_endpoint: <cisco-cloud-observability-endpoint>/v1/logs     
    auth: 
      authenticator: oauth2client
YML

Configure Extensions

Extensions are used for tasks that do not involve processing telemetry data. 

To authenticate with Cisco Cloud Observability, you must configure the following extension for the OAuth2 client. The <client-id>, <client-secret>, and <token-url> are generated from the Cisco Cloud Observability UI during step 4 of Install Kubernetes and App Service Monitoring Using Helm Charts.

extensions:   
  oauth2client:     
    client_id: <client-id>
    client_secret: <client-secret>
    token_url: <token-url>
YML

Configure the Service Pipeline

The service pipeline is what enables the configurations set in the receivers: , processors: , extensions:, and exporters: sections.

You must verify that all previously noted exporters, receivers, extensions, and processors are included in the service: pipelines: section. Splunk AppDynamics also recommends adding the memory limited processor to prevent the Collector from crashing when handling high volumes of data:

service:     
  extensions: [oauth2client]
  pipelines:       
    metrics:         
      receivers: [otlp]
      processors: [memory_limiter, batch] 
      exporters: [otlphttp]
    traces:         
      receivers: [otlp]
      processors: [memory_limiter, k8sattributes, batch]
      exporters: [otlphttp]
    logs:         
	  receivers: [otlp]
      processors: [memory_limiter, k8sattributes, batch]
      exporters: [otlphttp]
YML

Collector Configuration Example

This example displays the configuration for the processors, receivers, extensions, exporters, service, and attributes:

receivers:  
  otlp:    
    protocols:      
      grpc:        
        endpoint:      
      http:        
        endpoint:
processors:     
  k8sattributes: 
      passthrough: false
  batch:
      send_batch_size: 1000
      timeout: 10s
      send_batch_max_size: 1000
exporters:   
  otlphttp:     
    metrics_endpoint: <cisco-cloud-observability-endpoint>/v1/metrics
    traces_endpoint: <cisco-cloud-observability-endpoint>/v1/trace
    logs_endpoint: <cisco-cloud-observability-endpoint>/v1/logs
    auth: 
      authenticator: oauth2client
extensions:   
  oauth2client:     
    client_id: <client-id>
    client_secret: <client-secret>
    token_url: <token-url>
service:     
  extensions: [oauth2client]
  pipelines:       
    metrics:         
      receivers: [otlp]
      processors: [memory_limiter, batch] 
      exporters: [otlphttp]
    traces:         
      receivers: [otlp]
      processors: [memory_limiter, k8sattributes, batch]
      exporters: [otlphttp]
    logs:         
	  receivers: [otlp]
      processors: [memory_limiter, k8sattributes, batch]
      exporters: [otlphttp]
YML

Verify Configuration

To validate that your configuration is working properly, review your OpenTelemetry Collector logs. The output from the console log should not have any errors.

You can also set up a logging exporter to monitor the Collector. See Logging Exporter for more information.

Next Steps

Monitor Your Applications

  1. Log into the Cisco Cloud Observability UI.
  2. On the Observe page, navigate to the Application Performance Monitoring domain. This domain contains links to entity-centric pages that can be used to monitor your applications.
  3. Click an entity name to monitor your applications. For more information on the entities that can be monitored, see Application Performance Monitoring Entities. 

OpenTelemetry™ and Kubernetes® (as applicable) are trademarks of The Linux Foundation®.