The collector acts as the service pipeline by gathering telemetry data from your applications and exporting it to a backend. With Splunk AppDynamics for OpenTelemetry™, the collector collects and sends distributed traces to the Splunk AppDynamics backend.

You can use one of the following two collectors:

  1. Splunk AppDynamics Distribution for OpenTelemetry Collector
  2. Splunk AppDynamics Distribution for OpenTelemetry Collector MSI
  3. OpenTelemetry Collector

It is recommended to use Splunk AppDynamics Distribution for OpenTelemetry Collector for seamless integration with the Splunk AppDynamics backend.

The collector consists of these components:

  • Receivers: Receives data from OpenTelemetry sources (tracers and SDKs).
  • Processors: Allows additional processing of data, such as data enrichment and batching.
  • Exporters: Converts the data into a specified format and sends the converted data to a chosen destination.
  • Extensions: Extensions provide additional functionalities such as health monitoring, service discovery, and data forwarding. This does not involve processing telemetry data.

See the following sections to configure these components in your OpenTelemetry otel-config.yaml file. Once configured, the components must be enabled via pipelines in the service section.

Before You Begin

To get started with Splunk AppDynamics for OpenTelemetry:

  1. Download the collector of your choice and deploy it in your computing environment.

    1. Go to Downloads and search for Collector for OpenTelemetry. 
    2. Based on your environment, download any of the following Splunk AppDynamics Distribution for OpenTelemetry Collector >= 22.8.0.
      • OpenTelemetry Collector Linux ARM64
      • OpenTelemetry Collector Darwin ARM64
      • OpenTelemetry Collector Windows AMD64
      • OpenTelemetry Collector Linux AMD64
      • OpenTelemetry Collector Darwin AMD64
    1. Go to JFrog and download the applicable collector MSI. For example, appdotelcol-amd64-23.3.0765.msi.

    2. Run the following command to install the MSI.

      msiexec.exe /i "<msi-file-name>.msi"
      POWERSHELL
    3. See the following sections to prepare a working configuration for the collector.
    1. Go to the following file and replace the existing configurations with your configuration.
      C:\Program Files\AppDynamics\AppDynamics Distribution of OpenTelemetry Collector\otel-config.yaml

    2. Run the following command to restart the Windows service.

      net stop appdotelcol
net start appdotelcol
      POWERSHELL
    1. Go to GitHub and download otelcol-contrib version >= 0.36.0.
    2. Deploy the OpenTelemetry Collector in your computing environment.

    We recommend using OpenTelemetry Protocol (OTLP) version >= 0.9.0 as the specified trace receiver. For a list of verified OTLP open-source versions, see Verified Open Source Versions.

  2. In the Controller top navigation panel, click OTel > Get Started.
  3. Click Generate Access Key.  

    If you have already completed onboarding, you can retrieve the API Key in the Controller:

    1. In the top navigation panel, click OTel.
    2. Under the Access Key section, click Show.
    3. Copy the key to use in the Configure Exporters section below.

Configure Receivers

Receivers specify how data gets into the OpenTelemetry Collector. Receivers use OTLP to support transaction-oriented application data processing.

We recommend you use OpenTelemetry Protocol (OTLP) version >= 0.9.0 as the specified trace receiver. For a list of verified OTLP open-source versions, see Verified Open Source Versions.

In the receivers: OTLP: section, specify the Collector endpoints:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint:
      http:
        endpoint:
YML

Configure Processors

Processors allow additional processing of data, such as data enrichment and batching.

You must configure the processors section to add Controller information and (optional) custom data collection.

Resource Processor

Add the Controller account, host, and port to the resource processor:

In the resource: attributes: section, add the following:

  • appdynamics.controller.account: Your Splunk AppDynamics Controller account name
  • appdynamics.controller.host: Your Splunk AppDynamics Controller hostname
  • appdynamics.controller.port: Your Splunk AppDynamics Controller port number

You also must add service.name and service.namespace resource attributes, but you have three options for how to add them:

  • Set the service.name and service.namespace from inside your application code.
  • Set service.name in your OpenTelemetry otel-config.yaml file.
  • Set service.name and service.namespace in the OTEL_RESOURCE_ATTRIBUTES environment variable.

See Configure Resource Attributes for more information.

Resource Processor

processors:
  resource:
    attributes:
  	- key: appdynamics.controller.account
      action: upsert
      value: "acme"
  	- key: appdynamics.controller.host
      action: upsert
      value: "acme.saas.appdynamics.com"
  	- key: appdynamics.controller.port
      action: upsert
      value: 443 # can be an integer or string.
YML

Batch Processor

The batch processor accepts spans, metrics, or logs and places them into batches in the OpenTelemetry Collector pipeline.

Configure the following batch: attributes to help improve performance:

  • timeout: Time after which a batch is sent, regardless of size. The default is 30 seconds.

  • send_batch_size: Number of spans or metrics after which a batch is sent. The default is 8,192.

    We recommend batching 90 spans or less per request for optimal results.

Batch Processor

processors:
  batch:
    timeout: 30s
    send_batch_size: 90
YML

Configure Exporters

The exporter is how data is sent to the Splunk AppDynamics backend. For Splunk AppDynamics for OpenTelemetry, only HTTP is supported to transmit OTLP trace export calls. Your tracer/SDKs exporter should be based on OTLP version >= 0.9.0.

In the exporters: otlphttp: section, you must configure the following attributes with the Splunk AppDynamics endpoint and API key:

exporters:
  otlphttp:
    endpoint: "<appd-endpoint>"
    headers: {"x-api-key": "<x-api-key>"}
YML

For more details on <x-api-key> and <appd-endpoint> see Resource Attribute Descriptions.

For more details on <appd-endpoint> see Regions and Endpoints.

Configure the Service Pipeline

The service pipeline is what enables the configurations set in the receivers: , processors: , and exporters: sections.

You must verify that all previously noted exporters, receivers, and processors are included in the service: pipelines: section:

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [resource, batch]
      exporters: [otlphttp]
YML

Regions and Endpoints

Please note that your OpenTelemetry data may transit through regions different from where your Controller is hosted if you configure your OpenTelemetry Collector using an Splunk AppDynamics endpoint located outside of the region where your Controller is hosted.

Region

City

Endpoint

Americas

Oregon

See SaaS Domains and IP Ranges for OpenTelemetry endpoints.

Sao Paulo

EMEA

Frankfurt

London (Coming soon)

APAC

Mumbai

Singapore

Sydney

Attribute Descriptions

KeyLocationTypeExampleRequiredDescription
appd-endpoint

otlphttp exporter


string

See Supported Regions.

Yes

Your Splunk AppDynamics endpoint for your region where the OpenTelemetry Collector sends the ingested traces. 

x-api-keyotlphttp exporterstring<alpha numeric key>
Yes

Your Splunk AppDynamics API key must be defined as an HTTP header.

To obtain your unique x-api-key, you should work closely with your Splunk AppDynamics account team.

appdynamics.controller.accountresource attributestringacmeYes

Splunk AppDynamics Controller account name.

appdynamics.controller.hostresource attributestringacme.saas.appdynamics.comYes

Splunk AppDynamics Controller host used by OpenTelemetry.

Do not include http:// or https://.

appdynamics.controller.portresource attributestring443Yes

Splunk AppDynamics Controller port number.

service.nameresource attributestringshoppingcartYes

Logical name of the service; equivalent to your Splunk AppDynamics tier name. Set the corresponding service.name resource attribute for every service being monitored using either the SDK, or by setting the OTEL_RESOURCE_ATTRIBUTES environment variable.

Must be the same for all instances of horizontally scaled services.

service.namespaceresource attributestringShop
Yes

A namespace for the service.name; equivalent to your Splunk AppDynamics application name. Set the corresponding service.namespace  resource attribute for every service being monitored using either the SDK, or by setting the OTEL_RESOURCE_ATTRIBUTES environment variable.

A non-null, non-empty string value for service.namespace is required.

Collector Configuration Sample

This example includes configuration for the processors, receivers, exporters, service, and attributes:

processors:
  resource:
    attributes:
      - key: appdynamics.controller.account
        action: upsert
        value: "acme"
      - key: appdynamics.controller.host
        action: upsert
        value: "acme.saas.appdynamics.com"
      - key: appdynamics.controller.port
        action: upsert
        value: 443
      - key: service.namespace
        action: upsert
        value: shop
  batch:
    timeout: 30s
    send_batch_size: 90
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: "<grpc-endpoint>" # Specify the endpoint
      http:
        endpoint: "<http-endpoint>" # Specify the endpoint
exporters:
  otlphttp:
    endpoint: "https://<your-region>-sls-agent-api.saas.appdynamics.com" # See the Supported Regions section above.
    headers: {"x-api-key": "****************"}
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [resource, batch]
      exporters: [otlphttp]
YML

Start the Collector

  1. Ensure that the downloaded collector binary has execution privileges.
  2. Run the following commands to start sending data to the common ingestion stream.
./appdotelcol_* --config otel-config.yaml
CODE

To validate that your configuration is working properly, review your OpenTelemetry Collector logs. The output from the console log should not have any errors.

You can also set up a logging exporter to monitor the Collector. See Logging Exporter for more information.

Next Steps

After deploying and configuring the OpenTelemetry Collector, you must Configure Resource Attributes.

OpenTelemetry™ is a trademark of The Linux Foundation®.