OpenTelemetry Trace Sampling

Overview

This page introduces the concept of OpenTelemetry™ trace sampling.

This document contains references to third-party documentation. Splunk AppDynamics does not own any rights and assumes no responsibility for the accuracy or completeness of such third-party documentation.

Monitoring applications by OpenTelemetry can produce a high volume of spans and traces. This, in turn, can have a high impact on the computing resources engaged in processing the spans, such as CPU, memory, network bandwidth, or storage. It can also cause application performance degradation, and even instability. To address these issue, OpenTelemetry offers a capability of sampling spans, meaning only a fraction of the spans created by OpenTelemetry tracers will get collected.

Sampling can be also an attractive option if you simply want to reduce the total cost of your observability solution.

Cisco Cloud Observability offers a trace sampling solution that enables you to use head sampling (at this time, for Java only), tail sampling, or both. For both of these sampling options, Cisco Cloud Observability uses probabilistic sampling. The actual probability values used for sampling are encoded along each sampled span. This allows Cisco Cloud Observability to estimate such metrics as the true total number of calls per minute, even when sampling is active. At the same time, Cisco Cloud Observability sampling attempts to preserve as many complete traces as possible.

Even though both samplers use probabilistic sampling, they are configured by specifying maximum sampling rate (per time unit). This means that in response to varying volumes of spans, the samplers dynamically adjust sampling probability in attempt to not exceed the specified rate. This approach eliminates the need to periodically adjust sampling configuration in anticipation of application load changes and guarantees predictable volume of sampled traces.

Head Sampling

The Cisco Cloud Observability head sampler for Java is provided as an OpenTelemetry Java Agent extension, available for download from the  Splunk AppDynamics portal. To enable the Splunk AppDynamics head sampler, complete the following:

Before You Begin

The workflows on this page assume that:

• You are using OpenTelemetry Java Agent version >=1.18  
• You have downloaded the AppDynamics Extension for OSS OpenTelemetry Java Agent extension zip from the AppDynamics Download Portal under AppDynamics OpenTelemetry Extensions or from the public JFrog artifactory (group: com.appdynamics.opentelemetry.extension, name: appd_oss_java_agent_extension).

If your application is deployed in a container environment, you can download the AppDynamics Extension for OSS OpenTelemetry Java Agent docker image from Docker Hub into a shared volume utilizing an init container.

Using the Extension 

The extension cannot be used on its own and must always be used with a supported version of the OSS OpenTelemetry Java Agent. The agent can be downloaded from the OTel Java Agent releases.

1. Copy the appd_oss_java_agent_extension.jar file to the host that is running the application which you are monitoring with the OpenTelemetry Java Agent.
2. Modify the application startup command to add the full path to the extension jarfile. For example:

        java -javaagent:path/to/opentelemetry-javaagent.jar \
          -Dotel.javaagent.extensions=path/to/appd_oss_java_agent_extension.jar \
          -jar myapp.jar

Enabling Splunk AppDynamics Head Sampler

To enable the Splunk AppDynamics head sampler, in addition to loading the extension as above, use the following java command line option

-Dappdynamics.sampler.maximum.request.rate=<RATE> with the format of <RATE> being <INTEGER>/<TIMEUNIT>. The possible time units are s for seconds, m for minutes and h for hours. For example, to limit the number of sampled (exported by the tracer) requests to approximately 150 per minute, use -Dappdynamics.sampler.maximum.request.rate=150/m. 

The sampler uses probability sampling, adjusting the probability dynamically to deliver sampled requests at a rate not exceeding the specified rate. The actual rate may be lower or higher, especially when observed over short periods of time. The probability used for sampling a span is recorded in the span's tracestate. Cisco Cloud Observability uses this information to calculate metrics such as calls-per-minute, providing estimations for the true values, regardless of the sampling rate. Higher sampling rates result in more accurate estimations.

For distributed applications, it is recommended that the sampler is configured only for the application instances which start OpenTelemetry traces (i.e. create ROOT spans). The sampler will count the ROOT spans as requests. If all other application nodes use the default sampling configuration (i.e. ParentBasedSampler) then all reported traces will be complete and their rate will not exceed, on average, the rate specified as the maximum.

However, in microservice architectures, one or more services can get heavy load even with relatively low volume of transactions. To address this issue, additionally configure the sampler at the service node for which the volume of sampled requests needs to be lowered. For those services, the sampler will count their head spans (typically of the SERVER kind) as requests. Configuring the sampler for non-root services can make a number of traces incomplete, but the sampler will always attempt to keep as many complete traces as possible.

Tail Sampling

The Cisco Cloud Observability tail sampler is available as part of the AppDynamics distribution of the OpenTelemetry collector.

Enabling Splunk AppDynamics Tail Sampler

Tail sampling is enabled at a dedicated set of AppDynamics Distribution of OpenTelemetry Collectors, called tail sampling collectors. When tail sampling is enabled, the upstream AppDynamics Distribution of OpenTelemetry Collectors sends trace data to the tail sampling collectors where the spans get "down sampled" and sent to the Splunk AppDynamics Cloud. Tail sampling makes sampling decisions after receiving the entire trace; it enables the tail sampler to look into the characteristics of a full trace, for example, end to end latency or error status code, and sample different traces with different sampling rates. Similar to head sampling, tail sampling also configures the sampling rate by setting the desired output rate, so that the span export rate stabilizes, regardless of the change of the input traffic. 

To enable the tail sampling, set appdynamics-otel-collector.presets.tailsample.enable to true. By default, one tail sampling collector will be deployed and span exporting rate will be maxed at 60k spans per minute:

appdynamics-otel-collector:
  presets:
    tailsampler:
      enable: true

Configuring Tail Sampling

Tail sampling at AppDynamics Distribution of OpenTelemetry Collector has flexible configurations to meet various use cases. Before configuring tail sampling, you must understand tail sampling pipeline: 

Tail Sampling Pipeline

Tail sampling pipeline is the processor pipeline at tail sampling collector. The tail sampling pipeline contains 3 main steps:

1. Trace grouping to group all spans from one trace. Trace grouping is done by waiting a fixed period of time and after receiving the first span of a trace, the trace is considered completed when the waiting period ends.
2. Trace classification and sampling, samples the grouped traces by match policies. In this step, traces are classified by different policies according to its characteristic like end to end latency and error status code. A different sampling rate could be configured to apply to different categories of the traces.
3. Proportional sampling happens at the end of the sampling pipeline as it works as the final output control of the span traffic. When you configured too many trace categories and have difficulties controlling the span exporting rate from the previous step,  the proportional sampling helps you sample the traffic to a desired rate and keep the ratio of sampled categories unchanged.

Configuring Overall Span Exporting Rate

The span exporting rate could be configured in two ways, either configuring the replicas number of the tail sampling collector or configuring the proportional sampling step in the pipeline directly. To configure tail sampling after enabling it:

Configuring the replica number of the tail sampling collector

By default each tail sampling collector exported 60k spans per minute. You could increase the span exporting rate by increasing the replica number of the trail sampling collector:

appdynamics-otel-collector:
  presets:
    tailsampler:
      replicas: 1

Configuring the proportional sampling

To configure the span exporting rate for one tail sampling collector, you could configure the proportional sampler by its spans_per_period option. By default, the spans_per_period controls the span exporting number for one second. The following configuration, which is also the default configuration, displays the configured the span exporting rate to be 1,000 spans per second:

appdynamics-otel-collector:
  presets:
    tailsampler:
      consistent_proportional_sampler:
        spans_per_period: 1000

Configuring Trace Classification and Sampling

To have a different sampling rate for a distinct category of the traces, you can configure the trace classification and sampling step. In this step, traces are classified by policies, which are similar to OpenTelemetry Collector Tail Sampling Processor. The difference is that for each policy, one additional option sampler_name is added to indicate that traces matched with the policy should be sampled by a sampler with that name.

The sampler_name is composed by <type>/<name> and currently, consistent_reservoir_sampler is the only supported type. The name must be alphanumeric and unique.

The following example are three default policies:

1. Errors-policy matches traces containing error status code. The matched traces are sampled by consistent_reservoir_sampler/error.
2. High-latency matches traces with end to end latency longer than 10 seconds. The matched traces are sampled by consistent_reservoir_sampler/latency.
3. Always-on matches any trace that does not match at anyother policies. The matched traces are sampled by consistent_reservoir_sampler/anyother. 

The interested-and-high-latency policy is a customized policy, which composites latency and attribute match policies. It will only match a trace that contains attribute key-value pair "interested":"interested.url" and has latency higher than 5 seconds. The matched traces for this policy will be sampled by consistent_reservoir_sampler/interested_latency. 

At the samplers section you can find the corresponding samplers. By default, all samplers exports one batch of spans every second, so all samplers in this example are configured to output spans at 1,000 spans per second:

appdynamics-otel-collector:
  presets:
    tailsampler:    
      trace_classification_and_sampling:
        policies:
        - name: errors-policy
          type: status_code
          status_code:
            status_codes: [ERROR]
          sampler_name: "consistent_reservoir_sampler/error"
 
        - name: high-latency
          type: latency
          latency:
            threshold_ms: 10000
          sampler_name: "consistent_reservoir_sampler/latency"
 
        - name: interested-and-high-latency
          type: and
          and:
            and_sub_policy:
              - name: high-latency
                type: latency
                latency:
                  threshold_ms: 5000
              - name: interested-url
                type: string_attribute
                string_attribute:
                  key: "interested"
                  values: ["interested.url"]  
          sampler_name: "consistent_reservoir_sampler/interested_latency"
  
        - name: always-on
          type: always_sample
          sampler_name: "consistent_reservoir_sampler/anyother"
 
  
        
        samplers:
          consistent_reservoir_sampler:
            error:
              reservoir_size: 1000
            latency:
              reservoir_size: 1000
            interested_latency:
              reservoir_size: 1000
            anyother:
              reservoir_size: 1000

Configuring Trace Grouping

Trace grouping assumes that traces will be completed within a fix period of time. By default, you must wait 30 seconds for a trace to complete.

Longer wait times require more memory while shorter wait times may cause a broken trace. The trace may be sampled in parts and each part may have different sampling decisions.

The following example displays how to configure trace grouping:

appdynamics-otel-collector:
  presets:
    tailsampler:  
      trace_classification_and_sampling:
        decision_wait: 30s 

Trace grouping is done by waiting a fixed period of time after the tail sampling collector received the span from a trace at the first time. By default, it wait 30 second for a trace to complete. Please notice that long wait time will require more memory, while short wait time may cause trace broken, because the trace may be sampled in parts and each part may got different sampling decisions.

Other Configurations for Tail Sampling Collector

Other aspects of the tail sampler can also be configured. For example, after changing the trace grouping and sampling rate, the memory and CPU also needs adjusting; logging exporters could also be added for debugging purposes. All the configuration applied to the original AppDynamics Distribution of OpenTelemetry Collector can also be applied to the tail sampling collector.

The following example displays how to configure the CPU and memory and shows how to add logging exporters:

appdynamics-otel-collector:
  presets:
   multi_tier:
     tailsampler:
       spec:
         resources:
           limits:
             cpu: 200m
             memory: 1024Mi        
       configOverride:
         service:
           pipelines:
             traces:
               exporters: [otlphttp, logging]

Performance of Tail Sampling Collector

By default, the tail sampling collector is deployed as Deployment with one replica. The following resources are requested by default:

All performance test numbers are derived from test environments. Real world application results may vary depending on the size of the spans and traces.

Assuming the default resources are allocated, one tail sampling collector should be able to handle the following load profile:

Backend Sampling

You can configure backend trace sampling to reduce your storage costs. This feature enables you to selectively store trace data based on specific criteria (a filter). You can create and modify these trace storage filters through the Knowledge Store API.

Sample Filters

• Store everything: Use a wildcard filter. DEFAULT?

  *

  JSON

• Store only the traces with a specific attribute:

  ${function($v) {$v.attributes[name='ServiceName'].value.value = 'opentelemetry-trace-writer'}}

  JSON

• Store only the traces with a specific status code:

  ${function($v) {$v.statusCode = 'ERROR'}}

  JSON

• Store only the traces with a specific minimum latency: 

  ${function($v){$v.endedAt - $v.startedAt > 100}}

  JSON

• Store only the traces with a specific value for a given tag: 

  ${function($v) {$v.tags[name='service.instance.id'].value = 'abc'}}

  JSON

Send the Configuration to the Knowledge Store

Sample cURL command that adds a filter (in the request body key operation) to a Knowledge Store table named platformindexingv4:indexFilterV4:

// appd-tid is base64 encryption of tenantId, layer-id is the customer's id.
curl --location 'http://localhost:18084/v1beta/objects/platformindexingv4:indexFilterV4' \
--header 'accept: application/json' \
--header 'appd-pid: {{token provided to customers}}' \
--header 'appd-pty: {{another token provided to customers}}' \
--header 'appd-tid: {{tenantId's BASE64 encryption}}' \
--header 'layer-type: TENANT' \
--header 'layer-id: {{tenantId}}' \
--header 'Content-Type: application/json' \
--data '{
    "meltType": "trace",
    "filterName": "filter9_attributeMap_test",
    "operation": "${function($v) {$v.statusCode = 'ERROR'}}",
    "samplingPercentage": 100
}'

OpenTelemetry™ is a trademark of The Linux Foundation®.