Kafka Instrumentation for .NET

This document contains references to third-party documentation. Splunk AppDynamics does not own any rights and assumes no responsibility for the accuracy or completeness of such third-party documentation.

The .Net agent provides Kafka instrumentation support for the NuGet package Confluent.Kafka for versions 1.0.0 or later. This package is used to send and receive messages from a Kafka topic. This instrumentation is applicable to both .NET and .NET Framework. 

Exit Call Instrumentation

The agent discovers Kafka backend exit point when an upstream application sends a message to a topic. The agent identifies a backend with the following properties:

• Vendor: Apache
  
• Topic Name: It is the destination where the message is delivered. If the topic name is not available, the agent records the topic name as unknown-topic.

The agent names the backend as Topic Name.

Synchronous Exit Calls (Version 1.0.0 or later)

System.Void Confluent.Kafka.Producer<TKey,TValue>.Produce(
    Confluent.Kafka.TopicPartition,
    Confluent.Kafka.Message<TKey,TValue>,
    System.Action<Confluent.Kafka.DeliveryReport<TKey,TValue>>)

Asynchronous Exit Calls

System.Threading.Tasks.Task<Confluent.Kafka.DeliveryReport<TKey,TValue>>
Confluent.Kafka.Producer<TKey,TValue>.ProduceAsync(
    Confluent.Kafka.TopicPartition,
    Confluent.Kafka.Message<TKey,TValue>,
    System.Threading.CancellationToken)

System.Threading.Tasks.Task<Confluent.Kafka.DeliveryReport<TKey,TValue>>
Confluent.Kafka.Producer<TKey,TValue>.ProduceAsync(
    Confluent.Kafka.TopicPartition,
    Confluent.Kafka.Message<TKey,TValue>)

Entry Point Instrumentation

The agent discovers Kafka entry point when downstream application receives a message from a topic. The agent names the backend as Topic Name. Topic name is the message origin source. If the topic name is not available, the agent records the topic name as unknown-topic.

Instrumentation Methods

Confluent.Kafka.ConsumeResult<TKey,TValue>
Confluent.Kafka.Consumer<TKey,TValue>.Consume(System.Int32)

Confluent.Kafka.ConsumeResult<TKey,TValue>
Confluent.Kafka.Consumer<TKey,TValue>.ConsumeImpl<TKey,TValue>(
    System.Int32,
    Confluent.Kafka.IDeserializer<TKey>,
    Confluent.Kafka.IDeserializer<TValue>)

The Agent supports the Consume Loop message receiving pattern.

The Consume Loop

A typical Kafka consumer application revolves around a consume loop, where the Consume method is repeatedly called to retrieve records that have been pre-fetched by background threads.

...

using (var consumer = new ConsumerBuilder<Ignore, string>(config).Build())
{
    consumer.Subscribe(topics);

    while (!cancelled)
    {
        var consumeResult = consumer.Consume(cancellationToken);

        // handle consumed message.
        ...
    }

    consumer.Close();
}

If your application uses single threaded implementation between receiving messages, you can register queue-single-threaded app agent node property and set it to true. See App Agent Node Properties for instructions on how to register a node property.

The threading architecture defines how the agent calculates the call timing of the message receiver:

• For single-threaded queues: The agent calculates the call time between receive requests. The instrumentation begins at the end of the first Consume method and ends when the next Consume method begins.
• For multi-threaded queues: The agent does not capture timing and finishes entry point immediately. The agent does not detect exit calls while handling the message for this pattern.

If you are using asynchronous operations between message consumption, it's likely that you're not utilizing a single-threaded architecture. Therefore, you should either remove the queue-single-threaded node property or set it to false.