This page explains how to use add the Kafka Prometheus exporter within your deployed environment using Helm Charts. The Cisco Cloud Observability Helm chart package provides all the necessary dependencies. If you don’t have the Cisco Cloud Observability Helm chart installed, see Install Kubernetes and App Service Monitoring.


Requirements

Before you integrate Prometheus, you must have configured Kafka and JMX exporters in your environment. See Prometheus Exporters and integrations.

Scaling Requirements

Each Cisco Cloud Observability Distribution for OpenTelemetry™ Collector replica can process 5,000 Kafka partitions. This limit ensures that you have enough storage for Cisco Cloud Observability metrics, events, logs, and traces (MELT) data. To process additional partitions, add one Cisco Cloud Observability Distribution for OpenTelemetry™ Collector replica for every 5,000 partitions. For example, a 16-node Kubernetes cluster with 16 Cisco Cloud Observability Distribution for OpenTelemetry™ Collector replicas can handle a total of 80,000 Kafka partitions.

This document contains references to third-party documentation. Splunk AppDynamics does not own any rights and assumes no responsibility for the accuracy or completeness of such third-party documentation.

Enable Prometheus Exporter Monitoring for Kafka

To enable the Prometheus exporter monitoring for Kafka using the Cisco Cloud Observability Helm chart you'll need to annotate the collectors-values.yaml file:

  1. Copy and paste the following snippet into the collectors-values.yaml file in your Kubernetes deployment. For the full list of settings, see Advanced Settings for the Cisco AppDynamics Distribution of OpenTelemetry Collector.

    appdynamics-otel-collector:
  enablePrometheus: true
  spec: 
    replicas: <your-desired-number-of-replicas-here>
    CODE

    Set the replicas value to the number of Kubernetes nodes in your Kubernetes cluster. replicas must be set to enable the Cisco Cloud Observability deployment or the Cisco Cloud Observability Distribution for OpenTelemetry™ Collector will not run. See Scaling Requirements.

    After you change the enablePrometheus setting to true or false, you must restart your other collectors to see up-to-date entities and metrics on the Observe page.

  2. Run the Helm chart commands using the latest version to Upgrade Operators and Collectors. Once configured, run this command to validate:

    kubectl get po -n appdynamics
    YML

    Sample output:

    NAMESPACE      NAME                                                              READY   STATUS    RESTARTS        AGE
appdynamics    appd-prom-appdynamics-otel-collector-collector-0                  1/1     Running   0               115s
appdynamics    appd-prom-appdynamics-otel-collector-collector-1                  1/1     Running   0               114s
appdynamics    appd-prom-appdynamics-otel-collector-targetallocator-676bfkxxxv   1/1     Running   0               115s
appdynamics    opentelemetry-operator-controller-manager-58d65d7848-z5t5n        2/2     Running   0               7h19m
    YML

  3. Kubernetes annotations are designed to attach metadata to objects. Annotations are key/value pairs that clients can use to retrieve this metadata. Use Kubernetes annotations to expose the Prometheus exporter endpoints.

    If you start the Prometheus exporter with a path or port that is different from the default values, you need to update the prometheus.io/path and prometheus.io/port annotation values accordingly.


    1. Kafka exporter: 

      appdynamics.com/exporter_type: "kafka"
appdynamics.com/kafka_cluster_name: "<your-kafka-cluster-name-here>"
prometheus.io/path: "/metrics"
prometheus.io/port: "9308"
      CODE

      Once configured, run:

      kubectl describe svc [Kafka exporter service name] [-n namespace]
      CODE

      Sample output:

      Name:              kafka-demo-metrics
Namespace:         kafka
Annotations:       appdynamics.com/exporter_type: kafka
                   appdynamics.com/kafka_cluster_name: kafka-demo
                   prometheus.io/path: /metrics
                   prometheus.io/port: 9308
      YML

    2. JMX exporter for Kafka: 

      appdynamics.com/exporter_type: "kafkajmx"
appdynamics.com/kafka_cluster_name: "<your-kafka-cluster-name-here>"
prometheus.io/path: "/"
prometheus.io/port: "5556"
      CODE

      Once configured, run:

      kubectl describe svc [JMX exporter service name] [-n namespace]
      YML

      Sample output:

      Name:              kafka-demo-jmx-metrics
Namespace:         kafka
Annotations:       appdynamics.com/exporter_type: kafkajmx
                   appdynamics.com/kafka_cluster_name: kafka-demo
                   prometheus.io/path: /
                   prometheus.io/port: 5556
      YML

  4. Update your JMX configuration rules to make sure the metrics generated are properly ingested. These rules need to be applied instead of default rules.

    jmxUrl: <your-full-jmx-url-to-connect-to-here>
lowercaseOutputName: true
lowercaseOutputLabelNames: true
whitelistObjectNames: ["kafka.controller:*","kafka.server:*","java.lang:*","kafka.network:*","kafka.log:*"] 
rules:
  - pattern: kafka.controller<type=(KafkaController), name=(.+)><>(Value)
    name: kafka_controller_$1_$2_$3
    type: GAUGE
  - pattern: kafka.controller<type=(ControllerStats), name=(.+)><>(Count)
    name: kafka_controller_$1_$2_total
    type: COUNTER
  - pattern : kafka.network<type=(RequestMetrics), name=(RequestsPerSec), request=(.+), version=(.+)><>(Count)
    name: kafka_network_$1_$2_total
    type: COUNTER
    labels:
      request: $3
      version: $4
  - pattern : kafka.network<type=(RequestMetrics), name=(TotalTimeMs), request=(.+)><>(Count)
    name: kafka_network_$1_$2_$3_total
    type: COUNTER
    labels:
      request: $3
  - pattern: kafka.server<type=(KafkaServer|ReplicaManager), name=(.+)><>(Value)
    name: kafka_server_$1_total_$2_$3
    type: GAUGE
  - pattern: kafka.server<type=(.+), name=(.+), topic=(.+)><>(Count)
    name: kafka_server_$1_$2_total
    type: COUNTER
    labels:
      topic: $3
  - pattern: kafka.server<type=(DelayedOperationPurgatory), name=(PurgatorySize), delayedOperation=(.+)><>(Value)
    name: kafka_server_$1_$2_$3_$4
    type: GAUGE
  - pattern: kafka.server<type=(ReplicaManager), name=(.+)><>(Count)
    name: kafka_server_$1_broker_$2_total
    type: COUNTER
  - pattern: kafka.server<type=(BrokerTopicMetrics), name=(BytesOutPerSec|BytesInPerSec|MessagesInPerSec)><>(Count)
    name: kafka_server_$1_broker_$2_total
    type: COUNTER
  - pattern: kafka.server<type=(BrokerTopicMetrics), name=(FailedProduceRequestsPerSec|FailedFetchRequestsPerSec)><>(Count)
    name: kafka_server_$1_$2_total
    type: COUNTER
  - pattern: kafka.server<type=(SessionExpireListener), name=(.+)><>(Count)
    name: kafka_server_$2_total
    type: COUNTER
  - pattern: kafka.log<type=(LogFlushStats), name=(LogFlushRateAndTimeMs)><>(Count)
    name: kafka_log_$1_$2_total
    type: COUNTER
  - pattern: kafka.log<type=(Log), name=(Size), topic=(.+), partition=(.+)><>(.+)
    name: kafka_log_$1_$2
    labels:
      topic: $3
      partition: $4
    type: GAUGE
  - pattern : java.lang<type=(.*)>
    type: GAUGE
    CODE

Next Steps

Cisco Cloud Observability collects data and populates the Observe page with entity-centric pages (ECPs). You can now monitor your Kafka entities on the Cisco Cloud Observability UI. See Observe Kafka Entities.

KAFKA is a registered trademark of The Apache Software Foundation and has been licensed for use by Splunk AppDynamics and its affiliates (together, "Cisco AppDynamics"). Splunk AppDynamics has no affiliation with and is not endorsed by The Apache Software Foundation.

Prometheus® and Kubernetes® (as applicable) are trademarks of The Linux Foundation®.