Splunk AppDynamics Python Agent helps you to monitor GenAI apps built using the LangChain framework. 

Prerequisites

Ensure to set the enable-langchain flag to true in /path/to/appdynamics.cfg file of the Python Agent. See [instrumentation].

Python agent supports the following versions of LangChain LLMs in Splunk AppDynamics.

Monitor Ollama APIs

To monitor LLMs using Ollama APIs, the Python Agent reports these metrics:

• Input Tokens
• Output Tokens
• Prompt count
• Embedding queries count
• Errors

For Token Metrics, ensure to install transformers Python library. See transformers.

pip install transformers

CODE

Create a Custom Dashboard to Monitor Ollama APIs

1. Download Custom_dashboard_script.zip It contains:
   

   Directory	Description
   templates	Contains json templates to create the dashboard files.
   dashboard	Dashboard json files are created here. It contains the sample json files.
   main.py	Python script to create the dashboards.
   readme.md	Description of the solution.

    

2. Run the following command:

   cd Custom_dashboard_script
   python3 main.py

   CODE

   The preceding command prompts you to specify the dashboard type (application or tier) that you want to create.

3. Based on your selected dashboard type, you will be prompted to specify the following dashboard detail.

4. Import the dashboard in the Controller UI. See Create and Manage Custom Dashboards and Templates.

Navigate to Tiers & Nodes > My Dashboards and create a dashboard for the selected application by copying and pasting widgets from the existing dashboard for easier navigation.

Example Dashboard:

Monitor Vectorstores

When you instrument the app with Python Agent, it captures the following for Vectorstores:

• Exit calls to vector databases: pgvector and chroma.
• Counts of documents retrieved, similarity search score, and errors in vector databases
• Calls per minute