Monitor OpenAI APIs

AppDynamics Python Agent helps you monitor the OpenAI API usage and related metrics. It monitors the ChatCompletion, Completion, and Moderation APIs type requests created using the openai-python library to the OpenAI backend. The agent also captures the prompts and responses used by the OpenAI API models.

The following metrics are reported at the application, tier, and node levels:

• Calls per minute
• Errors per minute
• Tokens

The following are the example metric paths:

• Per model metric path - Application Infrastructure Performance|OpenAI_Tier|OpenAI|ada|Tokens
• All model cumulative metrics path - Application Infrastructure Performance|OpenAI_Tier|OpenAI|Tokens - All Models
• Net application level metrics path - Overall Application Summary|OpenAI - Tokens - ada

These metrics are reported even if the exit call is outside the business transaction. However, the Average Response Time(ms) metric is not reported when the exit call is outside the business transaction.

OpenAI Moderation API

From version 24.2.0, Python Agent supports OpenAI Moderation API. The moderation API will prompt the users if the queried statement is policy compliant or not. The Python Agent captures the following metrics:

• Calls to moderations API: The number of times moderation API is used every minute. (OpenAI|moderations|Calls per minute)
• Flagged category metrics: The number of times specific types of content, like hate speech, are flagged. (OpenAI|moderations|Flagged Calls by category|hate)
• Flagged queries count: The total amount of flagged content. (OpenAI|moderations|Flagged queries)
• Total queries count: The overall number of queries processed. (OpenAI|moderations|Total queries)

Prerequisites

Ensure that you meet the following prerequisites to monitor the OpenAI API exit calls with AppDynamics:

• Install Python Agent 23.8.0 or later. See Install Python Agent.
• Import custom dashboards and templates.
• Set the enable-openai flag to true in the /path/to/appdynamics.cfg file of the Python Agent. See [instrumentation].

Create a Custom Dashboard

1. Download Custom_dashboard_script.zip. It contains:
   

   Directory	Description
   templates	Contains json templates to create the dashboard files.
   dashboard	Dashboard json files are created here. It contains the sample json files.
   main.py	Python script to create the dashboards.
   readme.md	Description of the solution.

    

2. Run the following command:

   cd Custom_dashboard_script
   python3 main.py

   CODE

   The preceding command prompts you to specify the dashboard type (application or tier) that you want to create.

3. Based on your selected dashboard type, you will be prompted to specify the following dashboard details.
   

   Variable	Description	Default	Possible Values	Required
   Dashboard Name	The dashboard name to be displayed on Dashboard & Reports in the Controller UI.	OpenAI_App_dashboard	app_dashboard	Optional
   Application Name	Name of the Application with OpenAI backend calls.	none	openai_app_1	Yes
   OpenAI Endpoints Host Name	The hostname of the OpenAI endpoint.	api.openai.com	api.openai.com	Optional
   OpenAI Endpoints Port	The port of the OpenAI endpoint.	443	443	Optional

   Variable	Description	Default	Possible Values	Required
   Dashboard Name	The dashboard name to be displayed on Dashboard & Reports in the Controller UI.	tier_dashboard	tier_dashboard	Optional
   Application Name	Name of the Application with OpenAI backend calls.	none	openai_app_1	Yes
   Tier Name	Name of the Tier exposed to OpenAI backend calls.	none	openai_tier_1	Yes
   Business Transaction Name	The name of the business transaction reported to the host. This is to capture API Response time metric.	none	openai_bt	Yes
   OpenAI Endpoints Host Name	The hostname of the OpenAI endpoint.	api.openai.com	api.openai.com	Optional
   OpenAI Endpoints Port	The port of the OpenAI endpoint.	443	443	Optional

   Variable	Description	Default	Possible Values	Required
   Dashboard Name	The dashboard name to be displayed on Dashboard & Reports in the Controller UI.	OpenAI_Moderation_App	moderation_app_dashboard	Optional
   Application Name	Name of the Application with OpenAI backend calls.	none	openai_app_1	Yes

   Variable	Description	Default	Possible Values	Required
   Dashboard Name	The dashboard name to be displayed on Dashboard & Reports in the Controller UI.	OpenAI_Moderation_Tier	moderation_tier_dashboard	Optional
   Application Name	Name of the Application with OpenAI backend calls.	none	openai_app_1	Yes
   Tier Name	Name of the Tier exposed to OpenAI backend calls.	none	openai_tier_1	Yes

    

4. Import the dashboard in the Controller UI. See Create and Manage Custom Dashboards and Templates.

Navigate to Tiers & Nodes > My Dashboards and create a dashboard for the selected application by copying and pasting widgets from the existing dashboard for easier navigation.

Example Dashboard:

Create a Custom Cost Metric Expression

You can create a custom cost metric expression by multiplying the Input Tokens and Completion Tokens metrics by their respective costs.

For example, the cost per one million tokens for gpt-3.5-turbo-0125 is:

• Input Tokens = 0.5$
• Completion Tokens = 15$

In this scenario, you can create a custom expression for gpt-3.5-turbo-0125 to calculate the cost as follows:

({input Tokens}*0.5 + {Completion Tokens}*1.5)/1000000

Use this custom expression to create a pie chart for cost distribution across different models or an overall cost widget that includes the cost function for all models. Complete the following steps to create a cost per model widget to the existing OpenAI Dashboard:

1. Add a pie chart widget to an existing openai dashboard, see Metric Expression.
2. Specify the input_token and completion_token cost and multiply them by the token count.
   1. Data Source: Select the application name.
   2. Metric Category: Select Tier / Node Health -Hardware, JVM, CLR (cpu,heap, disk I/O, etc).
   3. Metric: Select Metric Expression.
   4. Edit the metric Expression.
      
3. In Metric Expression, add two variables and assign names to them.
4. In Metric Selection, specify the relative metric path for input and completion tokens:
   
   
   • Input Tokens: Agent|OpenAl|gpt-3.5-turbo|Input Tokens
   • Completion Tokens: Agent|OpenAI|gpt-3.5-turbo|Completion Tokens
5. In Metric Expression, define the metric expression as follows:
   

   ({input_token_gpt3.5)*0.5 + {completion_token_gpt3.5)*1.5)/1000000

   CODE

This metric expression includes the cost multiplier per million tokens for both types.