Monitor OpenAI API with Java Agent

This document contains references to the OpenAI® documentation. Splunk AppDynamics does not own any rights and assumes no responsibility for the accuracy or completeness of such third-party documentation.

The Java Agent helps you monitor the OpenAI API usage and related metrics. It monitors the Completion, ChatCompletion and Moderation APIs called using the TheoKanning OpenAI Java framework to the OpenAI backend.

The following metrics are reported at the application, tier, and node levels:

• Calls per minute
• Cost
• Errors per minute
• Tokens

The following are the example metric paths:

• Per model metric path - Application Infrastructure Performance|OpenAI_Tier|OpenAI|ada|Cost
• All model cumulative metrics path - Application Infrastructure Performance|OpenAI_Tier|OpenAI|Cost - All Models
• Net application level metrics path - Overall Application Summary|OpenAI - Cost - ada

These metrics are reported even if the exit call is outside the business transaction. However, the Average Response Time(ms) metric is not reported when the exit call is outside the business transaction.

OpenAI Moderation API

From version 24.1.0, Java Agent supports OpenAI Moderation API. The moderation API will prompt the users if the queried statement is policy compliant or not. The Java Agent captures the following metrics:

• Calls to moderations API: OpenAI|moderations|Calls per minute
• Flagged category metrics: OpenAI|moderations|Flagged Calls by category|hate
• Flagged queries count: OpenAI|moderations|Flagged queries
• Total queries count: OpenAI|moderations|Total queries

Prerequisites

Ensure that you meet the following prerequisites to monitor the OpenAI API exit calls with Splunk AppDynamics:

• Install Java Agent 23.7.0 or later. See Install the Java Agent.
• Set the enable-openai-metrics node property to true.
• Install Python 3.0, to run the Python script required to create custom dashboards. These custom dashboards are imported into the Controller UI and display OpenAI-related metrics. See Import and Export Custom Dashboards and Templates Using the UI.

Create a Custom Dashboard

1. Download Custom_dashboard_script_v24.1.zip. It contains:
   

   Directory	Description
   templates	Contains json templates to create the dashboard files.
   dashboard	Dashboard json files are created here. It contains the sample json files.
   main.py	Python script to create the dashboards.
   readme.md	Description of the solution.

   
   

2. Run the following command:

   cd Custom_dashboard_script
   python3 main.py

   CODE

   The preceding command prompts you to specify the dashboard type (application or tier) that you want to create.

3. Based on your selected dashboard type, you will be prompted to specify the following dashboard details.
   

   Variable	Description	Default	Possible Values	Required
   Dashboard Name	The dashboard name to be displayed on Dashboard & Reports in the Controller UI.	OpenAI_App_dashboard	app_dashboard	Optional
   Application Name	Name of the Application with OpenAI backend calls.	none	openai_app_1	Yes
   OpenAI Endpoints Host Name	The hostname of the OpenAI endpoint.	api.openai.com	api.openai.com	Optional
   OpenAI Endpoints Port	The port of the OpenAI endpoint.	443	443	Optional

   Variable	Description	Default	Possible Values	Required
   Dashboard Name	The dashboard name to be displayed on Dashboard & Reports in the Controller UI.	tier_dashboard	tier_dashboard	Optional
   Application Name	Name of the Application with OpenAI backend calls.	none	openai_app_1	Yes
   Tier Name	Name of the Tier exposed to OpenAI backend calls.	none	openai_tier_1	Yes
   Business Transaction Name	The name of the business transaction reported to the host. This is to capture API Response time metric.	none	openai_bt	Yes
   OpenAI Endpoints Host Name	The hostname of the OpenAI endpoint.	api.openai.com	api.openai.com	Optional
   OpenAI Endpoints Port	The port of the OpenAI endpoint.	443	443	Optional

   Variable	Description	Default	Possible Values	Required
   Dashboard Name	The dashboard name to be displayed on Dashboard & Reports in the Controller UI.	OpenAI_Moderation_App	moderation_app_dashboard	Optional
   Application Name	Name of the Application with OpenAI backend calls.	none	openai_app_1	Yes

   Variable	Description	Default	Possible Values	Required
   Dashboard Name	The dashboard name to be displayed on Dashboard & Reports in the Controller UI.	OpenAI_Moderation_Tier	moderation_tier_dashboard	Optional
   Application Name	Name of the Application with OpenAI backend calls.	none	openai_app_1	Yes
   Tier Name	Name of the Tier exposed to OpenAI backend calls.	none	openai_tier_1	Yes

   
   

4. Import the dashboard in the Controller UI. See Create and Manage Custom Dashboards and Templates.