Splunk Integration

Splunk^® Enterprise enables you to search, analyze, and visualize the data collected from the resources in your IT environment. Integrate Splunk Enterprise with Cisco AIOps to collect the alerts from Splunk. Cisco AIOps correlates the Splunk alerts with data from other monitoring tools. This correlation helps to better understand the impact of the outages, improves visibility, and expedites remediation of issues. Cisco AIOps provides AIOps Add-on for Splunk, a native Splunk add-on that sends alerts from Splunk Enterprise to Cisco Observability Platform.

Before You Begin

Before integration, ensure that you meet the following requirements:

• Splunk Enterprise platform and permission to install a Splunk add-on.
• Service principals to send Splunk alerts to Cisco Observability Platform.

Software Requirements

Ensure the following:

• Splunk Enterprise version >=6.5
• Python 3 runtime environment on your Splunk deployment

Install Splunk Add-on

To install the Splunk add-on on your Splunk platform: 

1. On the Splunk Enterprise instance, open JFrog and go to the location maven-releases>com>appdynamics>aiops>aiops-agt-splunk to download the TA-Splunk-AIOps.tar.gz file. If you are using a distributed Splunk Enterprise deployment, download the file on the search head.

2. From the Splunk Web home screen, click Install app from file. For more information, see the Splunk documentation.
3. Restart Splunk Enterprise.

Verify Splunk Add-on

After you install AIOps Add-on for Splunk, verify the following on your Splunk Enterprise:

• AIOps Add-on for Splunk appears on the apps list.
• AIOps Alert Integration appears on the Alert Actions list.

Create Alert

After you install and verify AIOps Add-on for Splunk, create an alert in Splunk Enterprise to push the specific events to Cisco AIOps. The Splunk alert uses search expression to track the events. To create an alert, see Splunk Documentation.

To create or save an alert for AIOps Add-on for Splunk, specify the following details on your Splunk Enterprise:

1. In Search field, specify a search expression to index the specific events. For example:

   index=network OR index=dcnet EIGRP-(IPv4 OR IPV4 OR ipv4).*(Down OR down OR DOWN) |  eval ci_name=dvc,eval ci_id=dvc_id, ci_type="interface"|eval event_name ="Syslog: EIGRP IPv4 Neighbor Down",severity="Medium",priority=3, source=dvc| table ci_name, ci_id, ci_type, event_name,severity,event_id,message,priority, policy_name, start_time, endTime, status, policy_name, source, source_type

   CODE
2. In the Trigger Actions field, select AIOps Alert Integration and specify the following:
   1. URL: The domain URL of your Cisco Observability Platform tenant.
   2. Token URL: The token URL that you obtain after generating the Service Principal in your Cisco Cloud Observability tenant.
   3. Client ID: The client ID that you obtain after generating the Service Principal in your Cisco Cloud Observability tenant.
   4. Client Secret: The secret key that you obtain after generating the Service Principal in your Cisco Cloud Observability tenant.
   5. Alert Severity: Severity of the alert that you want to define— Critical, Major, Minor, or Info.

Search Expression

On your Splunk Enterprise, use the following attributes to create a search expression to look for the events:

Splunk Data Validation

After installing the Splunk add-on, you need to validate the alerts and events that are ingested into Cisco Observability Platform.  You can validate the data by using either Query Builder or fsoc.

Data Validation using Query Builder 

You can validate the data by using Query Builder. Open Query Builder and enter the following query:

SINCE-1d fetch events("aiopscore:splunk_event") {timestamp, attributes, raw}

Data Validation using FSO CLI (FSOC) 

This section describes the specifications required to install Cisco Observability Platform CLI for executing the fsoc queries to validate the metrics and events.

Pre-requisites

Ensure that you meet the following requirements:

1. You have installed fsoc on your device. See Install and Configure FSOC.
2. You have configured a profile in fsoc.

To learn about UQL queries, see the Unified Query Language User Guide.

Validation Steps

Do the following:

1. Run the following command to setup the fsoc profile by using the service principals provided by your Company administrator:

   $sudo fsoc config set --profile <name-of-your-profile> --secret-file <name-of-secret-file>

   CODE

   Example:

   $sudo fsoc config set --profile demo --secret-file splunk-access.json

   CODE

2. Run the following command to use the profile:

   $sudo fsoc config use --profile <name-of-your-profile>

   CODE

   Example:

   $sudo fsoc config use --profile demo

   CODE

3. Run the following command to check the subscription status of your aiopsservices solution:

   $sudo fsoc solution status aiopsservices

   CODE

   Sample response:

   ✓ OAuth token refresh
   ✓ Platform API call, retry after login (GET /objstore/v1beta/objects/extensibility:solutionRelease?order=desc&filter=data.solutionName+eq+%22aiopsservices%22&max=1)
   ✓ Platform API call (GET /objstore/v1beta/objects/extensibility:solutionInstall?order=desc&filter=data.solutionName+eq+%22aiopsservices%22&max=1)
   ✓ Platform API call (GET /objstore/v1beta/objects/environment:subscription?order=desc&filter=data.solutionName+eq+%22aiopsservices%22+and+data.tenantId+eq+%22abd98ae8-a58d-4572-a807-9ab4fa98f464%22)
                  Solution Name: aiopsservices
   Solution Subscription Status: Subscribed
        Solution Upload Version: 1.0.378
               Upload Timestamp: 2024-02-20T15:58:33.034Z
       Solution Install Version: 1.0.378
   Solution Install Successful?: true
          Solution Install Time: 2024-02-20T15:58:34.824Z
       Solution Install Message:

   CODE

4. Run the following UQL queries to validate the Splunk data:

   $sudo fsoc uql "SINCE-1d fetch events("aiopscore:splunk_event") {timestamp, attributes}"

   CODE

   Sample response:

   ✓ Platform API call (POST /monitoring/v1/query/execute)
    events                                                                                                                    
    timestamp                         | attributes                                                                            
                                      | name                      | value                                                     
   ===========================================================================================================================
    2024-02-27 03:15:04.874 +0000 UTC | aiopscore.ci.type         | switch                                                    
                                      | aiopscore.splunk.event_id | 9b355e1b-c849-396e-a2ac-95b0abc06f60                      
                                      | appd.event.type           | aiopscore:splunk_event                                    
                                      | aiopscore.splunk.status   | Active                                                    
                                      | appd.isevent              | true                                                      
                                      | aiopscore.splunk.message  | aiops-switch-1. Switch link down                          
                                      | aiopscore.splunk.priority | 1                                                         
                                      | _parsing_status           | false                                                     
                                      | aiopscore.ci.name         | aiops-switch-1                                            
                                      | aiopscore.splunk.severity | CRITICAL                                                  
                                      | parsing_error_or_warnings | Missing mandatory parsing attribute _message_parser.type. 
                                      | telemetry.sdk.name        | aiops-fsoc-exporter                                       
                                      | aiopscore.ci.id           | aiops-switch-1                                            
   -----------------------------------+---------------------------+-----------------------------------------------------------
    2024-02-27 03:15:04.503 +0000 UTC | aiopscore.ci.type         | vm                                                        
                                      | aiopscore.splunk.event_id | 9b355e1b-c849-396e-a2ac-95b0abc06f59                      
                                      | appd.event.type           | aiopscore:splunk_event                                    
                                      | aiopscore.splunk.status   | Active                                                    
                                      | appd.isevent              | true                                                      
                                      | aiopscore.splunk.message  | aiops-vm-1 host not responding                            
                                      | aiopscore.splunk.priority | 2                                                         
                                      | _parsing_status           | false                                                     
                                      | aiopscore.ci.name         | aiops-vm-1                                                
                                      | aiopscore.splunk.severity | MAJOR                                                     
                                      | parsing_error_or_warnings | Missing mandatory parsing attribute _message_parser.type. 
                                      | telemetry.sdk.name        | aiops-fsoc-exporter                                       
                                      | aiopscore.ci.id           | aiops-vm-1                                                
   -----------------------------------+---------------------------+-----------------------------------------------------------

   CODE

Troubleshoot Splunk Issues

As a Splunk administrator, you can check for the issues encountered while sending Splunk alerts to Cisco AIOps. On your Splunk Enterprise, check for the issues in one of the following ways:

• View the issues in the log file: $SPLUNK_HOME/var/log/TA-Splunk-AIOps/AIOps_Integration.log
• In the Search tab, enter the following query to view the issues: 

  index=_internal sourcetype=splunkd component=sendmodalert action="AIOps_Integration"

• In the Alert Actions list, click View log events for AIOps Alert Integration to view the issues.

View Splunk Alerts

To view the Splunk alerts in Cisco AIOps:

1. Go to the Event Explorer.
2. In the Event Source list, select aiopscore.
3. In the Event Type list, select aiopscore:splunk_event.
4. Click Submit.
5. Click on each event to view the attributes and their values.

Uninstall the Splunk add-on

To stop sending the Splunk alerts to Cisco AIOps, you need to uninstall AIOps Add-on for Splunk on your Splunk Enterprise. For information on how to uninstall a Splunk add-on, see Splunk documentation.

Splunk^® is a trademark of Splunk Inc. in the United States and other countries. All other brand names, product names, or trademarks belong to their respective owners. © 2023 Splunk Inc. All rights reserved.

This document contains references to third-party documentation. Splunk AppDynamics does not own any rights and assumes no responsibility for the accuracy or completeness of such third-party documentation.