With the standard deployment option, Splunk AppDynamics On-Premises Virtual Appliance installs infrastructure and Splunk AppDynamics Services in your Kubernetes cluster.

Prepare to Install Splunk AppDynamics Services

Complete the following steps to prepare the environment:

1. Log in to one of the node console using the appduser credentials.

2. Navigate to the following folder:

   cd /var/appd/config

   CODE

3. Edit the globals.yaml.gotmpl file with the required configuration.

   vi globals.yaml.gotmpl

   CODE
   1. (Optional) Configure a custom ingress certificate (by default the ingress controller installs a fully-configured self-signed certificate). The custom ingress certificate needs certain SANs added to it. Run the following script on the console of your primary node to view a list of SANs. Add those SANs to your custom ingress certificate. See ingress in Customize the Helm File for instructions on how to configure the custom ingress certificate and key. 

      #!/bin/bash
      set -euo pipefail
      TENANT=$(helm secrets decrypt /var/appd/config/secrets.yaml.encrypted  | yq .hybrid.controller.tenantAccountName)
      DNS_DOMAIN=$(grep -v "^ *\t* *{{" /var/appd/config/globals.yaml.gotmpl | yq -r '.dnsDomain')
      DNS_NAMES=$(grep -v "^ *\t* *{{" /var/appd/config/globals.yaml.gotmpl | yq -r '.dnsNames|join(" ")')
      echo Verify the Virtual Appliance tenant should be \'${TENANT}\'
      echo Verify the Virtual Appliance domain name should be \'${DNS_DOMAIN}\'
      echo Verify the Virtual Appliance node names are: ${DNS_NAMES}
      
      echo If creating and importing into VA a Custom Ingress Certificate, include the following SANs:
      for server_name in "$DNS_DOMAIN" "${TENANT}.${DNS_DOMAIN}" "*.${DNS_DOMAIN}" "${TENANT}.auth.${DNS_DOMAIN}" "${TENANT}-tnt-authn.${DNS_DOMAIN}" $DNS_NAMES; do
              echo "  ${server_name}"
      done

      BASH
      
      
      Sample output of the script above: 

      Verify the Virtual Appliance tenant should be 'customer1'
      Verify the Virtual Appliance domain name should be 'va.mycompany.com'
      Verify the Virtual Appliance node names are: localhost vanodename-1
      If creating and importing into VA a Custom Ingress Certificate, include the following SANs:
        va.mycompany.com
        customer1.va.mycompany.com
        *.va.mycompany.com
        customer1.auth.va.mycompany.com
        customer1-tnt-authn.va.mycompany.com
        localhost
        vanodename-1

      BASH
      
       
   2. (Optional) Add any custom CA certificates for Controller outbound traffic by configuring  appdController.customCaCerts in Customize the Helm File. 
   3. (Optional) Disable the self-monitoring for the Controller.
      

      enableClusterAgent: false

      CODE

4. (Optional) Edit the /var/appd/config/secrets.yaml file to update usernames and passwords of theSplunk AppDynamics Services.

   vi /var/appd/config/secrets.yaml

   CODE
   

   When you install the Splunk AppDynamics service, the secrets.yaml file becomes encrypted.

   See Edit the secrets.yaml.encrypted file.

5. Save the following script to the console of your primary virtual appliance node as dnsinfo.sh and run it. Follow the instructions in its output: 

   #!/bin/bash
   set -euo pipefail
   TENANT=$(helm secrets decrypt /var/appd/config/secrets.yaml.encrypted  | yq .hybrid.controller.tenantAccountName)
   DNS_DOMAIN=$(grep -v "^ *\t* *{{" /var/appd/config/globals.yaml.gotmpl | yq -r '.dnsDomain')
   
   echo Verify the Virtual Appliance tenant should be \'${TENANT}\'
   echo Verify the Virtual Appliance domain name should be \'${DNS_DOMAIN}\'
   
   for server_name in "${TENANT}.auth.${DNS_DOMAIN}" "${TENANT}-tnt-authn.${DNS_DOMAIN}"; do
     if ! getent hosts "${server_name}" > /dev/null; then
       echo "Please double-check that DNS can resolve '${server_name}' as the VA ingress IP"
     fi
   done 

   BASH
   
   Sample output: 

   Verify the Virtual Appliance tenant should be 'customer1'
   Verify the Virtual Appliance domain name should be 'va.mycompany.com'
   Please double-check that DNS can resolve 'customer1.auth.va.mycompany.com' as the VA ingress IP
   Please double-check that DNS can resolve 'customer1-tnt-authn.va.mycompany.com' as the VA ingress IP 

   BASH
6. Copy the license files as the license.lic file to the node in the following location.
   

   cd /var/appd/config

   CODE
   
   This license will be automatically used to provision Splunk AppDynamics Services.
   If you do not have the license file at this time, you can apply the license and provision the services later using appdcli.
   
   

   For End User Monitoring, if you are using the Infrastructure-based Licensing model, make sure to specify EUM account and license key in the Administration Console. See Access the Administration Console.

Create a Three-Node Cluster

1. Log in to the primary node console.
2. Verify the boot status of each node of the cluster:
   

   appdctl show boot

   CODE
   

   • Ensure the status of the services in each node appears as Success. Else, restart the virtual machine that failed.
     If it is still failing, you might have to redeploy the virtual machine.
   • Ensure to configure the same time on all the cluster nodes.

3. Run the following command in the primary node and specify the IP address of the peer nodes:

   cd /home/appduser
   appdctl cluster init <Node-2-IP> <Node-3-IP>

   CODE

4. Run the following command to verify the node status:

   appdctl show cluster
   microk8s status

   CODE

   Ensure that the output displays the Running status as true for the nodes that are part of the cluster.

   Sample Output

    NODE           | ROLE  | RUNNING 
   ----------------+-------+---------
    10.0.0.1:19001 | voter | true    
    10.0.0.2:19001 | voter | true    
    10.0.0.3:19001 | voter | true 

   CODE
   

   You must re-login to the terminal if the following error appears:

   Insufficient Permissions to Access Microk8s 

   CODE

Install Services in the Cluster 

1. Log in to the cluster node console.

2. Run the command to install services:

   appdcli start appd [Profile]

   CODE
   appdcli start appd small

   CODE

   
   

   appdcli start appd medium

   CODE

   
   

   
   This command installs the Splunk AppDynamics services. We recommend you to specify the VA profile as same as the profile that you selected to create a virtual machine. See, Virtual Appliance Sizing.

   Sample Output

   NAME               CHART                     VERSION   DURATION
   cert-manager-ext   charts/cert-manager-ext   0.0.1           0s
   ingress-nginx      charts/ingress-nginx      4.8.3           1s
   redis-ext          charts/redis-ext          0.0.1           1s
   ingress            charts/ingress            0.0.1           2s
   cluster            charts/cluster            0.0.1           2s
   reflector          charts/reflector          7.1.216         2s
   monitoring-ext     charts/monitoring-ext     0.0.1           2s
   minio-ext          charts/minio-ext          0.0.1           2s
   eum                charts/eum                0.0.1           2s
   fluent-bit         charts/fluent-bit         0.39.0          2s
   postgres           charts/postgres           0.0.1           2s
   mysql              charts/mysql              0.0.1           3s
   redis              charts/redis              18.1.6          3s
   controller         charts/controller         0.0.1           3s
   events             charts/events             0.0.1           4s
   cluster-agent      charts/cluster-agent      1.16.37         4s
   kafka              charts/kafka              0.0.1           6s
   minio              charts/minio              5.0.14         47s

   CODE
3. Verify the status of the installed pods and service endpoints:
   • Pods: kubectl get pods --all-namespaces

   • Service endpoints: appdcli ping


     +---------------------+---------+
     |  Service Endpoint   | Status  |
     +=====================+=========+
     | Controller          | Success |
     +---------------------+---------+
     | Events              | Success |
     +---------------------+---------+
     | EUM Collector       | Success |
     +---------------------+---------+
     | EUM Aggregator      | Success |
     +---------------------+---------+
     | EUM Screenshot      | Success |
     +---------------------+---------+
     | Synthetic Shepherd  | Success |
     +---------------------+---------+
     | Synthetic Scheduler | Success |
     +---------------------+---------+
     | Synthetic Feeder    | Success |
     +---------------------+---------+
     | AD/RCA Services     | Failed  |
     +---------------------+---------+

     CODE

• When a Virtual Machine restarts, the MySQL service may not automatically restore. To troubleshoot this issue, see Restore the MySQL Service.
• If the EUM pod is failing even after multiple retries, see EUM Health is Failing After Multiple Retries.

Install the Anomaly Detection Services in the Cluster

1. Log in to the cluster node console.

2. Run the command to install services:

   appdcli start aiops small

   CODE

   appdcli start aiops medium

   CODE
3. Verify the status of the installed pods and service endpoints:
   • Pods: kubectl get pods -n cisco-aiops


   • Service endpoints: appdcli ping
     The status of the Anomaly Detection service appears as Success.

See Anomaly Detection.

Sometimes, IOException error occurs when you access Anomaly Detection in the Controller UI. See Troubleshoot Virtual Appliance Issues.

(Optional) Install Cisco Secure Application 

See Install Cisco Secure Application.

Apply Licenses to Splunk AppDynamics Services

Use appdcli to apply licenses after installingSplunk AppDynamics Services.

1. Log in to the cluster node console.
2. Copy the license files as the license.lic file to the node in the following location.
   

   cd /var/appd/config

   CODE

3. Run the following commands to apply licenses:

   Update the Controller license.

   appdcli license controller license.lic

   CODE

   1. Update the EUM license.

      appdcli license eum license.lic

      CODE
   2. (Optional) If you are using the Infrastructure-based Licensing model, make sure to specify EUM account and license key in the Administration Console. See Access the Administration Console.

For more information, see Virtual Appliance CLI.

Verify the Service Endpoints Paths

The Ingress controller checks the URL of an incoming request and redirects to the respectiveSplunk AppDynamics Service.

Log in to the Controller UI by accessing https://<DNS-Name>or<Cluster-Node-IP>/.

By default, the Controller UI username is set to admin and the password is set to welcome.

Download Splunk AppDynamics Agents

Splunk AppDynamics On-Premises Virtual Appliance supports you to install the AppDynamics agents. Download the agents from Download Portal.

For more information, see:

• Install App Server Agents
• End User Monitoring