Install Cisco AppDynamics Services in the Hybrid Deployment

With the hybrid deployment option, Splunk AppDynamics On-Premises Virtual Appliance leverages your current Controller, Events Service, and End User Monitoring components from Splunk AppDynamics On-Premises and installs Anomaly Detection and Cisco Secure Application Services in your Kubernetes cluster.

Prepare to Install Splunk AppDynamics Services

Complete the following steps to prepare the environment:

Log in to one of the node console using the appduser credentials.
This node is considered as the primary node after you create a three-node cluster.
Navigate to the following folder:
```
cd /var/appd/config
```
CODE

Edit the globals.yaml.gotmpl file with the required configuration.

vi globals.yaml.gotmpl

CODE

Specify the Controller key and the account name.
Enable the hybrid mode.
Set the Controller domain name.
Configure the port to access the standalone controller.
Set the sslEnabled field to true.
Upload the Controller CA certificates for the standalone Controller.
Configure the MySQL host. It is the same host as the Controller domain.
Configure a port for the standalone Controller database.
Update the MySQL CA certificates.

(Optional) Update CA certificates for Kafka.

# License details
license:
  controllerKey: &controllerKey  6946ecaa-8072-4825-ae9d-c68ba992f4bf
 
# Controller details
controller:
  tenantAccountName: &account customer1

# Hybrid mode
# domainName and dbHost/ dbPort is for standalone controller
hybrid:
  enable: true
  controller:
    domainName: controller.nip.io
    port: 8181
    sslEnabled: true
{{ if isFile "/var/appd/config/hybrid-controller-ca.crt" }}
    controllerCaCertsFile: {{ readFile "/var/appd/config/hybrid-controller-ca.crt" | b64enc | quote }}
{{ end }}
  mysql:
    dbHost: controller.nip.io
    dbPort: 3388
{{ if isFile "/var/appd/config/hybrid-mysql-ca.crt" }}
    mysqlCaCertsFile: {{ readFile "/var/appd/config/hybrid-mysql-ca.crt" | b64enc | quote }}
{{ end }}

kafka:
    kafkaHost0: <node1_ip>
    kafkaPort0: 32101
    kafkaHost1: <node2_ip>
    kafkaPort1: 32102
    kafkaHost2: <node3_ip>
    kafkaPort2: 32103
    defaultCert: true
{{ if isFile "/var/appd/config/hybrid-kafka.key" }}
    keyFile: {{ readFile "/var/appd/config/hybrid-kafka.key" | b64enc | quote }}
{{ end }}
{{ if isFile "/var/appd/config/hybrid-kafka.crt" }}
    certFile: {{ readFile "/var/appd/config/hybrid-kafka.crt" | b64enc | quote }}
{{ end }}

YML

(Optional) Update CA certificates for the Ingress controller.

By default, the Ingress controller installs the self-signed certificates.

CODE

The Ingress controller CA certificates are required for the Kubernetes cluster. See Edit the globals.yaml.gotmpl file.

ingress:
  defaultCert: true
{{ if isFile "server.key" }}
  keyFile: {{ readFile "server.key" | b64enc | quote }}
{{ end }}
{{ if isFile "server.crt" }}
  certFile: {{ readFile "server.crt" | b64enc | quote }}
{{ end }}

CODE

Edit the secret.yaml file to update the following:
```
vi secrets.yaml
```
CODE
1. Update the Controller and MySQL credentials.
```
 hybrid: 
  controller:
    rootPassword: welcome
    adminUsername: admin
    adminPassword: welcome
   mysql:
    dbUser: dbuser
    dbPassword: changeit
```
  YML
2. Edit the usernames and passwords of the Splunk AppDynamics Services.
  
  When you install the Splunk AppDynamics service, the secrets.yaml file becomes encrypted.
  
  See Edit the secrets.yaml.encrypted File.
Update the firewall rules to allow the standalone Controller to access the Kafka ports in the Kubernetes cluster. Update the firewall rule in each cluster node.
```
sudo ufw allow AppdNodePorts
```
CODE

Create a Three-Node Cluster

Log in to the primary node console.
Verify the boot status of each node of the cluster:
```
appdctl show boot
```
CODE
- Ensure the status of the services in each node appears as Success. Else, restart the virtual machine that failed.
  If it is still failing, you might have to redeploy the virtual machine.
- Ensure to configure the same time on all the cluster nodes.
Run the following command in the primary node and specify the IP address of the peer nodes:
```
cd /home/appduser
appdctl cluster init <Node-2-IP> <Node-3-IP>
```
CODE

Run the following command to verify the node status:

appdctl show cluster
microk8s status

CODE

Ensure that the output displays the Running status as true for the nodes that are part of the cluster.

Sample Output

 NODE           | ROLE  | RUNNING 
----------------+-------+---------
 10.0.0.1:19001 | voter | true    
 10.0.0.2:19001 | voter | true    
 10.0.0.3:19001 | voter | true

CODE

You must re-login to the terminal if the following error appears:

Insufficient Permissions to Access Microk8s

CODE

Configure the Standalone Controller in Splunk AppDynamics On-Premises

Log in to the Controller database by using the following path:
```
<controller-home>/bin/controller.sh login-db
```
CODE
Connect to the MySQL database.

Drop and create the secapp user for Cisco Secure Application Services and grant permissions to access the Controller database:

DROP USER secapp;
CREATE USER 'secapp'@'%' identified by 'changeit' REQUIRE SSL;
GRANT USAGE ON *.* TO `secapp`@`%`;
GRANT SELECT  ON `controller`.* TO `secapp`@`%`;

CODE

In Enterprise Console, add the authentication settings to the controller configurations as follows:
1. Access the Enterprise Console UI.
2. Select the platform.
3. Go to the Configuration tab.
4. Navigate to Controller Settings > Appserver Configurations > JVM Options.
5. Replace <DNS_DOMAIN> with the value configured as dnsDomain in /var/appd/config/globals.yaml.gotmpl.
```
-Dappdynamics.auth.service.hosted.zone=-tnt-authn.<DNS_DOMAIN>
-Dappdynamics.auth.service.port=443
-Dappdynamics.auth.service.ssl.access.enabled=true
-Dappdynamics.auth.service.token.generation.enabled=true
-Dappdynamics.controller.authentication.resource.accessible=true
-Dappdynamics.auth.client.token.validation.enabled=true
```
  CODE
  Alternatively, add these authentication settings in ~/appdynamics/platform/product/controller/appserver/jetty/start.d/start.ini

Install Services in the Cluster

Log in to the cluster node console.

Run the command to install services:

appdcli start appd small

CODE

appdcli start appd medium

CODE

This command installs the Splunk AppDynamics services. We recommend you to specify the VA profile as same as the profile that you selected to create a virtual machine. See, Virtual Appliance Sizing.

Sample Output

NAME               CHART                     VERSION   DURATION
cert-manager-ext   charts/cert-manager-ext   0.0.1           0s
ingress-nginx      charts/ingress-nginx      4.8.3           1s
redis-ext          charts/redis-ext          0.0.1           1s
ingress            charts/ingress            0.0.1           2s
cluster            charts/cluster            0.0.1           2s
reflector          charts/reflector          7.1.216         2s
monitoring-ext     charts/monitoring-ext     0.0.1           2s
minio-ext          charts/minio-ext          0.0.1           2s
eum                charts/eum                0.0.1           2s
fluent-bit         charts/fluent-bit         0.39.0          2s
postgres           charts/postgres           0.0.1           2s
mysql              charts/mysql              0.0.1           3s
redis              charts/redis              18.1.6          3s
controller         charts/controller         0.0.1           3s
events             charts/events             0.0.1           4s
cluster-agent      charts/cluster-agent      1.16.37         4s
kafka              charts/kafka              0.0.1           6s
minio              charts/minio              5.0.14         47s

CODE

Verify the status of the installed pods and service endpoints:
- Pods: kubectl get pods --all-namespaces

When a Virtual Machine restarts, the MySQL service may not automatically restore. To troubleshoot this issue, see Restore the MySQL Service.
If the EUM Collector, EUM Aggregator, and EUM Screenshot pods are failing even after multiple retries, see EUM Health is Failing After Multiple Retries.

Generate the Hybrid Configuration File

Run the script to connect the Kafka node ports with the standalone controller:

appduser@appd-demo-2:~/appd-charts/utils$ bash prepare-hybrid.sh

CODE

Sample Output

Writing Kafka CA Cert
Writing Schema registry CA Cert
Writing AnomalyDetectionKafkaSSLProducer
Writing AnomalyDetectionKafkaConsumerSSL
Writing configure script
Creating archive of hybrid config
hybrid-config/
hybrid-config/kafka-ca.crt
hybrid-config/AnomalyDetectionKafkaConsumerSSL
hybrid-config/configure.sh
hybrid-config/schema-registry-ca.crt
hybrid-config/AnomalyDetectionKafkaSSLProducer
Completed

CODE

This script generates the hybrid-config.tar file that includes the required configuration files.

Copy the hybrid-config.tar file to your standalone controller instance.
Locate and extract the hybrid-config.tar file contents.

Configure the Controller Properties by Using the Kubernetes Cluster CLI

Log in to your Kubernetes Cluster.

Run the command to update the standalone Controller properties:

appdcli run bootstrap_hybrid

CODE

This command configures the following properties in the Controller:

Key: anomaly.detection.enabled, Value: true
Key: anomaly.detection.deployed.onprem, Value: true
Key: controller.id, Value: onprem
Key: controller.unique.id, Value: onprem
Key: anomaly.detection.kafka.consumer.group.id, Value: pi-alarmservice-transition-consumer-group-onprem
Key: anomaly.detection.kafka.consumer.topic, Value: pi-alarmservice-transition-onprem
Key: anomaly.detection.kafka.producer.topic, Value: metadata-lifecycle-events
Key: anomaly.detection.onprem.output.topic, Value: blitz-processed-metrics
Key: anomaly.detection.schema.registry.enabled, Value: true
Key: anomaly.detection.consumer.schema.registry.url, Value: https://10.0.0.1.nip.io/schemaregistry
Key: anomaly.detection.producer.schema.registry.url, Value: https://10.0.0.2.nip.io/schemaregistry
Key: anomaly.detection.kafka.consumer.bootstrap.servers, Value: 10.0.0.1:32101
Key: anomaly.detection.kafka.producer.bootstrap.servers, Value: 10.0.0.2:32102
Key: anomaly.detection.kafka.producer.ssl.enabled, Value: true
Key: anomaly.detection.kafka.consumer.ssl.enabled, Value: true
Key: anomaly.detection.service.url, Value: https://10.0.0.1.nip.io/pi

CODE

Alternatively, you can access the administration console by using the Kubernetes cluster IP address and manually edit each parameter.

Run the configure.sh script to provision the controller with certs and secrets to connect to Kubernetes cluster:
This script is extracted from the hybrid-config.tar file. See, how to obtain hybrid-config.tar file.

$ bash configure.sh
Usage: configure.sh <controller-home> <keystore-passwd>
Using controller home as /opt/appdynamics and keystore password as changeit
Do you want to proceed? (y/n): y
Creating Kafka truststore
Certificate was added to keystore
Creating schema registry truststore
Certificate was added to keystore
Writing obfuscated producer config
Writing obfuscated consumer config
Completed

CODE

Restart the Controller using these commands:

cd ~/appdynamics/platform/product/controller/bin/
./controller.sh stop
./controller.sh start

CODE

Install the Anomaly Detection Services in the Cluster

Log in to the cluster node console.

Run the command to install services:

appdcli start aiops small

CODE

appdcli start aiops medium

CODE

Verify the status of the installed pods and service endpoints:
- Pods: kubectl get pods --all-namespaces
- Service endpoints: appdcli ping

See Anomaly Detection.

Sometimes, IOException error occurs when you access Anomaly Detection in the Controller UI. See Troubleshoot Virtual Appliance Issues.

Enable the Cisco Secure Application Service

Complete these steps in the Administration Console to enable the Cisco Secure Application service:

Log in to the Administration Console:

https://<controller-hostname>/controller/admin.jsp

CODE

Edit your account to add the property:
```
argento.enabled = true 
```
CODE
Log out of Administration Console, then log into the Controller:
```
https://<controller-hostname>/controller/
```
CODE
Create a role with the following permissions:
- View Cisco Secure Application
- Configure Cisco Secure Application
Assign the role to the administrator user.

Install and Set Up Cisco Secure Application Services

Ensure that Domain Name System (DNS) is configured. See Configure Domain Name System.
Review the requirements to Create a Virtual Appliance Ingress Certificate.
Log in to the cluster node console.
Run the command to install the Cisco Secure Application service:
```
appdcli start secapp small
```
CODE
```
appdcli start secapp medium
```
CODE
Verify the status of the installed pods with the command:
```
kubectl get pods -n cisco-secureapp 
```
CODE

Re-run the command:

appdcli start appd small

CODE

appdcli start secapp medium

CODE

Add an extra Subject Alternative Name (SAN) entry for DNS matching. See Controller SSL and Certificates.

Cisco Secure Application References

Follow the steps to configure Cisco Secure Application:

	Step	Reference
1	For the .NET and Java Agent, you must add node property: `enable-secapp-service` CODE For the Java Agent, you must be on version >= 24.4.1. For the .NET Agent, you must be on version >= 24.4.0.1.	Install the Java Agent Install the .NET Agent for Linux
2	Extract the SSL certificate for use with the agents. Log in to the cluster node and run the command: `kubectl get secret ingress-cert-secret -n ingress-master -o jsonpath="{.data.tls\.crt}" \| base64 --decode > certificate.crt` CODE If the cluster is using non default self signed certificate, then copy the existing certificate in the global config location. If `defaultCert` is `false`, then run the command: `kubectl get secret custom-ingress-secret -n ingress-master -o jsonpath="{.data.tls\.crt}" \| base64 --decode > certificate.crt` CODE See globals.yaml.gotmpl file. Copy this certificate for use with the agent.	Enable SSL for the Java Agent Configuring SSL Variables for .NET Agents
3	Assign roles using the Splunk AppDynamics Administration Console. Assign the Configure Cisco Secure Application account permission to the users who are required to modify configurable fields on the Cisco Secure Application dashboard. Assign View Cisco Secure Application account permissions to users who are required to only monitor the dashboard.	Account Permissions
4	Click on the Security tab in the top navigation bar. Launch the required Splunk AppDynamics Application dashboard using your account, and then click Security on the top pane. This redirects you to the Cisco Secure Application dashboard.	Monitor Application Security Using Cisco Secure Application
5	From the Cisco Secure Application Dashboard navigate to the Applications page, and then set Security Setting as Enabled for the target application. The Security Setting value is set to Inherit by default for all applications that inherit the non-configurable tenant setting of Disabled. To enable security for an application, you must set Security Setting to Enabled.	Monitor Security Status of Applications
6	From the Applicationspage, verify that the application nodes are registered and active. From the Applications page, check the Active Nodes and Registered Nodes fields for the specific application. Ensure that the application nodes are active. If the nodes are not active, then the application security data is not displayed on the dashboard.	Monitor Security Status of Applications
7	From the Libraries page view the risk-sorted libraries of secured applications. The Libraries page displays all the existing libraries of application(s) based on the selected application scope. You can use the risk score to prioritize the remediation task.	Monitor Libraries

For more information, see Getting Started with Cisco Secure Application.

Download Splunk AppDynamics Agents

Splunk AppDynamics On-Premises Virtual Appliance supports you to install the Splunk AppDynamics agents. Download the agents from Download Portal.

For more information, see: