Administer the Events Service

This page describes how to manage Events Service with the Enterprise Console. All of these tasks can be performed on the GUI or CLI.

Start and Stop the Events Service

The Events Service is an internal data storage engine used by the Database Visibility module. To use Database Monitoring, you need to start the Events Service.

The Events Service is automatically started after you install it.

On Linux

Start the Events Service on Linux by running this command:

bin/platform-admin.sh submit-job --platform-name <platform_name> --service events-service --job start

Stop the Events Service by running this command: 

bin/platform-admin.sh submit-job --platform-name <platform_name> --service events-service --job stop

On Windows

Start the Events Service on Windows by running this command:

bin/platform-admin.exe cli submit-job --platform-name <platform_name> --service events-service --job start

Stop the Events Service by running this command:

bin/platform-admin.exe cli submit-job --platform-name <platform_name> --service events-service --job stop

Monitor Cluster Node Health

It is important to carefully monitor the health of the Events Service cluster, for a new deployment, especially to monitor disk consumption.

Check the status of the Events Service cluster from:

• Enterprise Console UI, navigate to Platforms > <name of the platform> > Events Service.
  If there are any issues with the health of the cluster, the status is displayed as critical with red color. Green color represents a normal status. To view which nodes are not active, check the Running State column against each Host.

• Enterprise Console server, run the following command:

  bin/platform-admin.sh show-events-service-health  

  CODE

  The output shows possible issues and the steps you need to take to resolve them. For example, if the available disk is low, the resolution is to add nodes to the cluster.

The following are the potential errors and remediation steps: 

Expand the Cluster

You can use the Enterprise Console GUI or CLI to horizontally scale the Events Service cluster on Linux. To grow your existing deployment to contend with an increased workload, simply add nodes. 

Before starting, prepare the new cluster machine. Verify system requirements and prepare the environment as described in Events Service Requirements. 

It is important for any new machine in the cluster to have the same SSH-enabled user account as existing cluster members. 

Once you have prepared the system, run the command for adding nodes:

bin/platform-admin.sh add-events-service-nodes --hosts host1  host2  host3   

Alternatively, pass the hostnames of the new node as a file.  

bin/platform-admin.sh add-events-service-nodes --host-file=/home/appduser/hosts.txt 

The file you pass to the command (hosts.txt in the example) should contain the internal DNS hostnames or IP addresses of the nodes to add. It does not need to list existing nodes in the cluster. These hosts should be part of the platform. For more information about how to add a host to the platform, see Administer the Enterprise Console.

Be sure to modify your load balancer rules to include the new cluster member in its routing rules. See Load Balance Events Service Traffic for more information. 

Restart the Node

You can use the Enterprise Console GUI or CLI to restart your node.

Once you have prepared the system, run the command for restarting your node:

bin/platform-admin.sh submit-job --platform-name <platform_name> --service events-service --job restart-node --args nodeActionHost=<node_name>

where <node_name> is the hostname of the node from the command:

bin/platform-admin.sh list-nodes --platform-name <platform_name> --service events-service

Restart the Cluster

You can use the Enterprise Console GUI or CLI to restart your cluster.

Once you have prepared the system, run the command for restarting your cluster nodes:

bin/platform-admin.sh submit-job --platform-name <platform_name> --service events-service -job restart-cluster

Remove or Replace a Node

The uninstall-events-service command removes the Events Service software and data from all cluster nodes. The Controller and Events Service share a database, so if you are uninstalling the Controller instance under which you ran the Enterprise Console to install the Events Service, you need to uninstall the Events Service with this command before you uninstall the Controller.   

The remove-events-service-node command removes the Events Service software and data from a single node that you specify by hostname. You should only use this command if you have at least four nodes in your cluster. Removing an Events Service node from a three-node cluster is not supported. Identify the node to remove using the --node command line parameter. 

After you remove a node, be sure to adjust your load balancer rules to remove the old cluster member. See Load Balance Events Service Traffic for more information. 

If you are not using a load balancer with a cluster deployment, keep in mind that the connection settings for the first primary node that reports to the Controller at installation time are written to the Controller setting that identifies the Events Service to the Controller. If you remove a primary node in that case, check whether the removed primary node is node identified as the Events Service destination URL in the Controller connection settings (e.g., appdynamics.on.premise.event.service.url) and adjust the setting if so. See Connect to the Events Service for more information.

Note the following guidelines: 

• You cannot remove nodes such that the resulting cluster size is two 
• A cluster that consists of three or more nodes can't be reduced in size to a single node Events Service. 

The remove-events-service-node command removes the Events Service software and data from a single node that you specify by hostname. You should only use this command if you have at least four nodes in your cluster. Removing an Events Service node from a three-node cluster is not supported. Identify the node to remove using the --node command line parameter. 

This command removes the node specified in the argument:

bin/platform-admin.sh remove-events-service-node  --node 10.0.100.51

If you attempt to remove a primary node using the command shown above, the Enterprise Console notifies you that you are attempting to remove a primary node and cancels the operation. As indicated in the output, you can proceed to remove the primary node by rerunning the command with the -f force flag. When you remove a primary node, the cluster elects a new primary node from the existing data nodes. The election process may take a few seconds, during which new events cannot be processed. Be sure to perform this operation at a time when the impact of a brief interruption of service will be minimal. 

If you have an unreachable node you would like to remove, but cannot due to the above restrictions, you can choose to replace it instead.

This command replaces the old node specified in the argument with the new node:

bin/platform-admin.sh submit-job --service events-service --job replace-node --args originalNode=<old_host_address> newNode=<new_host_address>

• After removing the Events Service nodes from the cluster, you may observe that the value appdynamics.es.eum.key changed in the Controller admin.jsp and in the Events Service properties file, but not in the EUM properties file, analytics.accountAccessKey. 
• Check if the key value changed in the Controller and the Events Service, then replace the key value with the EUM properties file: analytics.accountAccessKey.

Enable the Events Service to Use SSL

You can use the Enterprise Console CLI to enable the Events Service to use SSL. You will need a KeyStore file in JKS format, the password and the alias for the KeyStore. See Create a Certificate and Generate a CSR for detailed instructions for creating the Keystore. 

1. Log in to the Enterprise Console:

   bin/platform-admin.sh login --user-name <admin_username> --password <admin_password>

   TEXT

2. After successfully logging in, submit an enable-ssl job for the Events Service, providing the path to the KeyStore file, the KeyStore password, and KeyStore alias.

   bin/platform-admin.sh submit-job --platform-name <platform_name> --service events-service --job enable-ssl --args keystorePath=<path-to-keystore-jks-file> keystorePassword=<keystore_password>  keystoreAlias=<keystore_alias>

   TEXT

3. Confirm that SSL has been enabled: 

    curl -k -v -X GET https://<events-service-domain>:9080/_ping

   TEXT

4. The output of the cURL command should show the TLS handshakes and the HTTP status 200:

   ...
   * TLSv1.2 (OUT), TLS handshake, Client hello (1):
   * TLSv1.2 (IN), TLS handshake, Server hello (2):
   ...
   < HTTP/1.1 200 OK
   < Date: Fri, 10 May 2019 00:13:49 GMT
   < X-Content-Security-Policy: default-src 'self'
   ...

   TEXT

Collect Events Service Logs

The Enterprise Console can collect logs from the nodes in the cluster. The following command retrieves node logs and bundles them, along with the Enterprise Console's own logs:

bin/platform-admin.sh retrieve-events-service-logs

When the command is finished, a ZIP file named events-service.log.zip is created in the location from which you ran the script. You can then extract the archive to troubleshoot or submit the archive for troubleshooting assistance to your AppDynamics representative. If the Enterprise Console failed to connect to one of the cluster nodes to retrieve logs for any reason, the connection error is written to a log file included in the archive.  

Changing the SSH Key File

After initial installation, you may need up update the PEM file that gives the Enterprise Console access to node machines. 

You can do so by creating the PEM file, as described in the discussion of configuring SSH passwordless login on Prepare the Events Service Host, and using the following command to install the new PEM file.  

bin/platform-admin.sh set-user-credentials --ssh-key-file newkeyfile.pem

The change takes effect immediately. 

Upgrade the Events Service

You can use the Enterprise Console to perform a rolling upgrade of the Events Service software on deployed nodes. For more information, see Upgrade the Events Service Using the Enterprise Console.

The general steps for upgrading an Events Service deployment are as follows: 

1. Upgrade the Controller. (See Upgrade the Controller Using the Enterprise Console.)

2. Apply the upgrade to the Events Service nodes using the following command: 

    bin/platform-admin.sh upgrade-events-service

The Enterprise Console checks whether the Events Service is up to date relative to the current Controller version and, if not, performs the update. 

Fine-Tune the Events Service Performance

The greater the number of scroll cursors on the elastic search nodes, the more resources they consume. Hence, you need to fine-tune the number of scroll cursors on the elastic search nodes. You need to consider the concurrent queries, workloads, and server configuration to fine-tune the scroll cursors. By default, the maximum number of scroll cursors is set to 500. You can define a custom scroll cursors limit. Set the following events property in the events-service-api-store.properties file:

 ad.search.max_open_scroll_context = 500

Monitor Events Service Nodes with AppDynamics

You can use AppDynamics agents to monitor an Events Service node or cluster and generate diagnostic information for troubleshooting. The following steps outline the workflow.

1. Download the following agents from the AppDynamics Download Center:
   • Java Agent
   • Machine Agent
2. Install both agents on each node in the cluster: first the Java Agent, then the Machine Agent.
3. On each node in the cluster, update the VM options for the Java Agent:
   1. Open the following file in a text editor: 
      <controller_home>/platform_admin/events-service/conf/events-service.vmoptions

   2. Add the following lines to the end of the file:

      -javaagent:/opt/appdynamics/events-service/java_agent/ver4.5.0.0/javaagent.jar
      -Dappdynamics.agent.accountName=<account_name>
      -Dappdynamics.agent.applicationName=<events_service_app_name>
      -Dappdynamics.controller.hostName=<controller_host>
      -Dappdynamics.controller.port=443
      -Dappdynamics.controller.ssl.enabled=true
      -Dappdynamics.agent.nodeName=<events_service_node_name>
      -Dappdynamics.agent.tierName=<events_service_tier_name>

      Adjust the path to the Java Agent JAR, account name, and other values as appropriate. 

      

      The business application name (events_service_app_name) and tier name (events_service_tier_name) should normally be the same for all nodes in an Events Service cluster, while each node must have a unique name (events_service_node_name). 

      
      For more information on modeling applications in AppDynamics, see Application Model Overview. 

4. On each node in the cluster, define the Node Name and Tier Name used by the Machine Agent, as described in Independent Machine Agent Installation. 

   

   The Node Name and Tier Name for each Machine Agent should be the same as the events_service_node_name and events_service_tier_name that you specify on each node.

5. Restart the Events Service on all nodes in the cluster: on the Controller host, navigate to <controller_home>/platform_admin/events-service/ and enter the following command: /bin/platform-admin.sh restart-events-service
6. In the Controller UI, go to the Applications table and open the dashboard for the events_service_app_name application. (You might need to wait a few minutes for this application to appear as the Events Services on the nodes restart and begin sending data to the Controller.) 
7. In the Application Dashboard, choose Configure > Instrumentation. 
8. Select the events_service_tier_name tier and choose Use Custom Configuration for this Tier.
9. Under Custom Match Rules, create a new rule with the following attributes:

• • Entry Point = Servlet
  • Split Transactions Using Request Data = Use the first 4 segments in Transaction names

Update the Java Temporary Directory for Events Service Nodes

You can set up an optional override of the temporary Java directory for the Events Service by using the Platform Admin in the CLI. Complete the steps below to update the Java Temporary Directory for the Event Service. 

After the Events Service settings update, the Events Service will require a restart initiated from the Platform Admin CLI or Enterprise Console GUI.

1. Run the command after installation:

/root/install/pa/platform-admin/bin/platform-admin.sh submit-job --service events-service --job update_jvm_temp_dir --args jvmTempDir=/var/tmp

     2. Make sure that the configuration is effective by running the following command:

/root/install/pa/platform-admin/bin/platform-admin.sh list-service-configurations --service events-service

   3. Add the configurations for job update_jvm_temp_dir:

jvmTempDir (STRING): [/var/tmp]

   4. To undo, use the following command:

/root/install/pa/platform-admin/bin/platform-admin.sh submit-job --service events-service --job update_jvm_temp_dir

   5. To add configurations for job update_jvm_temp_dir use the following command:

jvmTempDir (STRING): [null]