Download PDF
Download page Administer the Events Service.
Administer the Events Service
This page describes how to manage Events Service with the Enterprise Console. All of these tasks can be performed on the GUI or CLI.
Start and Stop the Events Service
The Events Service is an internal data storage engine used by the Database Visibility module. To use Database Monitoring, you need to start the Events Service.
The Events Service is automatically started after you install it.
On Linux
Start the Events Service on Linux by running this command:
bin/platform-admin.sh submit-job --platform-name <platform_name> --service events-service --job start
Stop the Events Service by running this command:
bin/platform-admin.sh submit-job --platform-name <platform_name> --service events-service --job stop
On Windows
Start the Events Service on Windows by running this command:
bin/platform-admin.exe cli submit-job --platform-name <platform_name> --service events-service --job start
Stop the Events Service by running this command:
bin/platform-admin.exe cli submit-job --platform-name <platform_name> --service events-service --job stop
Monitor Cluster Node Health
It is important to carefully monitor the health of the Events Service cluster, for a new deployment, especially to monitor disk consumption.
You can check the status of the cluster from the Controller page in the Enterprise Console GUI or the Controller machine using this command:
bin/platform-admin.sh show-events-service-health
The output shows possible issues and the steps you need to take to resolve them. For example, if the available disk is low, the resolution is to add nodes to the cluster.
The following are the potential errors and remediation steps:
Error | Explanation | Remediation |
---|---|---|
Cluster out of capacity | If the heap size of any Events Service Java process exceeds 80% utilization | Add Events Service nodes |
Disk size remaining drops below 30% | The disk size of the identified node dropped below 30% | Add Events Service nodes |
Events Service is not reachable but the host is reachable | The Events Service process on the identified node is not functioning properly | Restart the node |
Machine is not reachable | The machine may be down, disconnected, or suffering failure | Try restarting the machine; if it continues, the node may need to be removed from the cluster |
Cluster needs restart | A condition has been identified that requires a cluster restart | Restart the cluster |
Cluster size is 2 | Events Service cluster requires more than two nodes | Add a node |
Expand the Cluster
You can use the Enterprise Console GUI or CLI to horizontally scale the Events Service cluster on Linux. To grow your existing deployment to contend with an increased workload, simply add nodes.
Before starting, prepare the new cluster machine. Verify system requirements and prepare the environment as described in Events Service Requirements.
It is important for any new machine in the cluster to have the same SSH-enabled user account as existing cluster members.
Once you have prepared the system, run the command for adding nodes:
bin/platform-admin.sh add-events-service-nodes --hosts host1 host2 host3
Alternatively, pass the hostnames of the new node as a file.
bin/platform-admin.sh add-events-service-nodes --host-file=/home/appduser/hosts.txt
The file you pass to the command (hosts.txt
in the example) should contain the internal DNS hostnames or IP addresses of the nodes to add. It does not need to list existing nodes in the cluster. These hosts should be part of the platform. For more information about how to add a host to the platform, see Administer the Enterprise Console.
Be sure to modify your load balancer rules to include the new cluster member in its routing rules. See Load Balance Events Service Traffic for more information.
Restart the Node
You can use the Enterprise Console GUI or CLI to restart your node.
Once you have prepared the system, run the command for restarting your node:
bin/platform-admin.sh submit-job --platform-name <platform_name> --service events-service --job restart-node --args nodeActionHost=<node_name>
where <node_name>
is the hostname of the node from the command:
bin/platform-admin.sh list-nodes --platform-name <platform_name> --service events-service
Restart the Cluster
You can use the Enterprise Console GUI or CLI to restart your cluster.
Once you have prepared the system, run the command for restarting your cluster nodes:
bin/platform-admin.sh submit-job --platform-name <platform_name> --service events-service -job restart-cluster
Remove or Replace a Node
The uninstall-events-service
command removes the Events Service software and data from all cluster nodes. The Controller and Events Service share a database, so if you are uninstalling the Controller instance under which you ran the Enterprise Console to install the Events Service, you need to uninstall the Events Service with this command before you uninstall the Controller.
The remove-events-service-node
command removes the Events Service software and data from a single node that you specify by hostname. You should only use this command if you have at least four nodes in your cluster. Removing an Events Service node from a three-node cluster is not supported. Identify the node to remove using the --node
command line parameter.
After you remove a node, be sure to adjust your load balancer rules to remove the old cluster member. See Load Balance Events Service Traffic for more information.
If you are not using a load balancer with a cluster deployment, keep in mind that the connection settings for the first primary node that reports to the Controller at installation time are written to the Controller setting that identifies the Events Service to the Controller. If you remove a primary node in that case, check whether the removed primary node is node identified as the Events Service destination URL in the Controller connection settings (e.g., appdynamics.on.premise.event.service.url
) and adjust the setting if so. See Connect to the Events Service for more information.
Note the following guidelines:
- You cannot remove nodes such that the resulting cluster size is two
- A cluster that consists of three or more nodes can't be reduced in size to a single node Events Service.
The remove-events-service-node
command removes the Events Service software and data from a single node that you specify by hostname. You should only use this command if you have at least four nodes in your cluster. Removing an Events Service node from a three-node cluster is not supported. Identify the node to remove using the --node
command line parameter.
This command removes the node specified in the argument:
bin/platform-admin.sh remove-events-service-node --node 10.0.100.51
If you attempt to remove a primary node using the command shown above, the Enterprise Console notifies you that you are attempting to remove a primary node and cancels the operation. As indicated in the output, you can proceed to remove the primary node by rerunning the command with the -f
force flag. When you remove a primary node, the cluster elects a new primary node from the existing data nodes. The election process may take a few seconds, during which new events cannot be processed. Be sure to perform this operation at a time when the impact of a brief interruption of service will be minimal.
If you have an unreachable node you would like to remove, but cannot due to the above restrictions, you can choose to replace it instead.
This command replaces the old node specified in the argument with the new node:
bin/platform-admin.sh submit-job --service events-service --job replace-node --args originalNode=<old_host_address> newNode=<new_host_address>
- After removing the Events Service nodes from the cluster, you may observe that the value
appdynamics.es.eum.key
changed in the Controlleradmin.jsp
and in the Events Service properties file, but not in the EUM properties file,analytics.accountAccessKey
. - Check if the key value changed in the Controller and the Events Service, then replace the key value with the EUM properties file:
analytics.accountAccessKey
.
Enable the Events Service to Use SSL
You can use the Enterprise Console CLI to enable the Events Service to use SSL. You will need a KeyStore file in JKS format, the password and the alias for the KeyStore. See Create a Certificate and Generate a CSR for detailed instructions for creating the Keystore.
Log in to the Enterprise Console:
bin/platform-admin.sh login --user-name <admin_username> --password <admin_password>
TEXTAfter successfully logging in, submit an
enable-ssl
job for the Events Service, providing the path to the KeyStore file, the KeyStore password, and KeyStore alias.bin/platform-admin.sh submit-job --platform-name <platform_name> --service events-service --job enable-ssl --args keystorePath=<path-to-keystore-jks-file> keystorePassword=<keystore_password> keystoreAlias=<keystore_alias>
TEXTConfirm that SSL has been enabled:
curl -k -v -X GET https://<events-service-domain>:9080/_ping
TEXTThe output of the cURL command should show the TLS handshakes and the HTTP status 200:
... * TLSv1.2 (OUT), TLS handshake, Client hello (1): * TLSv1.2 (IN), TLS handshake, Server hello (2): ... < HTTP/1.1 200 OK < Date: Fri, 10 May 2019 00:13:49 GMT < X-Content-Security-Policy: default-src 'self' ...
TEXT
Collect Events Service Logs
The Enterprise Console can collect logs from the nodes in the cluster. The following command retrieves node logs and bundles them, along with the Enterprise Console's own logs:
bin/platform-admin.sh retrieve-events-service-logs
When the command is finished, a ZIP file named events-service.log.zip
is created in the location from which you ran the script. You can then extract the archive to troubleshoot or submit the archive for troubleshooting assistance to your AppDynamics representative. If the Enterprise Console failed to connect to one of the cluster nodes to retrieve logs for any reason, the connection error is written to a log file included in the archive.
Changing the SSH Key File
After initial installation, you may need up update the PEM file that gives the Enterprise Console access to node machines.
You can do so by creating the PEM file, as described in the discussion of configuring SSH passwordless login on Prepare the Events Service Host, and using the following command to install the new PEM file.
bin/platform-admin.sh set-user-credentials --ssh-key-file newkeyfile.pem
The change takes effect immediately.
Upgrade the Events Service
You can use the Enterprise Console to perform a rolling upgrade of the Events Service software on deployed nodes. For more information, see Upgrade the Events Service Using the Enterprise Console.
The general steps for upgrading an Events Service deployment are as follows:
- Upgrade the Controller. (See Upgrade the Controller Using the Enterprise Console.)
Apply the upgrade to the Events Service nodes using the following command:
bin/platform-admin.sh upgrade-events-service
The Enterprise Console checks whether the Events Service is up to date relative to the current Controller version and, if not, performs the update.
Monitor Events Service Nodes with AppDynamics
You can use AppDynamics agents to monitor an Events Service node or cluster and generate diagnostic information for troubleshooting. The following steps outline the workflow.
- Download the following agents from the AppDynamics Download Center:
- Java Agent
- Machine Agent
- Install both agents on each node in the cluster: first the Java Agent, then the Machine Agent.
- On each node in the cluster, update the VM options for the Java Agent:
- Open the following file in a text editor:
<controller_home>/platform_admin/events-service/conf/events-service.vmoptions
Add the following lines to the end of the file:
-javaagent:/opt/appdynamics/events-service/java_agent/ver4.5.0.0/javaagent.jar -Dappdynamics.agent.accountName=<account_name> -Dappdynamics.agent.applicationName=<events_service_app_name> -Dappdynamics.controller.hostName=<controller_host> -Dappdynamics.controller.port=443 -Dappdynamics.controller.ssl.enabled=true -Dappdynamics.agent.nodeName=<events_service_node_name> -Dappdynamics.agent.tierName=<events_service_tier_name>
Adjust the path to the Java Agent JAR, account name, and other values as appropriate.
The business application name (
events_service_app_name
) and tier name (events_service_tier_name
) should normally be the same for all nodes in an Events Service cluster, while each node must have a unique name (events_service_node_name
).
For more information on modeling applications in AppDynamics, see Application Model Overview.
- Open the following file in a text editor:
On each node in the cluster, define the Node Name and Tier Name used by the Machine Agent, as described in Independent Machine Agent Installation.
The Node Name and Tier Name for each Machine Agent should be the same as the
events_service_node_name
andevents_service_tier_name
that you specify on each node.- Restart the Events Service on all nodes in the cluster: on the Controller host, navigate to
<controller_home
>/platform_admin/events-service/
and enter the following command:/bin/platform-admin.sh restart-events-service
- In the Controller UI, go to the Applications table and open the dashboard for the events_service_app_name application. (You might need to wait a few minutes for this application to appear as the Events Services on the nodes restart and begin sending data to the Controller.)
- In the Application Dashboard, choose Configure > Instrumentation.
- Select the
events_service_tier_name
tier and choose Use Custom Configuration for this Tier. - Under Custom Match Rules, create a new rule with the following attributes:
- Entry Point = Servlet
- Split Transactions Using Request Data = Use the first 4 segments in Transaction names
Update the Java Temporary Directory for Events Service Nodes
You can set up an optional override of the temporary Java directory for the Events Service by using the Platform Admin in the CLI. Complete the steps below to update the Java Temporary Directory for the Event Service.
After the Events Service settings update, the Events Service will require a restart initiated from the Platform Admin CLI or Enterprise Console GUI.
- Run the command after installation:
/root/install/pa/platform-admin/bin/platform-admin.sh submit-job --service events-service --job update_jvm_temp_dir --args jvmTempDir=/var/tmp
2. Make sure that the configuration is effective by running the following command:
/root/install/pa/platform-admin/bin/platform-admin.sh list-service-configurations --service events-service
3. Add the configurations for job update_jvm_temp_dir
:
jvmTempDir (STRING): [/var/tmp]
4. To undo, use the following command:
/root/install/pa/platform-admin/bin/platform-admin.sh submit-job --service events-service --job update_jvm_temp_dir
5. To add configurations for job update_jvm_temp_dir
use the following command:
jvmTempDir (STRING): [null]