Related pages:

You can install and administer the Events Service on Microsoft Windows systems as a single node or a cluster. Common use cases include:

  • Single-node Events Service — Good for demonstration purposes and other scenarios where data redundancy and high availability are not required.
  • Three-node cluster — The minimum size for a production Events Service cluster.
  • Cluster of four nodes or more — For deployments where increased load or expected sizing exceeds the capacity of a three-node cluster.

 Contact AppDynamics customer support if you anticipate deploying a cluster of 10 or more nodes.

On Windows, creating more than one instance of the same service type is not supported.

You do not need to specify the installation or data directory for the Events Service installation. If you do, use a different one from the platform directory.

Preparation: Decide which Node(s) to Make Master Nodes

Every Events Service node is either a master node or a data node. In an Events Service cluster, the master node both acts as a storage node, and manages the state of the data across the cluster, including the state of the replica. In a single-node deployment, there's not much for the master to do.

The master node is the first node to start up. If the master node becomes unavailable, the worker nodes attempt to elect a new master.  

As you install Events Service, you specify the configuration for each node, which consists of two pieces of information: 

  • Whether to enable the node to serve as a master, and
  • How many nodes (minimum) must be available in the cluster for a new master to be elected

The table below tells you what values to specify as you install. 


Order of Node in the DeploymentMaster-enabled?Minimum available nodes required
to elect a new master
First (only) node in a single-node installationtrue1
First, second, and third nodes of a three-node clustertrue2
Fourth (or higher) node of a clusterfalse2

Specifying the installation or data directory is optional for the Events Service. If you do this, the directory you specify must not be the platform directory.

Deploying a Single-Node Events Service

If you are installing the Events Service on the same host as the Controller and do not wish to deploy manually, use the Express Install option in the Enterprise Console GUI.

To install the Events Service on a single node: 

  1. Unzip the Events Service distribution archive to a directory on the target host.

    • This creates the events-service directory with the Events Service artifacts.

  2. Begin configuring the connection to the Events Service in the Controller.

    1. With the Controller running, open the Administration Console as the root user. 
    2. In the Controller Settings page, search for appdynamics.on.premise.event.service.url.
    3. Replace the default value to the internal hostname for the Events Service machine and default Events Service listen port, 9080.
      • If you are putting a load balancer in front of the Events Service (required for a cluster), this will be the VIP for the Events Service as exposed at the load balancer. In this case, it is likely you will need to return to this step after you finish deploying the node and configuring the load balancer. 
  3. Get the Controller key
    1. Still in the Controller Settings page, search for the appdynamics.on.premise.event.service.key setting.
    2. Copy the setting's value—it is needed in Step 4.
    3. Close the Administration Console.

  4. Configure the connection from the Events Service to the Controller:

    1. In a terminal, navigate to the  events-service directory:

      cd events-service

    2. Open the conf\events-service-api-store.properties file for editing

    3. Find the following property and replace controller-key with the copied key: 

      ad.accountmanager.key.controller=controller-key

  5. Verify the heap settings for the Events Service processes 

    1. Verify that the minimum and maximum heap settings for the two Events Service processes (the Events Service JVM, and the Elasticsearch processes, respectively) are correct and sufficient for your deployment 

      • These settings: 

        • are found at the end of the events-service-api-store.properties file

        • use g for gigabyte (GB) and m for megabyte (MB)

    • For the Events Service process, verify:

      ad.jvm.options.name=events-service.vmoptions
ad.jvm.heap.min=1g
ad.jvm.heap.max=1g

    • For the ElasticSearch process:: 

      ad.es.jvm.options.name=events-service.vmoptions
ad.es.jvm.heap.min=8g
ad.es.jvm.heap.max=8g

      A production Elasticsearch installation requires 8 GB. For a demonstration installation, you can retain the default of 1 GB.

  6. Save and close the events-service-api-store.properties file. 

  7. Install the Events Service as a Windows service:  

    bin\events-service.exe service-install -p conf\events-service-api-store.properties --auto-start
    • This command also installs Elasticsearch (even though it contains no explicit reference to Elasticsearch).
    • The optional auto-start flag causes the Events Service to be installed as an automatically started service; without the flag the Events Service is installed as a manually started service.

    • For verbose installation and operation logging (useful for troubleshooting), include the ‑‑log-verbose flag.

  8. Find the service name for the Events Service:

    bin\events-service.exe service-list

  9. Enclose the service name in double-quotes and pass it to the service-start command:  

    bin\events-service.exe service-start -s "<Name from service-list>"
    • When the events-service process starts, its child Elasticsearch process starts also

  10. Check the health of the new node.  

    bin\events-service.exe check-health -hp localhost:9081

    For the port, pass the administration port for the Events Service, 9081 by default. Verify that "Healthy" appears as the service status, indicating that the process is operating normally:

    [appduser@controller-one events-service]$ bin/events-service.exe check-health -hp 192.168.33.22:9081
[2015-12-09T18:30:45,342-08:00] HV000001: Hibernate Validator 5.0.2.Final
[2015-12-09T18:30:45,956-08:00] Individual statuses below:
[2015-12-09T18:30:45,956-08:00] [192.168.33.22:9081] status is [200 OK]
[2015-12-09T18:30:45,956-08:00] Overall status Healthy
...

  11. Configure the connections from the Analytics Agent, EUM Server, or Database Monitoring agents to the Events Service, as described in Connect to the Events Service. 

Deploying an Events Service Cluster (3 Nodes)

The following steps describe how to deploy an Events Service cluster made up of three nodes, the minimum size of the Events Service cluster. These steps apply whether you are performing a new installation of a cluster or expanding a single node deployment into a three-node cluster. For information on expanding beyond a three-node cluster, see Adding Nodes to a Cluster. 

Note that all services on Windows machines must be installed on the Enterprise Console host since the Enterprise Console does not support remote operations on Windows. Therefore, you cannot use the Enterprise Console GUI to deploy an Events Service cluster. 

Before starting, review the topology notes in Events Service Deployment and make sure that all machines in the cluster meet the system requirements.

When ready, perform these steps on each of the three nodes.

  1. Follow the steps for configuring a single node cluster in the 1-node installation above. Additionally, configure the following settings in the  conf\events-service-api-store.properties  file:

    1. Change the value of the ad.es.node.minimum_master_nodes property to 2:

      ad.es.node.minimum_master_nodes=2

      The setting specifies the minimum number of master-eligible instances that must be available in order to elect a new master. Since an Events Service cluster has three master nodes, this value should be two for a cluster.

    2. Set the value of ad.es.event.index.shards to the number of nodes, in this case, three:

      ad.es.event.index.shards=3

      You do not need to change this value if it is already higher than the number of nodes.

    3. Set the replication factor to 1 by changing the ad.es.event.index.replicas and ad.es.metadata.replicas properties, as follows:

      ad.es.event.index.replicas=1
ad.es.event.index.hotLifespanDays=10

ad.es.metadata.replicas=1

    4. For the unicast hosts property, add the hostname or IP address, along with the port 9300, for each node in the cluster:

      ad.es.node.unicast.hosts=node1.example.com:9300,node2.example.com:9300,node3.example.com:9300

    5. Change the publish host to the IP address or hostname of this machine. For example:

      ad.es.node.network.publish.host=node2.example.com

    6. Configure heap space for the Events Service and ElasticSearch processes, as follows:

      1. To set the Events Service process heap size to 1 GB, for example, use the following properties:

        ad.jvm.options.name=events-service.vmoptions
ad.jvm.heap.min=1g
ad.jvm.heap.max=1g

        For the setting value, g indicates gigabyte (GB), and m indicates megabyte (MB). 

      2. For the ElasticSearch process, the heap size should be set to half the size of the available RAM on the system, up to a maximum of 31 GB. To set the ElasticSearch process heap size to 8 GB, for example, set the properties as follows:

        ad.es.jvm.options.name=events-service.vmoptions
ad.es.jvm.heap.min=8g
ad.es.jvm.heap.max=8g

        For the setting value, g indicates gigabyte (GB), and m indicates megabyte (MB). 

    7. Save and close the file.

  2. Install the Events Service as a Windows service:  

    bin\events-service.exe service-install -p conf\events-service-api-store.properties  --auto-start

    The optional auto-start flag causes the Events Service to be installed as an automatically started service. If you do not include the flag, the Events Service is installed as a manually started service. An additional option, ‑‑log-verbose, increases the verbosity of installation and operation logging, which is useful for troubleshooting.

  3. Enter the following command to find the service name for the Events Service:

    bin\events-service.exe service-list

  4. Pass the service name returned by the service-list command as the -s parameter argument in the following command: 

    bin\events-service.exe service-start -s "<Name from service-list>"

    Be sure to enclose the name in double-quotes. 

  5. Check the health of the new node using the following command. At least two nodes must be running before you run the command. 

    bin\events-service.exe check-health -hp localhost:9081

    For the port, pass the administration port for the Events Service, 9081 by default. Verify that "Healthy" appears as the service status, indicating that the process is operating normally:

    [appduser@controller-one events-service]$ bin/events-service.exe check-health -hp 192.168.33.22:9081
[2015-12-09T18:30:45,342-08:00] HV000001: Hibernate Validator 5.0.2.Final
[2015-12-09T18:30:45,956-08:00] Individual statuses below:
[2015-12-09T18:30:45,956-08:00] [192.168.33.22:9081] status is [200 OK]
[2015-12-09T18:30:45,956-08:00] Overall status Healthy
...

  6. Configure a load balancer to distribute traffic to the Events Service cluster, as described in Load Balance Events Service Traffic. 

  7. Connect the Controller and other clients—Analytics Agent, EUM Server, or Database Monitoring agents—to the Events Service, as described in Connect to the Events Service.  

Expanding an Events Service Cluster

The Events Service cluster is horizontally scalable. You can add nodes to an existing cluster without affecting or having to restart the existing nodes. 

Note that all services on Windows machines must be installed on the Enterprise Console host since the Enterprise Console does not support remote operations on Windows. Therefore, you cannot use the Enterprise Console GUI to expand an Events Service cluster. 

Before starting, prepare the new cluster machine. Verify system requirements and prepare the environment as described above. 

For each node beyond the original three master nodes, download and configure the nodes as previously described. The configuration steps for any nodes added to the cluster after the initial three master nodes are as follows:

  1. For each cluster nodes beyond the initial three master nodes, open the conf\events-service-api-store.properties for editing and make these configuration changes:

    1. Set the ad.es.node.master value to false: 

      ad.es.node.master=false

    2. Set the ad.es.node.minimum_master_nodes value to 2. 

      ad.es.node.minimum_master_nodes=2

    3. Set the value of ad.es.event.index.shards to the number of nodes in the cluster. You do not need to change this value if it is already higher than the number of nodes.

      ad.es.event.index.shards=<number_of_nodes>

       You do not need to change this value if it is already higher than the number of nodes.

    4. For the unicast hosts property, add the hostnames or IP addresses of all nodes in the cluster, including the node you are adding. For each node specify the ports on which the nodes communicate, 9300-9400. For example:

      ad.es.node.unicast.hosts=node1.example.com[9300-9400],node2.example.com[9300-9400],node3.example.com[9300-9400],node4.example.com[9300-9400]

      You do not need to reconfigure the unicast hosts settings for existing cluster members, as the new node can join the cluster dynamically. 

    5. Change the publish host to the IP address or hostname of this machine. For example:

      ad.es.node.network.publish.host=node4.example.com

    6. Configure heap space for the Events Service and ElasticSearch processes, as follows:

      1. To set the Events Service process heap size to 1 GB, for example, use the following properties:

        ad.jvm.options.name=events-service.vmoptions
ad.jvm.heap.min=1g
ad.jvm.heap.max=1g

        For the setting value, g indicates gigabyte (GB), and m indicates megabyte (MB). 

      2. For the ElasticSearch process, the heap size should be set to half the size of the available RAM on the system, up to a maximum of 31 GB. To set the ElasticSearch process heap size to 8 GB, set the properties as follows:

        ad.es.jvm.options.name=events-service.vmoptions
ad.es.jvm.heap.min=8g
ad.es.jvm.heap.max=8g

        For the setting value, g indicates gigabyte (GB), and m indicates megabyte (MB). 

    7. Save and close the file.  

  2. Install the Events Service as a Windows service:  

    bin\events-service.exe service-install -p conf\events-service-api-store.properties --auto-start

    The optional auto-start flag causes the Events Service to be installed as an automatically started service. If you do not include the flag, the Events Service is installed as a manually started service. An additional option, ‑‑log-verbose, increases the verbosity of installation and operation logging, which is useful for troubleshooting.

  3. Enter the following command to find the service name for the Events Service:

    bin\events-service.exe service-list

  4. Pass the service name returned by the service-list command as the -s parameter argument in the following command: 

    bin\events-service.exe service-start -s "<Name from service-list>"  

  5. Check the health of the new node:

    bin\events-service.exe check-health -hp localhost:9081

    Note:  At least two nodes must be running before you run the command.

    For the port, pass the administration port for the Events Service, 9081 by default. Verify that "Healthy" appears as the service status, indicating that the process is operating normally:

    [appduser@controller-one events-service]$ bin/events-service.bin check-health -hp 192.168.33.22:9081
[2015-12-09T18:30:45,342-08:00] HV000001: Hibernate Validator 5.0.2.Final
[2015-12-09T18:30:45,956-08:00] Individual statuses below:
[2015-12-09T18:30:45,956-08:00] [192.168.33.22:9081] status is [200 OK]
[2015-12-09T18:30:45,956-08:00] Overall status Healthy
...
  6. Modify your load balancer rules to include the new cluster node. For more information, see Load Balance Events Service Traffic.

Starting and Stopping the Events Service

At installation, the Events Service is installed as a service and is left running upon completion of the installation. You can stop and stop it as a service or as a foreground process, as described here or by using the GUI.

Starting and Stopping as a Foreground Process

To start the Events Service as a foreground process, however, use the following command: 

bin\events-service.exe start -p conf\events-service-api-store.properties

To stop the Events Service as a foreground process, use this command: 

bin\events-service.exe stop

Stopping and Starting as a Windows Service

You can stop and start the Events Service as a Windows service from the Services Manager. You can also stop and start it using the events-service.exe tool, as here: 

  1. Enter the following command to find the service name for the Events Service:

    bin\events-service.exe service-list

  2. Pass the service name returned by the service-list command as the -s parameter argument to the following command. Enclose the service name in double-quotes.  

    bin\events-service.exe service-start -s "<Name from service-list>"

To stop the service, run this command:

bin\events-service.exe service-stop -s "<Name from service-list>"

Removing a Node

To remove a node that is not enabled for operation as a master node from the cluster, simply stop the Events Services on the node or remove the machine it runs on from the network. 

Note the following guidelines: 

  • You cannot remove nodes such that the resulting cluster size is two 
  • A cluster that consists of three or more nodes can't be reduced in size to a single node Events Service. 

After you remove a node, be sure to adjust your load balancer rules to remove the old cluster member. See Load Balance Events Service Traffic for more information. 

If you are not using a load balancer with a cluster deployment, keep in mind that the connection settings for the first master node that reports to the Controller at installation time are written to the Controller setting that identifies the Events Service to the Controller. If you remove a master node in that case, check whether the removed master node is node identified as the Events Service destination URL in the Controller connection settings; adjust the setting if so. See Connect to the Events Service for more information.

To reconfigure an existing node to enable operation as a master node, or add a new node with the master option enabled:

ad.es.node.master=true

If reconfiguring a node, restart the node after changing the configuration.