Download PDF
Download page Install the Events Service on Linux.
Install the Events Service on Linux
Related Pages:
The Enterprise Console automates the task of installing and administering an Events Service deployment through either the GUI or CLI. For information on installing Events Service using the Enterprise Console, see Custom Install.
You do not need to specify the installation or data directory for the Events Service installation. If you do, use a different one from the platform directory.
Events Service Host Requirements
Before starting, be sure to review the Release Notes for known issues and late-breaking information on using the Events Service and Enterprise Console. Also, observe the following requirements:
The Events Service can be deployed as a single node or as a multi-node cluster of 3 or more nodes.
- The versions of Linux supported include the flavors and versions supported by the Controller, as indicated by Prepare Linux for the Controller.
The Events Service must run on a dedicated machine. The machine should not run other applications or processes not related to the Events Service.
- Use appropriately sized hardware for the Events Service machines. The Enterprise Console checks the target system for minimum hardware requirements. For more information on these requirements, see the description of the profile argument to the Events Service install command in Install the Events Service Cluster.
- The Controller and Events Service must reside on the same local network and communicate by the internal network. Do not deploy the cluster to nodes on different networks, whether relative to each other or to the Controller where the Enterprise Console runs. When identifying cluster hosts in the configuration, you will need to use the internal DNS name or IP address of the host, not the externally routable DNS name.
For example, in terms of an AWS deployment, use the private IP address such as172.31.2.19
rather than public DNS hostname such asec2-34-201-129-89.us-west-2.compute.amazonaws.
com.
Make sure that the appropriate ports on each Events Service host are open. See Port Settings for more information.
The Enterprise Console uses an SSH key to access the Events Services hosts. See the section below for information on generating the key.
Events Service nodes normally operate behind a load balancer. When installing an Events Service node, the Enterprise Console automatically configures a direct connection from the Controller to the node. If you deploy a cluster, the first primary node is automatically configured as the connection point in the Controller. You will need to reconfigure the Controller to connect through the load balancer VIP after installation, as described below. For sample configurations, see Load Balance Events Service Traffic.
We recommend you that to grant execute permissions to the /tmp directory, otherwise the installation fails with an error. However, to troubleshoot this error, see Error if the /tmp Directory is Missing the Execute Permission.
Port Settings
Each machine must have the following ports accessible to external (outside the cluster) traffic:
- Events Service API Store Port: 9080
- Events Service API Store Admin Port: 9081
For a cluster, ensure that the following ports are open for communication between machines within the cluster. Typically, this requires configuring iptables
or OS-level firewall software on each machine to open the ports listed
9300 – 9400
The following shows an example of iptables
commands to configure the operating system firewall:
-A INPUT -m state --state NEW -m tcp -p tcp --dport 9080 -j ACCEPT -A INPUT -m state --state NEW -m tcp -p tcp --dport 9081 -j ACCEPT -A INPUT -m state --state NEW -m multiport -p tcp --dports 9300:9400 -j ACCEPT
If a port on the Events Service node is blocked, the Events Service installation command will fail for the node and the Enterprise Console command output and logs will include an error message similar to the following:
failed on host: <ip_address> with message: Uri [http://localhost:9080/_ping] is un-pingable.
If you see this error, make sure that the ports indicated in this section are available to other cluster nodes.
Create the SSH Key
When installing Events Service, you will need to provide the SSH key that the Enterprise Console can use to access Events Service hosts remotely. Before starting, create the PEM public and private keys in RSA format. The key file must not use password protection.
For example, using ssh-keygen
, you can create the key using the following command:
ssh-keygen -t rsa -b 2048 -v -m pem
Configure SSH Passwordless Login
The Enterprise Console needs to be able to access each cluster machine using passwordless SSH. Before starting, enable key-based SSH access.
This setup involves generating a key pair on the Enterprise Console host and adding the Enterprise Console's public key as an authorized key on the cluster nodes. The following steps take you through the configuration procedure for an example scenario. You will need to adjust the steps based on your environment.
If you are using EC2 instances on AWS, the following steps are taken care of for you when you provision the EC2 hosts. At that time, you are prompted for your PEM file, which causes the public key for the PEM file to be copied to the authorized_keys
of the hosts. You can skip these steps in this case.
On the Enterprise Console machine, follow these steps:
Log in to the Enterprise Console machine or switch to the user you will use to perform the deployment:
su - $USER
TEXTCreate a directory for SSH artifacts (if it doesn't already exist) and set permissions on the directory, as follows:
mkdir -p ~/.ssh chmod 700 ~/.ssh
TEXTChange to the directory:
cd .ssh
TEXTGenerate PEM public and private keys in RSA format:
ssh-keygen -t rsa -b 2048 -v -m pem
TEXT
The key file must not use password protection.- Enter a name for the file in which to save the key when prompted, such as
appd-analytics
. Rename the key file by adding the
.pem
extension:mv appd-analytics appd-analytics.pem
TEXTYou will later configure the path to it as the
sshKeyFile
setting in the Enterprise Console configuration file, as described in Deploying an Events Service Cluster.Transfer a copy of the public key to the cluster machines. For example, you can use
scp
to perform the transfer as follows:scp ~/.ssh/myserver.pub host1:/tmp scp ~/.ssh/myserver.pub host2:/tmp scp ~/.ssh/myserver.pub host3:/tmp
TEXTContinuing with the example,
myserver
should beappd-analytics
.
The first time you connect you may need to confirm the connection to add the cluster machine to the list of known hosts and enter the user's password.On each cluster node (host1, host2, and host3), create the
.ssh
directory in the user home directory, if not already there, and add the public key you just copied as an authorized key:cat /tmp/appd-analytics.pub >> .ssh/authorized_keys chmod 600 ~/.ssh/authorized_keys
TEXTTest the configuration from the Controller machine by trying to log in to a cluster node by
ssh
:ssh host1
TEXTIf unable to connect, make sure that the cluster machines have the
openssh-server
package installed and that you have modified the operating system firewall rules to accept SSH connections. If successful, you can use the Enterprise Console on the Controller host to deploy the Events Service cluster, as described next.
If the Enterprise Console attempts to install the Events Service on a node for which passwordless SSH is not properly configured, you will see the following error message:
./bin/platform-admin.sh add-hosts --hosts <es_host_1> <es_host_2> <es_host_3> --credential <credential name> --platform-name <name of platform>
...
Failed to connect to the remote host. Please verify that the host name and credentials you provided are correct.
For more information, consult the documentation: https://docs.appdynamics.com/display/PRO45/Administer+the+Enterprise+Console#AdministertheEnterpriseConsole-manage-hostsManageHosts
If you encounter this error, use the instructions in this section to double-check your passwordless SSH configuration. Also, you need to and add-hosts
first then install event-service
. If SSH configuration is not setup correctly, add-hosts
commands will fail.
The add-hosts
command is to add hostnames into platforms. During the events-service
installation, the hosts come from the platform hosts, and then they are used on the events-service
.
Tune the Operating System for Production Cluster Nodes
Before installing the Events Service cluster, you need to perform a few manual changes as described below. These are particularly relevant for production Events Service deployments. On each node in the cluster, make these configuration changes:
Using a text editor, open
/etc/sysctl.conf
and add the following:vm.max_map_count=262144
CODERaise the open file descriptor limit in
/etc/security/limits.conf
, as follows:<username_running_eventsservice> soft nofile 96000 <username_running_eventsservice> hard nofile 96000 <username_running_eventsservice> soft memlock unlimited <username_running_eventsservice> hard memlock unlimited
CODEDisable swap memory by running the following command. Remove swap mount points by removing or commenting the lines in
/etc/fstab
that contain the word swap.swapoff -a
CODE
Installing the Events Service Using the GUI
In the GUI, the Express Install option automatically installs an Events Service on the same host as the Controller. The Custom Install option can install an embedded or scaled-up Events Service. If you install an embedded Events Service and want to switch to a scaled-up Events Service, complete the steps described in Scaling Up an Embedded Events Service.
Installing the Events Service Using the CLI
Set up load balancing. See Load Balance Events Service Traffic for information about configuring the load balancer.
At the command line, navigate to the
platform-admin
directory created at Enterprise Console installation. See Install the Enterprise Console.If it has been more than one day since your last session, you will have to log in with the following command:
bin/platform-admin.sh login --user-name <admin_username> --password <admin_password>
TEXTCreate a platform as follows:
bin/platform-admin.sh create-platform --name <platform_name> --installation-dir <platform_installation_directory>
TEXTThe installation directory is the directory where the Enterprise Console installs all platform components.
The same installation directory should exist and is used on all remote nodes. This is done to maintain the homogeneity of the configuration across different nodes.
Add the SSH key that the Enterprise Console will use to access and manage the Events Service hosts remotely. (See Create the SSH Key for more information):
bin/platform-admin.sh add-credential --credential-name <name> --type ssh --user-name <username> --ssh-key-file <file path to the key file> --platform-name <name of platform>
TEXT<file path to the key file>
is the private key for the Enterprise Console machine. The installation process uses the keys to connect to the Events Service hosts. The keys are not deployed, but instead, encrypted and stored in the Enterprise Console database.The
platform-name
parameter is optional.Add hosts to the platform, passing the credential you added to the platform:
bin/platform-admin.sh add-hosts --hosts es_host_1 es_host_2 es_host_3 --credential <credential name> --platform-name <name of platform>
TEXTThe
platform-name
parameter is optional.On each Events Service destination node in the cluster, create an installation directory for the Events Service. This is the directory you specified as the
installation-dir
argument while creating the platform in step (2).Again at the command line for the Enterprise Console machine, run the following command from the
platform-admin
directory. Pass the cluster configuration settings as arguments to the command. The format for the command is the following:bin/platform-admin.sh submit-job --platform-name <platform-name> --service events-service --job install --target-version <latest> --args profile=<dev> serviceActionHost=<es_host_1> serviceActionHost=<es_host_2> serviceActionHost=<es_host_3>
TEXTThe
platform-name
andjvmTempDir
parameters are optional.Arguments are:
jvmTempDir
: Use this argument to override default JVM temporary/tmp
directory in Linux installations.hosts
: Use this argument orhost-file
to specify the internal DNS hostnames or IP addresses of the cluster hosts in your deployment. With this argument, pass the hostnames or addresses as parameters. For example:--hosts 192.168.32.105 192.168.32.106 192.168.32.107
host-file
: As an alternative to specifying hosts as --hosts
arguments, pass them as a list in a text file you specify with this argument. Specify the internal DNS hostname or IP address for each cluster host as a separate line in the plain text file:192.168.32.105 192.168.32.106 192.168.32.107
profile
: By default (with profile not specified), the installation is considered a production installation. Specifying a developer profile (‑‑profile dev
) directs the Enterprise Console to use a reduced hardware profile requirement, suitable for non-production environments only. The Enterprise Console checks for the following resources:For a dev profile, 1 core CPU, 1 GB RAM, and 2 GB disk space.
Otherwise, 4 core CPU, 12 GB RAM, and 128 GB of disk space.
For example:
bin/platform-admin.sh add-hosts --hosts ip-172-31-20-21.us-west-2.compute.internal ip-172-31-20-22.us-west-2.compute.internal ip-172-31-20-23.us-west-2.compute.internal
TEXTIf using a hosts text file, use the following command:
bin/platform-admin.sh add-hosts --hosts --host-file=/home/appduser/hosts.txt
TEXTLog in to each Events Service node machine, and run the script for setting up the environment as follows:
- The
tune-system.sh
script is used for optimizing your environment. It is optional. - Enable the following property:
INSTALL_BOOTSTRAP_MASTER_ES8=true
CODE
Add execute permission to the
tune-system.sh
script:chmod +x tune-system.sh ./tune-system.sh
TEXTRun the script:
sudo <installation_dir>/events-service/processor/bin/tool/tune-system.sh
TEXT
- The
If you are using a load balancer, use the virtual IP for the Events Service as presented at the load balancer. Configure the Controller connection to the Events Service as follows:
Open the Administration Console.
In the Controller settings pane, find
appdynamics.on.premise.event.service.url and
change its value to the URL of the virtual IP for the Events Service at the load balancer.
It may take a few minutes for the Controller and Events Service to synchronize account information after you modify connection settings in the console.
When finished, use the Enterprise Console for any Events Service administrative functions. You should not need to access the cluster node machines directly once they are deployed. In particular, do not attempt to use scripts included in the Events Service node home directories.
The Enterprise Console automatically updates the Controller configurations after installation.
Scaling Up an Embedded Events Service
The following steps describe how to scale up an Events Service that is on a shared host with the Controller. This allows the embedded Events Service to run on a separate host. You can also install the Events Service on a separate host directly by using the Custom Install.
- Set up load balancing. See Load Balance Events Service Traffic for information about configuring the load balancer.
- Open the Enterprise Console GUI.
Verify that the credentials and hosts you want to use are added to the Cisco AppDynamics platform. For more information, see Administer the Enterprise Console.
On the Credential page, add the SSH credentials for the hosts on which you want to install the Events Service.
- On the Hosts page, add the hosts. The Enterprise Console uses these hosts for the scaled-up Events Service, which requires at least one host or three or more hosts.
On the Events Service page, navigate to More link and select the Events Service and click Scale Up Events Service under More and complete the wizard. When you enter hosts to use for a scaled-up Events Service, do not include the Controller host.
You do not need to restart the Controller since that is automatically done for you by the scale-up job.
Log in to each node machine, and run the script for setting up the environment as follows:
sudo <installation_dir>/events-service/latest/bin/tool/tune-system.sh
TEXT- Navigate to the Controller page.
By default, the Enterprise Console configures the Events Service connection in the Controller to refer to the first primary node defined in the cluster. If you are using a load balancer, as recommended, you need to change this Controller setting to point to the Events Service VIP as presented at the load balancer instead, as follows:
Open the Administration Console.
- In the Controller settings pane, find
appdynamics.on.premise.event.service.url.
Change the value of the setting for the URL to the VIP for the Events Service at the load balancer.
By default, Database Monitoring stores events data in the Events Service embedded in the Controller. To have it use the Events Service you just deployed, ensure that the
appdynamics.on.premise.event.service.key
value matches withad.accountmanager.key.controller
value inside theevents-service-api-store.properties
file.Note that only newly generated Database Monitoring data will be stored in the Events Service; previously collected data will remain in the embedded Events Service instance unless it is migrated to the new Events Service. See Connect to the Events Service.
It may take a few minutes for the Controller and Events Service to synchronize account information after you modify connection settings in the Enterprise Console.
Troubleshooting Installation
If the Enterprise Console crashes or shuts down while installing the Events Service, the GUI may display that the installation is in progress. To resolve this issue, uninstall the Events Service with the CLI. Then, install the Events Service with the CLI.
Handling ClusterFormationFailureHelper Error
When you run new Events Service 23.2.0 on a single node event service cluster, the following error may be displayed.
[WARN ][o.e.c.c.ClusterFormationFailureHelper] [ip] master not discovered yet, this node has not previously joined a bootstrapped cluster, and [cluster.initial_master_nodes] is empty on this node.
To resolve this issue:
- Uninstall the Events Service from Enterprise Console.
- Ensure that you have tuned the operating system for cluster nodes as defined in this section.
- Re-install the Events Service.
Error if the /tmp Directory is Missing the Execute Permission
If the /tmp
directory is missing execute permission, the Events Service installation or upgrade fails with the following errors:
failed to map segment from shared object or
failed to allocate closure
Run the following command to verify the execute permission of the folder:
mount | grep '/tmp' /dev/sda1 on /tmp type ext4 (rw,noexec)
CODECheck the privileges for the following folders:
mount | grep '/var/tmp' mount | grep 'home'
CODE
You can troubleshoot the error by:
- Exporting the files to the Events Service directory.
- Removing the
noexec
mount option for the required directories.
Exporting the Files to the Events Service Directory
- Remove the Events Service files that were added during the failed installation.
Locate the binaries of Events Service in the following path:
cd /opt/appdynamics/platform/platform-admin/archives/events-service/<version>cd /opt/appdynamics/platform/platform-admin/archives/events-service/<version>
CODEUnzip
events-service.zip
unzip events-service.zip
CODERename the
events-service.zip
toevents-services.zip.original
.mv events-service.zip events-service.zip.original
CODEIn the
extracted events-service
directory, add the following lines to both theevents-service.vmoptions
andevents-service-elasticsearch.vmoptions
files:-Djna.tmpdir=/opt/appdynamics/platform/product/events-service/tmp -Djansi.tmpdir=/opt/appdynamics/platform/product/events-service/tmp
CODEArchive the
events-service
folder with the modified.vmoptions
files:zip -r events-service.zip events-service
CODECreate a temporary directory in the
events-service
folder where.vmoptions
files is mentioned.mkdir -p /opt/appdynamics/platform/product/events-service/tmp
CODEExport the temporary directory using the following command:
export TMPDIR=<path/to/es_tmp>
CODE- Install the Events Service using Enterprise Console.
Removing the noexec
Mount Option for the Required Directories
Remove the noexec
mount option for the following directories:
sudo umount /tmp
sudo mount -o remount,rw /tmp