Download PDF
Download page Set Up a High Availability Deployment.
Set Up a High Availability Deployment
This page describes how to set up and deploy Controllers as a high availability (HA) pair. For installation and upgrade details, see Custom Install and Upgrade an HA Pair.
About the HA Deployment Using the Enterprise Console
The Enterprise Console automates HA-related setup and administration tasks for the Linux operating system. It does not require sudo
privileges and can be deployed as a non-root user on Unix operating systems. It works with most flavors of Linux, including Ubuntu and Red Hat/CentOS.
The servers of Controllers in an HA pair must be identical in terms of OS, CPU, RAM, and Disk. See Controller System Requirements.
You can:
- Configure Controllers in a high availability pair arrangement.
- Use the Enterprise Console to monitor the health of the primary Controller, App Server, and database, and failover to the secondary when needed.
- Use scripts that allows you to install the Controllers as a Linux service, and gracefully stop and start service in the event of a machine reboot.
- Failover to a secondary Controller manually (for example, when you need to perform maintenance on the primary).
- Revive a Controller (restore a Controller as an HA secondary after its database is more than seven days behind the primary as a replica).
- Set up a Controller HA pair.
Deploying Controllers as an HA pair ensures that service downtime in the event of a Controller machine failure is minimized. It also facilitates other administrative tasks, such as backing up data. For more background information, including the benefits of HA, see Controller High Availability (HA).
Before Starting
For general guidelines and requirements on how to deploy HA in your environment, see Prerequisites for High Availability. Your environment must meet the prerequisites.
User Privilege Escalation Requirements
After installing a Controller high availability via the Enterprise Console, it is recommended to install MySQL as a Linux service. This is to prevent against MySQL data integrity issues. Installing the Controller and MySQL as a Unix service will ensure that whenever the machine reboots, the service will be shut down and started gracefully.
/etc/sudoers.d/appdynamics
contains entries to allow the AppDynamics user to access the/sbin/service
utility usingsudo
without a password. This mechanism is not available if the AppDynamics user is authenticated by LDAP./sbin/appdservice
is asetuid
root program distributed in source form in<controller_home>/controller-ha/init/appdservice.c.
It is written explicitly to support auditing by security audit systems. Theinstall-init.sh
script compiles and installs the program. It is executable only by the AppDynamics user and the root user. The script requires a C compiler to be available on the system. You can install a C compiler using the package manager for your operating system. For example, on Yum-based Linux distributions, you can use the following command to install the GNU Compiler, which includes a C compiler:sudo yum install gcc
Load Balancer Requirements and Considerations
Before setting up HA, a reverse proxy or load balancer needs to be available and configured to route traffic to the active Controller in the HA pair. Using a load balancer to route traffic between Controllers (rather than other approaches, such as DNS manipulation) ensures that a failover can occur quickly, without, for example, delays due to DNS caching on the agent machines.
An HA deployment requires the following IP addresses:
- One for the virtual IP of the Controller pair as presented by the load balancer used by clients, such as App Agents, to access the Controller. (Callout in the following diagram.)
- Two IP addresses for each Controller machine, one for the HTTP primary port interface () and one for the dedicated link interface between the Controllers () on each machine. The dedicated link is recommended but not mandatory.
- If the Controllers will reside within a protected, internal network behind the load balancer, you also need an additional internal virtual IP for the Controller within the internal network ().
When configuring replication, you specify the external address at which Controller clients, such as app agents and UI users, will address the Controller at the load balancer. The Controllers themselves need to be able to reach this address as well. If the Controller will reside within a protected network relative to the load balancer, preventing them from reaching this address, there needs to be an internal VIP on the protected side that proxies the active Controller from within the network. This is specified using the -
i parameter.
The load balancer can check the availability of the Controller at the following address:
http://<controller_host>:<port>/controller/rest/serverstatus
If the Controller is active, it responds to a GET request at this URL with an HTTP 200 response. The body of the response indicates the status of the Controller in the following manner:
<serverstatus vendorid="" version="1">
...
<available>true</available>
...
Ensure that the load balancer policy you configure for the Controller pair can send traffic to only a single Controller in the pair at a time (i.e., do not use round-robin or similar routing distribution policy at the load balancer). For more information about setting up a load balancer for the Controller, see Use a Reverse Proxy.
Set Up the Controller High Availability Pair
To set up high availability:
Step 1: Configure the Controller High Availability Pair Environment
The following sections provide more information on how to configure a few of the system requirements. They describe how to configure the settings on Red Hat Linux as a sample deployment. Note that the specific steps for configuring these requirements may differ on different systems. Consult your system documentation for specific details.
Host Reverse Lookups
You need to set up a reliable symmetrical reverse host lookup on each machine. To do this, enter the hostnames of the pair into the hosts files (/etc/hosts
) on each machine. This is preferable over other approaches, such as using reverse DNS, which adds a point of failure.
To enable reverse host lookups, on each host:
- In
/etc/nsswitch.conf
, enterfiles
beforedns
to have thehosts
file entries take precedence over DNS. For example:hosts: files dns
In
/etc/hosts
file, add an entry for each host in the HA pair. For example:192.168.144.128 host1.domain.com host1
192.168.144.137 host2.domain.com host2
To reduce errors, use the correct format of
/etc/hosts
files. If you have both dotted hostnames and short versions, you need to list the dotted hostnames with the most dots first and the other versions subsequently. This should be done consistently for both HA server entries in each of the two/etc/hosts
files. Note that in the examples provided, the aliases are listed last.
Set Up the SSH Key
SSH must be installed on both hosts in a way that gives the user who runs the Controller passwordless SSH access to the other Controller system in the HA pair. You can accomplish this by generating a key pair on each node, and placing the public key of the other Controller into the authorized keys (authorized_keys) file on each Controller.
The following steps describe how to perform this configuration. The instructions assume an AppDynamics user named appduser,
and the Controller hostnames are node1, the active primary, and node2, the secondary. Adjust the instructions for your particular environment. Also note that you may not need to perform every step (for example, you may already have the .ssh
directory and don't need to create a new one).
Although not shown here, some of the steps may prompt you for a password.
On the primary (node1) host:
Change to the AppDynamics user,
appduser
in our example:su - appduser
Create a directory for SSH artifacts (if it doesn't already exist) and set permissions on the directory, as follows:
mkdir -p .ssh chmod 700 .ssh
Generate the RSA-formatted key:
ssh-keygen -t rsa -N "" -f .ssh/id_rsa -m pem
Secure copy the key to the other Controller:
scp .ssh/id_rsa.pub node2:/tmp
On the secondary (node2) host:
As you did for node1, run these commands:
su - appduser mkdir -p .ssh chmod 700 .ssh ssh-keygen -t rsa -N "" -f .ssh/id_rsa -m pem scp .ssh/id_rsa.pub node1:/tmp
Add the public key of node1 that you previously copied to the secondary Controller host's authorized keys and set permissions on the authorized keys file:
cat /tmp/id_rsa.pub >> .ssh/authorized_keys chmod 700 ~/.ssh/authorized_keys
On the primary (node1) again:
Move the secondary's public key to the authorized keys
cat /tmp/id_rsa.pub >> ~/.ssh/authorized_keys chmod 700 ~/.ssh/authorized_keys
To test the configuration, enter:
ssh -oNumberOfPasswordPrompts=0 <other_node> "echo success"
Verify that the echo
command is successful.
Step 2: Install a Controller High Availability Pair
Once you have set up the environment, you can install a Controller HA pair on the primary machine. To add an HA secondary to an existing standalone Controller deployment, you only need to verify that the user who runs the Controller has write access to the Controller home.
To install a Controller HA pair:
Check that you have fulfilled the Enterprise Console prerequisites before starting.
Open a browser and navigate to the GUI:
http(s)://<hostname>:<port>
9191 is the default port.
Verify that the credentials and hosts you want to use are added to the AppDynamics platform. For more information, see Administer the Enterprise Console.
On the Credential page, add the SSH credentials for the host you want to install the primary Controller on. You can also run the following command on the Enterprise Console host:
bin/platform-admin.sh add-credential --credential-name <name> --type <ssh> --user-name <username> --ssh-key-file <file path to the key file>
Remember to provide the private key file for the Enterprise Console machine when adding a credential.On the Hosts page, add the host using the credentials from above. You can also run the following command on the Enterprise Console host:
bin/platform-admin.sh add-hosts --hosts secondaryhost --credential <credential name>
Navigate to the Install homepage and click Custom Install.
Name the platform:
Enter a Name and the Installation Path for your platform.
The Installation Path is an absolute path under which all of the platform components are installed. The same path is used for all hosts added to the platform. Use a path which does not have any existing AppDynamics components installed under it.
The path you choose must be writeable, i.e. the user who installed the Enterprise Console should have write permissions to that folder. Also, the same path should be writable on all of the hosts that the Enterprise Console manages.Example path:
<path_to_AppD>/appdynamics/platform
Or run the following command on the Enterprise Console host:
bin/platform-admin.sh create-platform --name <platform_name> --installation-dir <platform_installation_directory>
Add two hosts:
For each of the two hosts, enter their host machine information: Host Name, Username, and Private Key.
This is the location onto which the Controllers will be installed. For more information about how to add credentials and hosts, see Administer the Enterprise Console.
- Install Controller:
- Select Install.
- Select a Profile size for your Controller. See Controller System Requirements for more information on the sizing requirements.
Enter the Controller Primary and Secondary hosts.
Enter the required Username and Passwords. The default Controller Admin Username is
admin
.If you do not install a Controller at this time, you can always do so later by navigating to the Controller page in the GUI and clicking Install Controller.Or run the following command on the Enterprise Console host:
bin/platform-admin.sh submit-job --service controller --job install --args controllerPrimaryHost=<primaryhost> controllerSecondaryHost=<secondaryhost> controllerAdminUsername=<user1> controllerAdminPassword=<password> controllerRootUserPassword=<rootpassword> mysqlRootPassword=<dbrootpassword>
Click Install.
Step 3: Activate the Controller High Availability Pair
This step ensures that the Enterprise Console is no longer in the critical path of the HA Controller failover process.
Open a command shell on the Enterprise Console host and enter:
platform-admin.sh submit-job --service controller --job activate-ha-modules
Your output should be similar to the following:
Step 4: Install as a Service
The Enterprise Console does not install the Controller as a service on Linux because it requires root user or sudo
privileges. However, the Enterprise Console copies the init
scripts on the Controller hosts in <controller_home>/controller-ha
. To complete the installation, manually run the following:
- Change directories to
<controller_home>/controller-ha
. As the user who owns the Controller folder, run:
set_mysql_password_file.sh -p <db root password> -s <secondary host>
CODE-s
copies the file to the secondary host. Enter the MySQL database root user password in the command.- Change directories to
<controller_home>/controller-ha/init
. Run
install-init.sh
as root user with one of the following options to select how to elevate the user privilege:-c
#use setuid c wrapper-s
#use sudo-p
#use prune wrapper-x
#use user privilege wrapper
You must run this script on both Controller HA pair servers. If you need to uninstall the service later, run the uninstall-init.sh
script.
After installing the Controller as a service, the following services are included:
appdcontroller-db
appdcontroller
The status and progress of the deployment's various components are written to the logs.
Convert a Standalone Controller to a Controller High Availability Pair
You can convert a standalone Controller to a Controller HA pair through the Enterprise Console GUI. Ensure that you have completed the prerequisites in the above Configure the Controller High Availability Pair Environment section that requires both the primary and secondary hosts to talk to each other via passwordless SSH.
Additionally, you can use incremental replication to add a secondary Controller. See Initiate Controller Database Incremental Replication for more information.
If you are starting from a fresh installation, you will need to first create a platform, then add two credentials and hosts for your HA pair.
To convert a standalone Controller to a Controller HA pair:
- Open the Enterprise Console GUI.
Verify that the credentials and hosts you want to use are added to the AppDynamics platform. For more information, see Administer the Enterprise Console.
On the Credential page, add the SSH credentials for the host you want to install the secondary Controller on. You can also run the following command on the Enterprise Console host:
bin/platform-admin.sh add-credential --credential-name <name> --type <ssh> --user-name <username> --ssh-key-file <file path to the key file>
Remember to provide the private key file for the Enterprise Console machine when adding a credential.On the Hosts page, add the host. You can also run the following command on the Enterprise Console host:
bin/platform-admin.sh add-hosts --hosts secondaryhost --credential <credential name>
The Enterprise Console uses this host for the HA pair.
On the Controller page, click Add Secondary Controller, and complete the wizard:
- Select the Controller Secondary Host that you added for the secondary Controller.
Optional: Enter the External URL. This is the external load balancer URL, which should reflect this format:
http(s)://<external.vip>:<port>
Enter the DB Root Password, and re-enter it for confirmation.
Ensure to provide the same passwords during the secondary server installation as those that you provided for the primary server.
- Select Submit.
Your HA pair will automatically set up, each with their own MySQL node.