Set Up a High Availability Deployment

This page describes how to set up and deploy Controllers as a high availability pair. For installation and upgrade details, see Custom Install and Upgrade an HA Pair.

The Enterprise Console HA deployment works on Linux systems only. Controller HA pairs are not available on Windows machines using Enterprise Console.

About the HA Deployment Using the Enterprise Console

The Enterprise Console automates HA-related setup and administration tasks for the Linux operating system. It does not require sudo privileges and can be deployed as a non-root user on Unix operating systems. It works with most flavors of Linux, including Ubuntu and Red Hat/CentOS.

The servers of Controllers in an HA pair must be identical in terms of OS, CPU, RAM, and Disk. See Controller System Requirements.

You can:

Configure Controllers in a high availability pair arrangement.
Use the Enterprise Console to monitor the health of the primary Controller, App Server, and database, and failover to the secondary when needed.
Use scripts that allows you to install the Controllers as a Linux service, and gracefully stop and start service in the event of a machine reboot.
Failover to a secondary Controller manually (for example, when you need to perform maintenance on the primary).
Revive a Controller (restore a Controller as an HA secondary after its database is more than seven days behind the primary as a replica).
Set up a Controller HA pair.

Deploying Controllers as an HA pair ensures that service downtime in the event of a Controller machine failure is minimized. It also facilitates other administrative tasks, such as backing up data. For more background information, including the benefits of HA, see Controller High Availability (HA).

Before Starting

For general guidelines and requirements on how to deploy HA in your environment, see Prerequisites for High Availability. Your environment must meet the prerequisites.

User Privilege Escalation Requirements

After installing a Controller high availability via the Enterprise Console, it is recommended to install MySQL as a Linux service. This is to prevent against MySQL data integrity issues. Installing the Controller and MySQL as a Unix service will ensure that whenever the machine reboots, the service will be shut down and started gracefully.

/etc/sudoers.d/appdynamics contains entries to allow the AppDynamics user to access the /sbin/service utility using sudo without a password. This mechanism is not available if the AppDynamics user is authenticated by LDAP.
/sbin/appdservice is a setuid root program distributed in source form in <controller_home>/controller-ha/init/appdservice.c. It is written explicitly to support auditing by security audit systems. The install-init.sh script compiles and installs the program. It is executable only by the AppDynamics user and the root user. The script requires a C compiler to be available on the system. You can install a C compiler using the package manager for your operating system. For example, on Yum-based Linux distributions, you can use the following command to install the GNU Compiler, which includes a C compiler:
```
sudo yum install gcc  
```

Load Balancer Requirements and Considerations

Before setting up HA, a reverse proxy or load balancer needs to be available and configured to route traffic to the active Controller in the HA pair. Using a load balancer to route traffic between Controllers (rather than other approaches, such as DNS manipulation) ensures that a failover can occur quickly, without, for example, delays due to DNS caching on the agent machines.

An HA deployment requires the following IP addresses:

One for the virtual IP of the Controller pair as presented by the load balancer used by clients, such as App Agents, to access the Controller. (Callout in the following diagram.)
Two IP addresses for each Controller machine, one for the HTTP primary port interface () and one for the dedicated link interface between the Controllers () on each machine. The dedicated link is recommended but not mandatory.
If the Controllers will reside within a protected, internal network behind the load balancer, you also need an additional internal virtual IP for the Controller within the internal network ().

HA deployment

When configuring replication, you specify the external address at which Controller clients, such as app agents and UI users, will address the Controller at the load balancer. The Controllers themselves need to be able to reach this address as well. If the Controller will reside within a protected network relative to the load balancer, preventing them from reaching this address, there needs to be an internal VIP on the protected side that proxies the active Controller from within the network. This is specified using the -i parameter.

The load balancer can check the availability of the Controller at the following address:

http://<controller_host>:<port>/controller/rest/serverstatus

If the Controller is active, it responds to a GET request at this URL with an HTTP 200 response. The body of the response indicates the status of the Controller in the following manner:

<serverstatus vendorid="" version="1">
   ...
   <available>true</available>
   ...

CODE

Ensure that the load balancer policy you configure for the Controller pair can send traffic to only a single Controller in the pair at a time (i.e., do not use round-robin or similar routing distribution policy at the load balancer). For more information about setting up a load balancer for the Controller, see Use a Reverse Proxy.

Set Up the Controller High Availability Pair

To set up high availability:

Step 1: Configure the Controller High Availability Pair Environment

The following sections provide more information on how to configure a few of the system requirements. They describe how to configure the settings on Red Hat Linux as a sample deployment. Note that the specific steps for configuring these requirements may differ on different systems. Consult your system documentation for specific details.

Host Reverse Lookups

You need to set up a reliable symmetrical reverse host lookup on each machine. To do this, enter the hostnames of the pair into the hosts files (/etc/hosts) on each machine. This is preferable over other approaches, such as using reverse DNS, which adds a point of failure.

To enable reverse host lookups, on each host:

In /etc/nsswitch.conf, enter files before dns to have the hosts file entries take precedence over DNS. For example:
hosts: files dns
In /etc/hosts file, add an entry for each host in the HA pair. For example:
192.168.144.128 host1.domain.com host1
192.168.144.137 host2.domain.com host2

To reduce errors, use the correct format of /etc/hosts files. If you have both dotted hostnames and short versions, you need to list the dotted hostnames with the most dots first and the other versions subsequently. This should be done consistently for both HA server entries in each of the two /etc/hosts files. Note that in the examples provided, the aliases are listed last.

Set Up the SSH Key

SSH must be installed on both hosts in a way that gives the user who runs the Controller passwordless SSH access to the other Controller system in the HA pair. You can accomplish this by generating a key pair on each node, and placing the public key of the other Controller into the authorized keys (authorized_keys) file on each Controller.

The following steps describe how to perform this configuration. The instructions assume an AppDynamics user named appduser, and the Controller hostnames are node1, the active primary, and node2, the secondary. Adjust the instructions for your particular environment. Also note that you may not need to perform every step (for example, you may already have the .ssh directory and don't need to create a new one).

Although not shown here, some of the steps may prompt you for a password.

On the primary (node1) host:

Change to the AppDynamics user, appduser in our example:
```
su - appduser  
```
Create a directory for SSH artifacts (if it doesn't already exist) and set permissions on the directory, as follows:
```
mkdir -p .ssh 
chmod 700 .ssh
```

Generate the RSA-formatted key:

ssh-keygen -t rsa -N "" -f .ssh/id_rsa -m pem

Secure copy the key to the other Controller:
```
scp .ssh/id_rsa.pub node2:/tmp
```

On the secondary (node2) host:

As you did for node1, run these commands:

su - appduser 
mkdir -p .ssh 
chmod 700 .ssh
ssh-keygen -t rsa -N "" -f .ssh/id_rsa -m pem
scp .ssh/id_rsa.pub node1:/tmp

Add the public key of node1 that you previously copied to the secondary Controller host's authorized keys and set permissions on the authorized keys file:
```
cat /tmp/id_rsa.pub >> .ssh/authorized_keys
chmod 700 ~/.ssh/authorized_keys
```

On the primary (node1) again:

Move the secondary's public key to the authorized keys

cat /tmp/id_rsa.pub >> ~/.ssh/authorized_keys
chmod 700 ~/.ssh/authorized_keys

To test the configuration, enter:

ssh -oNumberOfPasswordPrompts=0 <other_node> "echo success"

Verify that the echo command is successful.

Step 2: Install a Controller High Availability Pair

Once you have set up the environment, you can install a Controller HA pair on the primary machine. To add an HA secondary to an existing standalone Controller deployment, you only need to verify that the user who runs the Controller has write access to the Controller home.

To install a Controller HA pair:

Check that you have fulfilled the Enterprise Console prerequisites before starting.
Open a browser and navigate to the GUI:
```
http(s)://<hostname>:<port>
```
9191 is the default port.
Verify that the credentials and hosts you want to use are added to the AppDynamics platform. For more information, see Administer the Enterprise Console.
1. On the Credential page, add the SSH credentials for the host you want to install the primary Controller on. You can also run the following command on the Enterprise Console host:
```
bin/platform-admin.sh add-credential --credential-name <name> --type <ssh> --user-name <username> --ssh-key-file <file path to the key file>
```
  Remember to provide the private key file for the Enterprise Console machine when adding a credential.
2. On the Hosts page, add the host using the credentials from above. You can also run the following command on the Enterprise Console host:
```
bin/platform-admin.sh add-hosts --hosts secondaryhost --credential <credential name>
```
Navigate to the Install homepage and click Custom Install.
Name the platform:
1. Enter a Name and the Installation Path for your platform.
  
  The Installation Path is an absolute path under which all of the platform components are installed. The same path is used for all hosts added to the platform. Use a path which does not have any existing AppDynamics components installed under it.
  The path you choose must be writeable, i.e. the user who installed the Enterprise Console should have write permissions to that folder. Also, the same path should be writable on all of the hosts that the Enterprise Console manages.
  Example path: <path_to_AppD>/appdynamics/platform
  
  Or run the following command on the Enterprise Console host:
```
bin/platform-admin.sh create-platform --name <platform_name> --installation-dir <platform_installation_directory>
```
Add two hosts:
1. For each of the two hosts, enter their host machine information: Host Name, Username, and Private Key.
  This is the location onto which the Controllers will be installed. For more information about how to add credentials and hosts, see Administer the Enterprise Console.
Install Controller:
1. Select Install.
2. Select a Profile size for your Controller. See Controller System Requirements for more information on the sizing requirements.
3. Enter the Controller Primary and Secondary hosts.
4. Enter the required Username and Passwords. The default Controller Admin Username is admin.
  
  If you do not install a Controller at this time, you can always do so later by navigating to the Controller page in the GUI and clicking Install Controller.
  
  Or run the following command on the Enterprise Console host:
```
bin/platform-admin.sh submit-job --service controller --job install --args controllerPrimaryHost=<primaryhost> controllerSecondaryHost=<secondaryhost> controllerAdminUsername=<user1> controllerAdminPassword=<password> controllerRootUserPassword=<rootpassword> mysqlRootPassword=<dbrootpassword>
```
Click Install.

Step 3: Activate the Controller High Availability Pair

This step ensures that the Enterprise Console is no longer in the critical path of the HA Controller failover process.

Open a command shell on the Enterprise Console host and enter:

platform-admin.sh submit-job --service controller  --job activate-ha-modules

CODE

Your output should be similar to the following:

Output

Step 4: Install as a Service

The Enterprise Console does not install the Controller as a service on Linux because it requires root user or sudo privileges. However, the Enterprise Console copies the init scripts on the Controller hosts in <controller_home>/controller-ha. To complete the installation, manually run the following:

Change directories to <controller_home>/controller-ha.
As the user who owns the Controller folder, run:
```
set_mysql_password_file.sh -p <db root password> -s <secondary host>
```
CODE
-s copies the file to the secondary host. Enter the MySQL database root user password in the command.
Change directories to <controller_home>/controller-ha/init.
Run install-init.sh as root user with one of the following options to select how to elevate the user privilege:
- -c #use setuid c wrapper
- -s #use sudo
- -p #use prune wrapper
- -x #use user privilege wrapper

You must run this script on both Controller HA pair servers. If you need to uninstall the service later, run the uninstall-init.sh script.

The status and progress of the deployment's various components are written to the logs.

Convert a Standalone Controller to a Controller High Availability Pair

You can convert a standalone Controller to a Controller HA pair through the Enterprise Console GUI. Ensure that you have completed the prerequisites in the above Configure the Controller High Availability Pair Environment section that requires both the primary and secondary hosts to talk to each other via passwordless SSH.

Additionally, you can use incremental replication to add a secondary Controller. See Initiate Controller Database Incremental Replication for more information.

If you are starting from a fresh installation, you will need to first create a platform, then add two credentials and hosts for your HA pair.

To convert a standalone Controller to a Controller HA pair:

Open the Enterprise Console GUI.
Verify that the credentials and hosts you want to use are added to the AppDynamics platform. For more information, see Administer the Enterprise Console.
1. On the Credential page, add the SSH credentials for the host you want to install the secondary Controller on. You can also run the following command on the Enterprise Console host:
```
bin/platform-admin.sh add-credential --credential-name <name> --type <ssh> --user-name <username> --ssh-key-file <file path to the key file>
```
  Remember to provide the private key file for the Enterprise Console machine when adding a credential.
2. On the Hosts page, add the host. You can also run the following command on the Enterprise Console host:
```
bin/platform-admin.sh add-hosts --hosts secondaryhost --credential <credential name>
```
  The Enterprise Console uses this host for the HA pair.
On the Controller page, click Add Secondary Controller, and complete the wizard:
1. Select the Controller Secondary Host that you added for the secondary Controller.
2. Optional: Enter the External URL. This is the external load balancer URL, which should reflect this format: http(s)://<external.vip>:<port>
3. Enter the DB Root Password, and re-enter it for confirmation.
  
  Ensure to provide the same passwords during the secondary server installation as those that you provided for the primary server.
Select Submit.

Your HA pair will automatically set up, each with their own MySQL node.