PDFs

On this page:

Your Rating:
Results:
PatheticBadOKGoodOutstanding!
43 rates
The AppDynamics Enterprise Console can help you set up and administer a high availability (HA) deployment of AppDynamics Controllers.

The Enterprise Console HA deployment works on Linux systems only. Controller HA pairs are not available on Windows Enterprise Console machines.

This topic describes how to set up Controllers as a high availability pair. For installation and upgrade details, see Custom Install and Upgrade an HA Pair.

About the HA Deployment Using the Enterprise Console

The Enterprise Console automates HA-related setup and administration tasks for the Linux operating system. It does not require sudo privileges and can be deployed as a non-root user on Unix operating systems. It works with most flavors of Linux, including Ubuntu and Red Hat/CentOS. 

You can:

  • Configure Controllers in a high availability pair arrangement. 
  • Use the Enterprise Console to monitor the health of the primary Controller, App Server, and database, and fail over to the secondary when needed.
  • Use scripts that allows you to install the Controllers as a linux service, and gracefully stop and start service in the event of a machine reboot.
  • Fail over to a secondary Controller manually (for example, when you need to perform maintenance on the primary). 
  • Revive a Controller (restore a Controller as an HA secondary after its database is more than seven days behind the master as a replication slave).
  • Upgrade an HA pair.

Deploying Controllers as an HA pair ensures that service downtime in the event of a Controller machine failure is minimized. It also facilitates other administrative tasks, such as backing up data. For more background information, including the benefits of HA, see Controller High Availability (HA).

Before Starting

General guidelines and requirements for the environment in which you deploy HA are:

  • Two dedicated machines running Linux. The Linux operating systems can be Fedora-based Linux distributions (such as Red Hat or CentOS) or Debian-based Linux distributions (such as Ubuntu). 
  • In a Controller HA pair, a load balancer should route traffic from Controller clients (Controller UI users and App Agents) to the active Controller. Before starting, make sure that a load balancer is available in your environment and that you have the virtual IP for the Controller pair as presented by the load balancer.
  • Port number 3388 should be open between the machines in an HA pair.
  • The login shell must be bash (/bin/bash).
  • A network link connecting the HA hosts can support a high volume of data. The primary and secondary must be in the same data center, and a dedicated network link between the hosts is recommended.

  • The host machines should have identical directory structures.  
  • SSH keys on each host that allows SSH and sync operations by the AppDynamics user (instructions below). 
  • The hosts file (/etc/hosts) on both Controller machines should contain entries to support reverse lookups for the other node in the HA pair (instructions below).
  • Because Controller licenses are bound to the network MAC address of the host machine, the HA secondary Controller will need an additional HA license. You should request a secondary license for HA purposes in advance.  
  • While adding high availability hosts as part of the add host operation, you can determine and provide the remote user, which the Controller needs to be installed as. The platform path specified while creating the platform must be writable on the two HA hosts for the remote user specified during add host operation. 

User Privilege Escalation Requirements 

After installing a Controller high availability via the Enterprise Console, it is recommended to install MySQL as a linux service. This is to prevent against MySQL data integrity issues. Installing the Controller and MySQL as a Unix service will ensure that whenever the machine reboots, the service will be shut down and started gracefully.

  • /etc/sudoers.d/appdynamics contains entries to allow the AppDynamics user to access the /sbin/service utility using sudo without a password. This mechanism is not available if the AppDynamics user is authenticated by LDAP.
  • /sbin/appdservice is a setuid root program distributed in source form in <controller_home>/init/appdservice.c. It is written explicitly to support auditing by security audit systems. The install-init.sh script compiles and installs the program. It is executable only by the AppDynamics user and the root user. The script requires a C compiler to be available on the system. You can install a C compiler using the package manager for your operating system. For example, on Yum-based Linux distributions, you can use the following command to install the GNU Compiler, which includes a C compiler: 

    sudo yum install gcc  

Load Balancer Requirements and Considerations

Before setting up HA, a reverse proxy or load balancer needs to be available and configured to route traffic to the active Controller in the HA pair. Using a load balancer to route traffic between Controllers (rather than other approaches, such as DNS manipulation) ensures that a failover can occur quickly, without, for example, delays due to DNS caching on the agent machines. 

An HA deployment requires the following IP addresses:

  • One for the virtual IP of the Controller pair as presented by the load balancer used by clients, such as App Agents, to access the Controller. (Callout  in the following diagram.)
  • Two IP addresses for each Controller machine, one for the HTTP primary port interface () and one for the dedicated link interface between the Controllers () on each machine. The dedicated link is recommended but not mandatory.  
  • If the Controllers will reside within a protected, internal network behind the load balancer, you also need an additional internal virtual IP for the Controller within the internal network (). 

When configuring replication, you specify the external address at which Controller clients, such as app agents and UI users, will address the Controller at the load balancer. The Controllers themselves need to be able to reach this address as well. If the Controller will reside within a protected network relative to the load balancer, preventing them from reaching this address, there needs to be an internal VIP on the protected side that proxies the active Controller from within the network. This is specified using the -i parameter. 

The load balancer can check the availability of the Controller at the following address: 

http://<controller_host>:<port>/controller/rest/serverstatus

If the Controller is active, it responds to a GET request at this URL with an HTTP 200 response. The body of the response indicates the status of the Controller in the following manner: 

<serverstatus vendorid="" version="1">
   ...
   <available>true</available>
   ...

Ensure that the load balancer policy you configure for the Controller pair can send traffic to only a single Controller in the pair at a time (i.e., do not use round-robin or similar routing distribution policy at the load balancer). For more information about setting up a load balancer for the Controller, see Use a Reverse Proxy.

Setting Up the Controller High Availability Pair

Setting up high availability involves the following steps:

Step 1: Configure the Controller High Availability Pair Environment

The following sections provide more information on how to configure a few of the system requirements. They describe how to configure the settings on Red Hat Linux for a sample deployment. Note that the specific steps for configuring these requirements may differ on different systems. Consult the documentation for your system for details on that system. 

Host Reverse Lookups

Reliable symmetrical reverse host lookup needs to be set up on each machine. The best way to accomplish this is by placing the hostnames of the pair into the hosts files (/etc/hosts) on each machine. This is preferable over other approaches, namely using reverse DNS, which adds a point of failure.

To enable reverse host lookups, on each host: 
  1. In /etc/nsswitch.conf, put "files" before "dns" to have the hosts file entries take precedence over DNS. For example:
    hosts: files dns
  2. In /etc/hosts file, add an entry for each host in the HA pair, as in the following example:
    192.168.144.128 host1.domain.com host1
    192.168.144.137 host2.domain.com host2

    It is important to adhere to the correct format of /etc/hosts files in order to reduce errors. If you have both dotted hostnames and short versions, you need to list the dotted hostnames with the most dots first and the other versions subsequently. This should be done consistently for both HA server entries in each of the two /etc/hosts files. Note in the examples above that the aliases are listed last.

Set up the SSH key

SSH must be installed on both hosts in a way that gives the user who runs the Controller passwordless SSH access to the other Controller system in the HA pair. You can accomplish this by generating a key pair on each node, and placing the public key of the other Controller into the authorized keys (authorized_keys) file on each Controller.

The following steps illustrate how to perform this configuration. The instructions assume an AppDynamics user named appduser, and the Controller hostnames are node1, the active primary, and node2, the secondary. Adjust the instructions for your particular environment. Also note that you may not need to perform every step (for example, you may already have the .ssh directory and don't need to create a new one).

Although not shown here, some of the steps may prompt you for a password.  

On the primary (node1):
  1. Change to the AppDynamics user, appduser in our example:

    su - appduser  
  2. Create a directory for SSH artifacts (if it doesn't already exist) and set permissions on the directory, as follows:

    mkdir -p .ssh 
    chmod 700 .ssh
  3. Generate the RSA-formatted key: 

    ssh-keygen -t rsa -N "" -f .ssh/id_rsa
  4. Secure copy the key to the other Controller:

    scp .ssh/id_rsa.pub node2:/tmp
On the secondary (node2):
  1. As you did for node1, run these commands: 

    su - appduser 
    mkdir -p .ssh 
    chmod 700 .ssh
    ssh-keygen -t rsa -N "" -f .ssh/id_rsa
    scp .ssh/id_rsa.pub node1:/tmp

      

  2. Add the public key of node1 that you previously copied to the secondary Controller host's authorized keys and set permissions on the authorized keys file:

    cat /tmp/id_rsa.pub >> .ssh/authorized_keys
    chmod 700 ~/.ssh/authorized_keys
On the primary (node1) again:
  1. Move the secondary's public key to the authorized keys 

    cat /tmp/id_rsa.pub >> ~/.ssh/authorized_keys
    chmod 700 ~/.ssh/authorized_keys

To test the configuration, try this:

ssh -oNumberOfPasswordPrompts=0 <other_node> "echo success"

Make sure the echo command succeeds.

Step 2: Install the Primary Controller

Once you have set up the environment, you can install the primary Controller for high availability. This should be done on the primary machine. To add an HA secondary to an existing standalone Controller deployment, you only need to verify that the user who runs the Controller has write access to the Controller home, as described below. 

To install the primary Controller:

  1. Check that you have fulfilled the Enterprise Console prerequisites before starting.

  2. Open a browser and navigate to the GUI: 

    http(s)://<hostname>:<port>

    9191 is the default port.

  3. Verify that the credentials and hosts you want to use are added to the AppDynamics platform. For more information, see Administer the Enterprise Console.

    1. On the Credential page, add the SSH credentials for the host you want to install the primary Controller on. You can also run the following command on the Enterprise Console host:

      bin/platform-admin.sh add-credential --credential-name <name> --type <ssh> --user-name <username> --ssh-key-file <file path to the key file>
      Remember to provide the private key file for the Enterprise Console machine when adding a credential.
    2. On the Hosts page, add the host using the credentials from above. You can also run the following command on the Enterprise Console host:

      bin/platform-admin.sh add-hosts --hosts secondaryhost --credential <credential name>
  4. Navigate to the Install homepage and click Custom Install.

  5. Name the Platform:

    1. Enter a Name and the Installation Path for your platform.

      The Installation Path is an absolute path under which all of the platform components are installed. The same path is used for all hosts added to the platform. Use a path which does not have any existing AppDynamics components installed under it.
      The path you choose must be writeable, i.e. the user who installed the Enterprise Console should have write permissions to that folder. Also, the same path should be writable on all of the hosts that the Enterprise Console manages.

      Example path: <path_to_AppD>/AppDynamics/Platform

      Or run the following command on the Enterprise Console host:

      bin/platform-admin.sh create-platform --name <platform_name> --installation-dir <platform_installation_directory>
  6. Add a Host:

    1. Enter the host machine related information: Host Name, Username, and Private Key. This is where the Controller will be installed onto. For more information about how to add credentials and hosts, see Administer the Enterprise Console.

  7. Install Controller:
    1. Select Install.
    2. Select a Profile size for your Controller. See Controller System Requirements for more information on the sizing requirements.
    3. Enter the Controller Primary Host.

    4. Enter the required Username and Passwords. The default Controller Admin Username is admin.

      If you do not install a Controller at this time, you can always do so later by navigating to the Controller page in the GUI and clicking Install Controller.

    Or run the following command on the Enterprise Console host:

    bin/platform-admin.sh submit-job --service controller --job install --args controllerPrimaryHost=<primaryhost> controllerAdminUsername=<user1> controllerAdminPassword=<password> controllerRootUserPassword=<rootpassword> mysqlRootPassword=<dbrootpassword>
  8. Click Install.

Once the installation is finished, ensure that the Controller home and data directories are writable by the AppDynamics user. 

Use the ls command to verify write privileges. The output should look similar to the output below, which shows the current privileges for the sample AppDynamics user, appduser

ls -lad /opt/AppDynamics/Controller
drwxr-xr-x. 18 appduser users 4096 Jan 26 18:18 /opt/AppDynamics/Controller

Step 3: Set Up the Secondary Controller and Initiate Replication

You can add a secondary Controller to an existing platform through the Enterprise Console GUI. Ensure that you have completed the prerequisites in the above Configure the Controller High Availability Pair Environment section that requires both the primary and secondary hosts to talk to each other via passwordless SSH.

If you are starting from a fresh installation, you will need to first create a platform, then add two credentials and hosts for your HA pair.

To add a secondary Controller, use the following commands: 

  1. Open the Enterprise Console GUI.
  2. Verify that the credentials and hosts you want to use are added to the AppDynamics platform. For more information, see Administer the Enterprise Console.

    1. On the Credential page, add the SSH credentials for the host you want to install the secondary Controller on. You can also run the following command on the Enterprise Console host:

      bin/platform-admin.sh add-credential --credential-name <name> --type <ssh> --user-name <username> --ssh-key-file <file path to the key file>
      Remember to provide the private key file for the Enterprise Console machine when adding a credential.
    2. On the Hosts page, add the host. You can also run the following command on the Enterprise Console host:

      bin/platform-admin.sh add-hosts --hosts secondaryhost --credential <credential name>

      The Enterprise Console uses this host for the HA pair.

  3. On the Controller page, click Add Secondary Controller, and complete the wizard:

    1. Select the Controller Secondary Host that you added for the secondary Controller.
    2. Optional: Enter the External URL. This is the external load balancer URL, which should reflect this format: http(s)://<external.vip>:<port>
    3. Optional: Enter the Internal Virtual IP Address. This is the internal load balancer virtual IP address.

      Both the External URL and Internal Virtual IP Address fields are optional. However, if you specify the Internal Virtual IP Address, then you must also specify the External URL. You may leave the Internal Virtual IP Address blank though while providing the External URL.

    4. Enter the DB Root Password, and re-enter it for confirmation.

      Ensure the same passwords are provided during the secondary server installation that were provided for the primary server.

  4. Click Submit.

Your HA pair will automatically be set up for you, each with their own MySQL node.

Step 4: Install as a Service

The Enterprise Console does not install the Controller as a service on linux as it requires root user or sudo privileges. However, the Enterprise Console copies the init scripts on the Controller hosts in <controller>/init. You can run the following script manually to complete the installation:

  1. Change directories into <controller_home>/init.  
  2. Run save_mysql_passwd.sh as the user who owns the Controller folder, and enter the MySQL database root user password when prompted.

  3. Run install-init.sh as root user with one of the following options to choose how to elevate the user privilege:

    • -c #use setuid c wrapper
    • -s #use sudo
    • -p #use prune wrapper
    • -x #use user privilege wrapper

This script must be run on both Controller HA pair servers. If you need to uninstall the service later, use the uninstall-init.sh script. 

The status and progress logs of the deployment's various components are written to the logs.

  • No labels