Versions Compared

    Key

    • This line was added.
    • This line was removed.
    • Formatting was changed.
    Comment: Published by Scroll Versions from this space and version 20.5
    Sv translation
    languageen

    The AppDynamics Enterprise Console can help you This page describes how to set up and administer deploy Controllers as a high availability (HA) deployment of AppDynamics Controllerspair. For installation and upgrade details, see Custom Install and Upgrade an HA Pair.

    Note
    The Enterprise Console HA deployment works on Linux systems only. Controller HA pairs are not available on Windows machines using Enterprise Console.

    This topic describes how to set up and deploy Controllers as a high availability pair. For installation and upgrade details, see Custom Install and Upgrade an HA Pair.

    About the HA Deployment Using the Enterprise Console

    The Enterprise Console automates HA-related setup and administration tasks for the Linux operating system. It does not require sudo privileges and can be deployed as a non-root user on Unix operating systems. It works with most flavors of Linux, including Ubuntu and Red Hat/CentOS. 

    Note

    The servers of Controllers in an HA pair must be identical in terms of OS, CPU, RAM, and Disk. See Controller System Requirements.

    You can:

    • Configure Controllers in a high availability pair arrangement. 
    • Use the Enterprise Console to monitor the health of the primary Controller, App Server, and database, and fail over failover to the secondary when needed.
    • Use scripts that allows you to install the Controllers as a Linux service, and gracefully stop and start service in the event of a machine reboot.
    • Failover to a secondary Controller manually (for example, when you need to perform maintenance on the primary). 
    • Revive a Controller (restore a Controller as an HA secondary after its database is more than seven days behind the primary as a replica).
    • Set up a Controller HA pair.

    Deploying Controllers as an HA pair ensures that service downtime in the event of a Controller machine failure is minimized. It also facilitates other administrative tasks, such as backing up data. For more background information, including the benefits of HA, see Controller High Availability (HA).

    Before Starting

    For general guidelines and requirements on how to deploy HA in your environment, see Prerequisites for High Availability. Your environment must meet the prerequisites.

    Anchor
    privilegeescalation
    privilegeescalation
    User Privilege Escalation Requirements 

    After installing a Controller high availability via the Enterprise Console, it is recommended to install MySQL as a Linux service. This is to prevent against MySQL data integrity issues. Installing the Controller and MySQL as a Unix service will ensure that whenever the machine reboots, the service will be shut down and started gracefully.

    • /etc/sudoers.d/appdynamics contains entries to allow the AppDynamics user to access the /sbin/service utility using sudo without a password. This mechanism is not available if the AppDynamics user is authenticated by LDAP.
    • /sbin/appdservice is a setuid root program distributed in source form in <controller_home>/controller-ha/init/appdservice.c. It is written explicitly to support auditing by security audit systems. The install-init.sh script compiles and installs the program. It is executable only by the AppDynamics user and the root user. The script requires a C compiler to be available on the system. You can install a C compiler using the package manager for your operating system. For example, on Yum-based Linux distributions, you can use the following command to install the GNU Compiler, which includes a C compiler: 

      No Format
      sudo yum install gcc  

    Load Balancer Requirements and Considerations

    Before setting up HA, a reverse proxy or load balancer needs to be available and configured to route traffic to the active Controller in the HA pair. Using a load balancer to route traffic between Controllers (rather than other approaches, such as DNS manipulation) ensures that a failover can occur quickly, without, for example, delays due to DNS caching on the agent machines. 

    An HA deployment requires the following IP addresses:

    • One for the virtual IP of the Controller pair as presented by the load balancer used by clients, such as App Agents, to access the Controller. (Callout  in the following diagram.)
    • Two IP addresses for each Controller machine, one for the HTTP primary port interface () and one for the dedicated link interface between the Controllers () on each machine. The dedicated link is recommended but not mandatory.  
    • If the Controllers will reside within a protected, internal network behind the load balancer, you also need an additional internal virtual IP for the Controller within the internal network (). 

    HA deployment

    When configuring replication, you specify the external address at which Controller clients, such as app agents and UI users, will address the Controller at the load balancer. The Controllers themselves need to be able to reach this address as well. If the Controller will reside within a protected network relative to the load balancer, preventing them from reaching this address, there needs to be an internal VIP on the protected side that proxies the active Controller from within the network. This is specified using the -i parameter. 

    The load balancer can check the availability of the Controller at the following address: 

    http://<controller_host>:<port>/controller/rest/serverstatus

    If the Controller is active, it responds to a GET request at this URL with an HTTP 200 response. The body of the response indicates the status of the Controller in the following manner: 

    Code Block
    <serverstatus vendorid="" version="1">
       ...
       <available>true</available>
       ...

    Ensure that the load balancer policy you configure for the Controller pair can send traffic to only a single Controller in the pair at a time (i.e., do not use round-robin or similar routing distribution policy at the load balancer). For more information about setting up a load balancer for the Controller, see Use a Reverse Proxy.

    Set Up the Controller High Availability Pair

    To set up high availability: 

    Table of Contents
    maxLevel3
    minLevel3
    stylenone

    Anchor
    configureHAenvironment
    configureHAenvironment
    Step 1: Configure the Controller High Availability Pair Environment

    The following sections provide more information on how to configure a few of the system requirements. They describe how to configure the settings on Red Hat Linux as a sample deployment. Note that the specific steps for configuring these requirements may differ on different systems. Consult your system documentation for specific details. 

    Host Reverse Lookups

    You need to set up a reliable symmetrical reverse host lookup on each machine. To do this, enter the hostnames of the pair into the hosts files (/etc/hosts) on each machine. This is preferable over other approaches, such as using reverse DNS, which adds a point of failure.

    To enable reverse host lookups, on each host: 
    1. In /etc/nsswitch.conf, enter files before dns to have the hosts file entries take precedence over DNS. For example:
      hosts: files dns
    2. In /etc/hosts file, add an entry for each host in the HA pair. For example:
      192.168.144.128 host1.domain.com host1
      192.168.144.137 host2.domain.com host2

      Info

      To reduce errors, use the correct format of /etc/hosts files. If you have both dotted hostnames and short versions, you need to list the dotted hostnames with the most dots first and the other versions subsequently. This should be done consistently for both HA server entries in each of the two /etc/hosts files. Note that in the examples provided, the aliases are listed last.

    Anchor
    setupSSHkey
    setupSSHkey
    Set Up the SSH Key

    SSH must be installed on both hosts in a way that gives the user who runs the Controller passwordless SSH access to the other Controller system in the HA pair. You can accomplish this by generating a key pair on each node, and placing the public key of the other Controller into the authorized keys (authorized_keys) file on each Controller.

    The following steps describe how to perform this configuration. The instructions assume an AppDynamics user named appduser, and the Controller hostnames are node1, the active primary, and node2, the secondary. Adjust the instructions for your particular environment. Also note that you may not need to perform every step (for example, you may already have the .ssh directory and don't need to create a new one).

    Although not shown here, some of the steps may prompt you for a password.  

    On the primary (node1) host:
    1. Change to the AppDynamics user, appduser in our example:

      No Format
      su - appduser  
    2. Create a directory for SSH artifacts (if it doesn't already exist) and set permissions on the directory, as follows:

      No Format
      mkdir -p .ssh 
      chmod 700 .ssh
    3. Generate the RSA-formatted key: 

      No Format
      ssh-keygen -t rsa -N "" -f .ssh/id_rsa -m pem
    4. Secure copy the key to the other Controller:

      No Format
      scp .ssh/id_rsa.pub node2:/tmp
    On the secondary (node2) host:
    1. As you did for node1, run these commands: 

      No Format
      su - appduser 
      mkdir -p .ssh 
      chmod 700 .ssh
      ssh-keygen -t rsa -N "" -f .ssh/id_rsa -m pem
      scp .ssh/id_rsa.pub node1:/tmp
    2. Add the public key of node1 that you previously copied to the secondary Controller host's authorized keys and set permissions on the authorized keys file:

      No Format
      cat /tmp/id_rsa.pub >> .ssh/authorized_keys
      chmod 700 ~/.ssh/authorized_keys
    On the primary (node1) again:
    1. Move the secondary's public key to the authorized keys 

      No Format
      cat /tmp/id_rsa.pub >> ~/.ssh/authorized_keys
      chmod 700 ~/.ssh/authorized_keys

    To test the configuration, enter:

    No Format
    ssh -oNumberOfPasswordPrompts=0 <other_node> "echo success"

    Verify that the echo command is successful.

    Anchor
    install-primary
    install-primary
    Step 2: Install a Controller High Availability Pair

    Once you have set up the environment, you can install a Controller HA pair on the primary machine. To add an HA secondary to an existing standalone Controller deployment, you only need to verify that the user who runs the Controller has write access to the Controller home. 

    To install a Controller HA pair:

    1. Check that you have fulfilled the Enterprise Console prerequisites before starting.

    2. Open a browser and navigate to the GUI: 

      No Format
      http(s)://<hostname>:<port>

      9191 is the default port.

    3. Verify that the credentials and hosts you want to use are added to the AppDynamics platform. For more information, see Administer the Enterprise Console.

      1. On the Credential page, add the SSH credentials for the host you want to install the primary Controller on. You can also run the following command on the Enterprise Console host:

        No Format
        bin/platform-admin.sh add-credential --credential-name <name> --type <ssh> --user-name <username> --ssh-key-file <file path to the key file>
        Info
        Remember to provide the private key file for the Enterprise Console machine when adding a credential.
      2. On the Hosts page, add the host using the credentials from above. You can also run the following command on the Enterprise Console host:

        No Format
        bin/platform-admin.sh add-hosts --hosts secondaryhost --credential <credential name>
    4. Navigate to the Install homepage and click Custom Install.

    5. Name the platform:

      1. Enter a Name and the Installation Path for your platform.

        Info

        The Installation Path is an absolute path under which all of the platform components are installed. The same path is used for all hosts added to the platform. Use a path which does not have any existing AppDynamics components installed under it.
        The path you choose must be writeable, i.e. the user who installed the Enterprise Console should have write permissions to that folder. Also, the same path should be writable on all of the hosts that the Enterprise Console manages.

        Example path: <path_to_AppD>/appdynamics/platform

        Or run the following command on the Enterprise Console host:

        No Format
        bin/platform-admin.sh create-platform --name <platform_name> --installation-dir <platform_installation_directory>
    6. Add two hosts:

      1. For each of the two hosts, enter their host machine information: Host Name, Username, and Private Key. 
        This is the location onto which the Controllers will be installed. For more information about how to add credentials and hosts, see Administer the Enterprise Console.

    7. Install Controller:
      1. Select Install.
      2. Select a Profile size for your Controller. See Controller System Requirements for more information on the sizing requirements.
      3. Enter the Controller Primary and Secondary hosts.

      4. Enter the required Username and Passwords. The default Controller Admin Username is admin.

        Info
        If you do not install a Controller at this time, you can always do so later by navigating to the Controller page in the GUI and clicking Install Controller.

        Or run the following command on the Enterprise Console host:

        No Format
        bin/platform-admin.sh submit-job --service controller --job install --args controllerPrimaryHost=<primaryhost> controllerSecondaryHost=<secondaryhost> controllerAdminUsername=<user1> controllerAdminPassword=<password> controllerRootUserPassword=<rootpassword> mysqlRootPassword=<dbrootpassword>
    8. Click Install.

    Anchor
    activatehapair
    activatehapair
    Step 3: Activate the Controller High Availability Pair

    This step ensures that the Enterprise Console is no longer in the critical path of the HA Controller failover process.

    Open a command shell on the EConsole Enterprise Console host and enter:

    Code Block
    platform-admin.sh submit-job --service controller  --job activate-ha-modules


    Your output should be similar to the following:

    Output

    Anchor
    installservice
    installservice
    Step 4: Install as a Service

    The Enterprise Console does not install the Controller as a service on Linux because it requires root user or sudo privileges. However, the Enterprise Console copies the init scripts on the Controller hosts in <controller_home>/controller-ha. To complete the installation, manually run the following:

    1. Change directories to <controller_home>/controller-ha.  
    2. As the user who owns the Controller folder, run:

      Code Block
      set_mysql_password_file.sh -p <db root password> -s <secondary host>

      -s copies the file to the secondary host. Enter the MySQL database root user password when prompted.

    3. Change directories to <controller_home>/controller-ha/init.  
    4. Run install-init.sh as root user with one of the following options to select how to elevate the user privilege:

      • -c #use setuid c wrapper
      • -s #use sudo
      • -p #use prune wrapper
      • -x #use user privilege wrapper

    You must run this script on both Controller HA pair servers. If you need to uninstall the service later, run the uninstall-init.sh script. 

    The status and progress of the deployment's various components are written to the logs.

    Anchor
    secondary-replication
    secondary-replication
    Convert a Standalone Controller to a Controller High Availability Pair

    You can convert a standalone Controller to a Controller HA pair through the Enterprise Console GUI. Ensure that you have completed the prerequisites in the above Configure the Controller High Availability Pair Environment section that requires both the primary and secondary hosts to talk to each other via passwordless SSH.

    Additionally, you can use incremental replication to add a secondary Controller. See Initiate Controller Database Incremental Replication for more information.

    If you are starting from a fresh installation, you will need to first create a platform, then add two credentials and hosts for your HA pair.

    To convert a standalone Controller to a Controller HA pair: 

    1. Open the Enterprise Console GUI.
    2. Verify that the credentials and hosts you want to use are added to the AppDynamics platform. For more information, see Administer the Enterprise Console.

      1. On the Credential page, add the SSH credentials for the host you want to install the secondary Controller on. You can also run the following command on the Enterprise Console host:

        No Format
        bin/platform-admin.sh add-credential --credential-name <name> --type <ssh> --user-name <username> --ssh-key-file <file path to the key file>
        Info
        Remember to provide the private key file for the Enterprise Console machine when adding a credential.
      2. On the Hosts page, add the host. You can also run the following command on the Enterprise Console host:

        No Format
        bin/platform-admin.sh add-hosts --hosts secondaryhost --credential <credential name>

        The Enterprise Console uses this host for the HA pair.

    3. On the Controller page, click Add Secondary Controller, and complete the wizard:

      1. Select the Controller Secondary Host that you added for the secondary Controller.
      2. Optional: Enter the External URL. This is the external load balancer URL, which should reflect this format: http(s)://<external.vip>:<port>

      3. Enter the DB Root Password, and re-enter it for confirmation.

        Note

        Ensure to provide the same passwords during the secondary server installation as those that you provided for the primary server.

    4. Select Submit.

    Your HA pair will automatically set up, each with their own MySQL node.


    ...