As part of the new Controller HA Module in the Enterprise Console, Enterprise Console no longer manages the Controller failover. Instead, the new HA Module installs the Controller watchdog on the Controller hosts and the Controller hosts are now responsible for performing a failover. The new HA Module is packaged in Enterprise Console and allows you to migrate seamlessly from older HA implementations, such as the HA Toolkit (HATK), to this new HA Module. The new HA Module is included with the latest version of Enterprise Console, and is installed when you install or upgrade your Controller. However, it is not activated until you migrate an HA pair.

Who Should Use the HA Module?

  • If you are a new user just starting to implement Splunk AppDynamics HA, then you should use the HA Module (this is the default option).

  • If you are an existing user and you prefer to use the Enterprise Console UI integration instead of the command line (CLI) and HATK, then you should use the HA Module.

  • If you are an existing user and you do not want to change existing DevOps, conduct additional training, or would like to continue using HATK features, then use HATK.

If your Controller version is earlier than version 4.5.13, then you must use the HA Toolkit (HATK) instead of the HA Module to install and configure High Availability.

Before You Begin

In addition the Prerequisites for High Availability, ensure the following requirements are met:

  • Both servers have the same Linux username and groupname.

  • Both servers have the same Splunk AppDynamics directory structure, same Splunk AppDynamics pathname on both servers without using symbolic links.

Migrate a Controller High Availability Pair

This section describes the Enterprise Console and Controller HA Pair upgrade processes required for two different scenarios:

Scenario One: Upgrade Deployment that uses Enterprise Console and HA Tookit

  1. Download and install the latest version of Enterprise Console from the Downloads page.
  2. Open Enterprise Console and select Controller. In the Controller list, it displays only the primary Controller of the HA Pair. If you have an existing pair of HA Controllers which are not managed by Enterprise Console, you need to forget the Controller from within Enterprise Console. It will only show the primary Controller.

  3. Select the Controller and select Remove. 

    In the Remove Controller dialog, deselect Remove Binaries. At this point, you are just removing the Controller from Enterprise Console but not the Splunk AppDynamics On-Premises software that is running on the HA Controllers.

  4. Before we can activate the new HA Module, we need to discover both Controllers from within Enterprise Console. Once the remove Controller job completes, select Controller > Discover & Upgrade Controller. 
  5. Note that in the Discover Controller dialog, you specify the hostnames of both the Controller Primary Host, and the Controller Secondary Host, or replica.
     
  6. Complete the fields of the Discover Controller dialog and select Continue. This adds the HA Pair to Enterprise Console where you can then manage and upgrade.
    Access the Jobs page to see several jobs that have completed successfully. The Controller Discover & Upgrade Job will take a while to complete. Select View Details to track the progress of the tasks involved to discover and upgrade the Controller. 


    When the discovery and upgrade process for the HA Pair is complete, the Controller page should be similar to the following:


    Now, Enterprise Console knows about the HA Pair and has copied the new HA Module to the primary and secondary Controllers in the HA Pair.

  7. To activate the HA Pair, open a command shell on the Enterprise Console host and enter the following:

    platform-admin.sh submit-job --service controller  --job activate-ha-modules
    CODE


    You should see output similar to the following: 

  8. Verify that the Controller HA failover is working by terminating the Primary Controller and allow Controller Watchdog to trigger a failover. Then, wait a few minutes to ensure Splunk AppDynamics agents are reporting metrics to the Controller.

Scenario Two: Upgrade Deployment Managed by Enterprise Console Alone

  1. Download and install the latest version of Enterprise Console from the Downloads page.
  2. Upgrade the installed version of Enterprise Console by running the installer on the Enterprise Console host.
  3. Upgrade both Controllers by selecting Upgrade Controller.

  4. To activate the HA Pair, open a command shell on the Enterprise Console host and enter:

    platform-admin.sh submit-job --service controller  --job activate-ha-modules
    CODE


    You should see output similar to the following: 

  5. Verify that the Controller HA failover is working by terminating the Primary Controller and allows Controller Watchdog to trigger a failover. Then wait a few minutes to ensure Splunk AppDynamics agents are reporting metrics to the Controller. 
  6. When you enable the HA module, the HATK folder is renamed which results in broken services. 
    To re-install the services correctly on a pair of HA servers, you must enter the following on the primary and secondary servers:
    1. Log in with root privileges.
    2. Change directories to: <controller_home>/controller-ha/init

    3. Using the HA module, you must first remove the services installed by HATK by running this script:

      ./uninstall-init.sh
      CODE

    4. Run install-init.sh and install the same services with one of the following options to elevate the user privilege:

      • -c #use setuid c wrapper
      • -s #use sudo
      • -p #use prune wrapper
      • -x #use user privilege wrapper

Watchdog Widget 

After activating the HA Module, a new widget displays in the Enterprise Console UI indicating the Controller Watchdog status. It has three states:

Controller Watchdog StatusDefinition
The Controller watchdog process is constantly checking the health of your primary Controller. No user action is required.
In the event that your primary Controller goes down, the failover is not automatic. We recommend that you start the Controller watchdog using the Start Controller Watchdog option in Enterprise Console.
Failover is currently in progress where your primary Controller and secondary Controller switch roles 

 
Prior to performing any maintenance operations on your Primary Controller host (which could result in stopping the Controller), we recommend that you select Stop Controller Watchdog to prevent a failover from initiating. Use the Enterprise Console UI to start and stop watchdog by clicking the watchdog widget, and then selecting Stop or Start Controller Watchdog. 

 

Start and Stop the Controller Watchdog with CLI

You can start and stop the Controller Watchdog using the following commands:

Stop the Controller Watchdog

./platform-admin.sh submit-job --job stop-controller-watchdog --service controller
CODE

Start the Controller Watchdog

./platform-admin.sh submit-job --job start-controller-watchdog --service controller
CODE