If you are using the HA ToolKit (HATK) and made any customizations to it, AppDynamics recommends that you review your particular situation and determine if you should proceed with the migration. For more details, see Migrate to the New HA Module Using Enterprise Console.

Steps 1 through 3 consist of different procedures depending on your HA Controller pair deployment:

  • Follow Option 1 - Discover and Upgrade for deployments that are not managed by the Enterprise Console.
    Use the discover and upgrade job to onboard your HA Controller pair to the Enterprise Console before upgrading the primary Controller.
  • Follow Option 2 - Upgrade for HA pair deployments that are managed by the Enterprise Console.
    Use the upgrade option to upgrade your primary Controller managed by the current Enterprise Console instance.

To upgrade using the CLI:


Step 1: Prepare the Upgrade

  1. Download the latest release from AppDynamics Download Center.
    If you prefer to use the Linux shell, see Download AppDynamics Software.

  2. Install the Enterprise Console.

  1. Download the latest release from AppDynamics Download Center.
    If you prefer to use the Linux shell, see Download AppDynamics Software.

  2. Upgrade the Enterprise Console.


Step 2: Perform a Failover and Check the HA Pair State

To begin the upgrade, perform a failover to the secondary Controller. Since the primary Controller is known to be working, this provides a stable configuration to fail back to in case of upgrade issues.

To perform a failover and check the HA pair state:

  1. Run the HA Toolkit failover.sh script on the secondary Controller to initiate failover.
    This kills the watchdog process, starts the app server on the secondary, and makes the database on the secondary the replication primary.

  2. Run the Linux service utility on either node to report the current state:

    service appdcontroller status
  3. Shut down the HA Toolkit (HATK) implementation.

  1. Run the following command on the Enterprise Console host to initiate a failover:

    bin/platform-admin.sh submit-job --service controller --job ha-failover --platform-name <name_of_the_platform>

  2. Run the following command on the Enterprise Console host to check the state:

    bin/platform-admin.sh list-platform-service --platform-name <name_of_the_platform>

    The desired output is "Controller: provisioned".

  3. Run the following command on the Enterprise Console host to check the Controller versions:

    bin/platform-admin.sh list-node --service controller --platform-name <name_of_the_platform>

    The output will display the host versions.

    Sample Output

    Available nodes in the controller service:
	Controller: role Primary, host 172.12.0.1, version 4.5.0.2, state running
         MySQL: role Primary, host 172.12.0.1, version 5.7.21 state running      
	Controller: role Secondary, host 172.12.0.2, version 4.5.0.2, state stopped
         MySQL: role Secondary, host 172.12.0.2, version 5.7.21, state running
    CODE

    The primary Controller's Appserver and MySQL should be running before upgrading. The secondary Controller's MySQL must also be running.

Step 3: Upgrade the Primary Controller

You can upgrade the primary Controller using one of the following commands:

To discover and upgrade the primary Controller, run the following command on the Enterprise Console host:

bin/platform-admin.sh submit-job --service controller --job  discover-and-upgrade-primary --platform-name <name_of_the_platform> --target-version=<version> --args  controllerPrimaryHost=<primary_host> controllerSecondaryHost=<secondary_host> destinationDirectory=<controller_home_dir> controllerRootUserPassword=<controller_root_password> mysqlRootPassword=<mysql_root_password> controllerDBPassword=<controller_db_user_password>

To upgrade the primary Controller, run the following command on the Enterprise Console host:

bin/platform-admin.sh submit-job --service controller --job upgrade-primary --platform-name <name_of_the_platform> --target-version=<version> --args controllerRootUserPassword=<controller_root_password> mysqlRootPassword=<mysql_root_password>
See Troubleshooting the Upgrade if the primary upgrade fails.

Step 4: Verify the Primary Upgrade

A primary upgrade success message displays if the upgrade is successful. However, AppDynamics recommends that you perform your own in-house verification on the Controller before proceeding to the secondary upgrade. For example, you can check that your agents are continuing to report to your Controller.

The following describes the expected state in the Enterprise Console.

Run the following commands on the Enterprise Console host:

bin/platform-admin.sh list-platform-service --platform-name <name_of_the_platform>

The desired output is Controller: primary_upgraded.

bin/platform-admin.sh list-node --service controller --platform-name <name_of_the_platform>

The output will display the host versions.

Sample Output

Available nodes in the controller service:
	Controller: role Primary, host 172.12.0.1, version 4.5.5.0, state running
         MySQL: role Primary, host 172.12.0.1, version 5.7.24 state running      
	Controller: role Secondary, host 172.12.0.2, version 4.5.0.2, state stopped
         MySQL: role Secondary, host 172.12.0.2, version 5.7.21, state running
CODE

Check that all hosts show the new versions, that the Controller is running on the primary and stopped on the secondary, and that MySQL is running on both hosts. If you are not satisfied with the upgrade, see Verify the Primary Upgrade is Unsatisfactory.

Step 5: Upgrade the Secondary Controller

To upgrade the secondary Controller, run the the upgrade-secondary command on the Enterprise Console host:

bin/platform-admin.sh submit-job --service controller --job upgrade-secondary --platform-name <name_of_the_platform> --args controllerRootUserPassword=<controller_root_password> mysqlRootPassword=<mysql_root_password>

If the secondary Controller upgrade fails, see Upgrade the Secondary Controller Fails for possible recovery options. 

Step 6: Verify the Secondary Upgrade

A secondary upgrade success message displays if the upgrade is successful. However, AppDynamics recommends that you perform your own in-house verification on the Controller before completing the upgrade.

The following describes the expected state in the Enterprise Console.

Run the following commands on the Enterprise Console host:

bin/platform-admin.sh list-platform-service --platform-name <name_of_the_platform>

The desired output is "Controller: provisioned".

bin/platform-admin.sh list-node --service controller --platform-name <name_of_the_platform>

The output will display the host versions.

Sample Output

Available nodes in the controller service:
	Controller: role Primary, host 172.12.0.1, version 4.5.5.0, state running
         MySQL: role Primary, host 172.12.0.1, version 5.7.24 state running      
	Controller: role Secondary, host 172.12.0.2, version 4.5.5.0, state stopped
         MySQL: role Secondary, host 172.12.0.2, version 5.7.24, state running
CODE

Ensure that the secondary host version has been upgraded. Also, the primary should be in a running state, while the secondary Controller appserver should remain stopped. However, its MySQL process should be running.

Troubleshooting the Upgrade

Failover Issues

If you experience failover issues before upgrading, determine the condition of the secondary Controller. You may need to fix a broken secondary Controller before you attempt an upgrade.  

The Primary Controller Upgrade Fails

If the primary Controller failed for discover and upgrade, try the following recovery options:

  • Option 1: Retry the discover and upgrade by entering the following command on the Enterprise Console host:

    bin/platform-admin.sh submit-job --service controller --job  discover-and-upgrade-primary --platform-name <name_of_the_platform> --args  controllerPrimaryHost=<primary_host> controllerSecondaryHost=<secondary_host> destinationDirectory=<controller_home_dir> controllerRootUserPassword=<controller_root_password> mysqlRootPassword=<mysql_root_password> useCheckpoint=true
    CODE

  • Option 2: Perform a manual failover using the HA Toolkit (HATK). Due to the recently performed failover, the current secondary host is a known-good host because it had been functioning as the primary host. You can quickly restore service by failing over to it, and then repairing the host (which experienced the failed upgrade) by rebuilding it as a secondary host.

If you receive the following error confirming a failed primary upgrade:

Controller: primary_upgrade_error

You can try the following recovery options:

  •  Option 1: Retry the upgrade by entering the following command on the Enterprise Console host:

    bin/platform-admin.sh submit-job --service controller --job upgrade-primary --platform-name <name_of_the_platform> --args controllerRootUserPassword=<controller_root_password> mysqlRootPassword=<mysql_root_password> useCheckpoint=true
    CODE

  • Option 2: Perform a failover and rebuild on the secondary host. Due to the recently performed failover, the current secondary host is a known-good host because it had been functioning as the primary host. You can quickly restore service by failing over to it, and then repairing the host (which experienced the failed upgrade) by rebuilding it as a secondary host.
    From the Enterprise Console host:

    1. Enter the ha-failover command to revert the primary host to the older version:

      bin/platform-admin.sh submit-job --service controller --job ha-failover --platform-name <name_of_the_platform> --args forceFailover=true
      CODE

    2. Enter the incremental-replication command to reduce downtime. When dealing with large data, this is recommended because replication time and downtime depend on data size.

      bin/platform-admin.sh submit-job --service controller --job incremental-replication --platform-name <name_of_the_platform>
      CODE

    3. Enter the finalize-replication command to rebuild the secondary host:

      bin/platform-admin.sh submit-job --service controller --job finalize-replication --platform-name <name_of_the_platform>
      CODE

Verify the Primary Upgrade is Unsatisfactory

If the primary upgrade succeeded, but you discovered problems during manual verification, you can roll back the upgrade by performing a failover and rebuilding the secondary. 

If the downtime maintenance window is closing, you need to restore the older deployment version and service. Due to the recently performed failover, the current secondary host is a known-good host because it had been functioning as the primary host. You can quickly restore service by failing over to it, and then repairing the host (which experienced the failed upgrade) by rebuilding it as a secondary host. 

From the Enterprise Console host:

  1. Enter the ha-failover command to revert the primary host to the older version:

    bin/platform-admin.sh submit-job --service controller --job ha-failover --platform-name <name_of_the_platform> --args forceFailover=true
    CODE

  2. Enter the incremental-replication command to reduce downtime. When dealing with large data, this is recommended because replication time and downtime depend on data size.

    bin/platform-admin.sh submit-job --service controller --job incremental-replication --platform-name <name_of_the_platform>
    CODE

  3. Enter the finalize-replication command to rebuild the secondary host:

    bin/platform-admin.sh submit-job --service controller --job finalize-replication --platform-name <name_of_the_platform>
    CODE

Upgrade the Secondary Controller Fails

If you receive the following error confirming a failed secondary upgrade:

Controller: secondary_upgrade_error

You can try the following recovery options:

  • Option 1: If the failure is recoverable, you can retry the upgrade by entering the following command on the Enterprise Console host: 

    bin/platform-admin.sh submit-job --service controller --job upgrade-secondary --platform-name <name_of_the_platform> --args controllerRootUserPassword=<controller_root_password> mysqlRootPassword=<mysql_root_password> useCheckpoint=true
    CODE

  • Option 2: If retrying the upgrade does not work, you can run the incremental-replication and finalize-replication jobs. This involves downtime on the primary. Enter the following commands on the Enterprise Console host:

    bin/platform-admin.sh submit-job --service controller --job incremental-replication --platform-name <name_of_the_platform>
bin/platform-admin.sh submit-job --service controller --job finalize-replication --platform-name <name_of_the_platform>
    CODE

    For more details, see Initiate Controller Database Incremental Replication.

  • Option 3: If the machine dies or cannot be fixed by running the incremental-replication and finalize-replication jobs, you can remove the secondary Controller. This is a unique fix for an uncommon situation. Enter the following command on the Enterprise Console host:

    bin/platform-admin.sh submit-job --service controller --job remove --platform-name <name_of_the_platform> --args entireCluster=false removeBinaries=true
    CODE

    The flag, removeBinaries=true, is optional. The use of this flag depends on the situation of your deployment.
    If removeBinaries=false then the Enterprise Console forgets the Controller without impacting or uninstalling the Controller.

How to Upgrade the HA Modules Without Upgrading the Controller

Using the Enterprise Console UI or the CLI, you can upgrade the HA modules without having to upgrade the Controller. When both Controllers are managed by the Enterprise Console, the HA module is automatically upgraded when you upgrade your HA Controllers.

The following procedure describes an example scenario:

  1. You are currently running version 4.5.13 of both Enterprise Console and Controller.
  2. You activate the HA modules. 
  3. You then perform an upgrade only for the Enterprise Console to the latest version (4.5.15 or later).

  4. You can upgrade the HA modules from either the Enterprise Console UI or the CLI:

    • From the Enterprise Console UI:

      1. Log in to the Enterprise Console and access the Controller page.

      2. From the More menu, select Upgrade HA Modules.

    • From the CLI, run the following command on the Enterprise Console host to upgrade the HA modules: 

      bin/platform-admin.sh submit-job --job upgrade-ha-modules --service controller

The HA modules are upgraded to the latest version without upgrading the Controller. When you upgrade the HA modules, no downtime is required on the Controller, and all HA settings are preserved.