On this page:

Related pages:

Your Rating:
Results:
PatheticBadOKGoodOutstanding!
17 rates

You can use the Enterprise Console to onboard and upgrade a Controller HA pair. For HA pairs that are not managed by the Enterprise Console, use the discover and upgrade command, while HA pairs that are managed by the Enterprise Console require the upgrade command.

This topic covers how to use the phased upgrade method to upgrade a Controller HA pair. If you would like to use previous upgrade methods, see Upgrade an HA Pair.

About the Upgrade

The upgrade is currently only available via the CLI, with UI support planned for a later version.

This upgrade method improves upon the previous upgrade methods by:

  • retaining the HA roles of the Controllers.
  • performing a set of pre-upgrade validations. You can see a summary of validation errors upfront before modifying the state of the system.
  • having the ability to quickly restore the older deployment version and service from the preserved secondary in the case of any upgrade issues. While the primary server is being upgraded, the secondary is isolated, serving as a backup which allows you to quickly restore the service.

Upgrade the HA Pair 

Steps 1 to 3 below consists of different procedures depending on your HA pair deployment:

  • Follow Option 1 - Discover and Upgrade for deployments that are not managed by the Enterprise Console.
    This option uses the discover and upgrade job to onboard your HA pair to the Enterprise Console before upgrading the primary Controller.
  • Follow Option 2 - Upgrade for HA pair deployments that are managed by the Enterprise Console.
    This option uses the upgrade job to upgrade your primary Controller that is managed by the current Enterprise Console instance.

Completing an upgrade involves the following steps:

 

Step 1: Prepare the Upgrade

Option 1 - Discover and Upgrade

  1. Download the latest release from AppDynamics Download Center.
    If you prefer to use the Linux shell, see Downloading from the Linux Shell.

  2. Install the Enterprise Console.

Option 2 - Upgrade

  1. Download the latest release from AppDynamics Download Center.
    If you prefer to use the Linux shell, see Downloading from the Linux Shell.

  2. Upgrade the Enterprise Console.

 

Step 2: Perform a Failover and Check the HA Pair State

Upgrades begin by performing a failover to the secondary Controller. Since the primary Controller is known to be working, this provides a known-good configuration to fail back to in case of a problem with the upgrade.

To perform a failover and check the HA pair state:

Option 1 - Discover and Upgrade

  1. Run the HA Toolkit failover.sh script on the secondary Controller to initiate failover.
    This kills the watchdog process, starts the app server on the secondary, and makes the database on the secondary the replication master.

  2. Run the Linux service utility on either node to report the current state:

    service appdcontroller status
  3. Shut Down the HA Toolkit Implementation.

Option 2 - Upgrade

  1. Run the following command on the Enterprise Console host to initiate a failover:

    bin/platform-admin.sh submit-job --service controller --job ha-failover --platform-name <name_of_the_platform>
  2. Run the following command on the Enterprise Console host to check the state:

    bin/platform-admin.sh list-platform-service --platform-name <name_of_the_platform>

    The desired output is "Controller: provisioned".

  3. Run the following command on the Enterprise Console host to check the Controller versions:

    bin/platform-admin.sh list-node --service controller --platform-name <name_of_the_platform>

    The output will display the host versions.

    Sample Output
    Available nodes in the controller service:
    	Controller: role Primary, host 172.12.0.1, version 4.5.0.2, state running
             MySQL: role Primary, host 172.12.0.1, version 5.7.21 state running      
    	Controller: role Secondary, host 172.12.0.2, version 4.5.0.2, state running
             MySQL: role Secondary, host 172.12.0.2, version 5.7.21, state running

    The primary Controller's Appserver and MySQL should be running before upgrading. The secondary Controller's MySQL must also be running.


Step 3: Upgrade the Primary Controller

You can upgrade the primary Controller using one of the following commands:

Option 1 - Discover and Upgrade

Discover and upgrade primary
Run the following command on the Enterprise Console host:

bin/platform-admin.sh submit-job --service controller --job  discover-and-upgrade-primary --platform-name <name_of_the_platform> --target-version=<version> --args  controllerPrimaryHost=<primary_host> controllerSecondaryHost=<secondary_host> destinationDirectory=<controller_home_dir> controllerRootUserPassword=<controller_root_password> mysqlRootPassword=<mysql_root_password> controllerDBPassword=<controller_db_user_password>

Option 2 - Upgrade

Upgrade primary
Run the following command on the Enterprise Console host:

bin/platform-admin.sh submit-job --service controller --job upgrade-primary --platform-name <name_of_the_platform> --target-version=<version> --args controllerRootUserPassword=<controller_root_password> mysqlRootPassword=<mysql_root_password> controllerDBPassword=<controller_db_user_password>
See Troubleshooting the Upgrade if the primary upgrade fails.

Step 4: Verify the Primary Upgrade

A primary upgrade success message will display if the upgrade succeeds. However, it is recommended that you perform your own in-house verification on the Controller before proceeding to the secondary upgrade. For example, you can check that your agents are continuing to report to your Controller.

The following is the expected state in the Enterprise Console:

  1. Run the following commands on the Enterprise Console host:

    bin/platform-admin.sh list-platform-service --platform-name <name_of_the_platform>

    The desired output is "Controller: primary_upgraded".

    bin/platform-admin.sh list-node --service controller --platform-name <name_of_the_platform>

    The output will display the host versions.

    Sample Output
    Available nodes in the controller service:
    	Controller: role Primary, host 172.12.0.1, version 4.5.5.0, state running
             MySQL: role Primary, host 172.12.0.1, version 5.7.24 state running      
    	Controller: role Secondary, host 172.12.0.2, version 4.5.0.2, state stopped
             MySQL: role Secondary, host 172.12.0.2, version 5.7.21, state running

    Check that all hosts show the new versions, that the Controller is running on the primary and stopped on the secondary, and that MySQL is running on both hosts.

Step 5: Upgrade the Secondary Controller

You can upgrade the secondary Controller by using the upgrade-secondary command:

  1. Run the upgrade-secondary command on the Enterprise Console host:

    bin/platform-admin.sh submit-job --service controller --job upgrade-secondary --platform-name <name_of_the_platform> --args controllerRootUserPassword=<controller_root_password> mysqlRootPassword=<mysql_root_password>

See Troubleshooting the Upgrade if the secondary upgrade fails.

Step 6: Verify the Secondary Upgrade

A secondary upgrade success message will display if the upgrade succeeds. However, it is recommended that you perform your own in-house verification on the Controller before completing the upgrade.

The following is the expected state in the Enterprise Console:

  1. Run the following commands on the Enterprise Console host:

    bin/platform-admin.sh list-platform-service --platform-name <name_of_the_platform>

    The desired output is "Controller: provisioned".

    bin/platform-admin.sh list-node --service controller --platform-name <name_of_the_platform>

    The output will display the host versions.

    Sample Output
    Available nodes in the controller service:
    	Controller: role Primary, host 172.12.0.1, version 4.5.5.0, state running
             MySQL: role Primary, host 172.12.0.1, version 5.7.24 state running      
    	Controller: role Secondary, host 172.12.0.2, version 4.5.5.0, state stopped
             MySQL: role Secondary, host 172.12.0.2, version 5.7.24, state running

    Ensure that the secondary host version has been upgraded. Also, the primary should be in a running state, while the secondary should remain stopped. However, its MySQL host should be running.

Troubleshooting the Upgrade

Failover Issues

If you discover failover issues before upgrading, determine whether the secondary Controller is not in a good shape. You need to fix the broken secondary prior to attempting an upgrade. 

Primary Discover and Upgrade Fails

If the primary discover and upgrade failed, try these recovery steps.

There are three options you can take:

  • Retry the discover and upgrade:

    Run the following command on the Enterprise Console host:

    bin/platform-admin.sh submit-job --service controller --job  discover-and-upgrade-primary --platform-name <name_of_the_platform> --args  controllerPrimaryHost=<primary_host> controllerSecondaryHost=<secondary_host> destinationDirectory=<controller_home_dir> controllerRootUserPassword=<controller_root_password> mysqlRootPassword=<mysql_root_password> useCheckpoint=true
  • Manual failover using the HA Toolkit: Recall that in step 2, you performed a failover. That means that the current secondary host is known-good since it was recently the primary. You can quickly restore service by failing over to it, and then fix the host which experienced the failed upgrade by rebuilding it as a secondary.

  • Restore from backup: If you performed a backup before initiating the upgrade, restoring from backup may be a faster option than "Manual failover using the HA Toolkit", as the latter requires extensive data copy between Controller hosts. See Back up the Existing Controller for more information.

Primary Upgrade Fails

If the primary upgrade failed, try these recovery steps:

bin/platform-admin.sh list-platform-service --platform-name <name_of_the_platform>

The output "Controller: primary_upgrade_error" confirms a failed upgrade.

There are three options you can take after the primary upgrade fails:

  • Retry the upgrade:

    Run the following command on the Enterprise Console host:

    bin/platform-admin.sh submit-job --service controller --job upgrade-primary --platform-name <name_of_the_platform> --args controllerRootUserPassword=<controller_root_password> mysqlRootPassword=<mysql_root_password> useCheckpoint=true
  • Failover and rebuild secondary: Recall that in step 2, you performed a failover. That means that the current secondary host is known-good since it was recently the primary. You can quickly restore service by failing over to it, and then fix the host which experienced the failed upgrade by rebuilding it as a secondary:

    • Use failover so the primary reverts to the older version, then run finalize-replication, which is required to rebuild the secondary

      Run the following commands on the Enterprise Console host:

      bin/platform-admin.sh submit-job --service controller --job ha-failover --platform-name <name_of_the_platform> --args forceFailover=true
      bin/platform-admin.sh submit-job --service controller --job finalize-replication --platform-name <name_of_the_platform>
    • Use incremental-replication to reduce downtime. This is recommended in cases dealing with large data since replication time and downtime depend on data size.

      Run the following command on the Enterprise Console host before submitting finalize-replication:

      bin/platform-admin.sh submit-job --service controller --job incremental-replication --platform-name <name_of_the_platform>
  • Restore from backup: If you performed a backup before initiating the upgrade, restoring from backup may be a faster option than "Failover and rebuild secondary", as the latter requires extensive data copy between Controller hosts. See Back up the Existing Controller for more information.

Primary Upgrade Succeeds But With New Version Issue

If the primary upgrade succeeded, but you discovered problems during manual verification, you can roll back the upgrade by following "Failover and rebuild secondary" or "Restore from backup":

  • Failover and rebuild: If the downtime maintenance window is closing, you need to restore the older deployment version and service. You can do so by rebuilding the secondary. There are two ways to rebuild, which involves downtime, but no manual step:

    • Use failover so the primary reverts to the older version, then run finalize-replication, which is required to rebuild the secondary.

      Run the following commands on the Enterprise Console host:

      bin/platform-admin.sh submit-job --service controller --job ha-failover --platform-name <name_of_the_platform> --args forceFailover=true
      bin/platform-admin.sh submit-job --service controller --job finalize-replication --platform-name <name_of_the_platform>
    • Use incremental-replication to reduce downtime. This is recommended in cases dealing with large data since replication time and downtime depend on data size.

      Run the following command on the Enterprise Console host before submitting finalize-replication:

      bin/platform-admin.sh submit-job --service controller --job incremental-replication --platform-name <name_of_the_platform>
  • Restore from backup: If a manual rollback will be quicker than incremental and finalize jobs, perform a manual rollback. Cases where restoring from backup would be quicker are when you have large data and a full data backup available prior to the upgrade.
    To perform a manual rollback, you must manually copy back binaries and data on the primary, and manually restart db replication. See Back up the Existing Controller for more information.

Secondary Upgrade Fails

If the secondary upgrade fails, try:

bin/platform-admin.sh list-platform-service --platform-name <name_of_the_platform>

The output "Controller: secondary_upgrade_error" confirms a failed upgrade.

There are three options you can take after the secondary upgrade fails:

  • Retry the upgrade: If the failure is recoverable, you can retry the failed upgrade.

    Run the following command on the Enterprise Console host:

    bin/platform-admin.sh submit-job --service controller --job upgrade-secondary --platform-name <name_of_the_platform> --args controllerRootUserPassword=<controller_root_password> mysqlRootPassword=<mysql_root_password> useCheckpoint=true
  • Run incremental/finalize replication: If retrying the upgrade does not work, you can use the incremental-replication and finalize-replication jobs. This involves downtime on the primary.
    Run the following commands on the Enterprise Console host:

    bin/platform-admin.sh submit-job --service controller --job incremental-replication --platform-name <name_of_the_platform>
    bin/platform-admin.sh submit-job --service controller --job finalize-replication --platform-name <name_of_the_platform>

    See Initiate Controller Database Incremental Replication for more information.

  • Remove the secondary: If the machine dies or cannot be fixed by incremental/finalize, you can remove the secondary Controller. This is a unique fix for an uncommon situation.

    bin/platform-admin.sh submit-job --service controller --job remove --platform-name <name_of_the_platform> --args entireCluster=false removeBinaries=true

    The flag, removeBinaries=true, is optional. The use of this flag depends on the situation of your deployment.
    If removeBinaries=false then the Enterprise Console forgets the Controller without impacting or uninstalling the Controller.

Shut Down the HA Toolkit Implementation

Before you start the Controller HA upgrade from a pre-Enterprise Console deployment, you must first shut down the watchdog and assassin processes. This is only required if you have installed the HA Toolkit before.

To shut down the HA Toolkit implementation, follow one of the applicable steps below:

  • If the Controller services are installed with privilege escalation using setups option (-c option in install-init.sh) then running the following command on the secondary will stop the secondary appserver, watchdog, and assassin.

    /sbin/appdservice appdcontroller stop

    or

  • If the Controller services are installed with privilege escalation using sudoers option (-s option in install-init.sh) then running the following command on the secondary will stop the secondary appserver, watchdog, and assassin.

    sudo /sbin/service appdcontroller stop

To check if the appserver, watchdog, and assassin stopped on the secondary, you can use the following commands based on the privilege escalation method used to install the Controller services:

  • /sbin/appdservice appdcontroller status
  • sudo /sbin/service appdcontroller status

For both stopping and checking the statuses, if you do not remember the privilege escalation method used to install services, then both the variants can be used one after the other in any order. At least one variant of the command should work.

  • No labels