AppDynamics switched from Semantic Versioning to Calendar Versioning starting in February 2020 for some agents and March 2020 for the entire product suite.


    Skip to end of metadata
    Go to start of metadata

    If are using the HA ToolKit (HATK) and made any customizations to it, AppDynamics recommends that you review your particular situation and determine if you should proceed with the migration. For more details, see HA module in the Enterprise Console.

    Steps 1 through 3 consist of different procedures depending on your HA Controller pair deployment:

    • Follow Option 1 - Discover and Upgrade for deployments that are not managed by the Enterprise Console.
      Use the discover and upgrade job to onboard your HA Controller pair to the Enterprise Console before upgrading the primary Controller.
    • Follow Option 2 - Upgrade for HA pair deployments that are managed by the Enterprise Console.
      Use the upgrade option to upgrade your primary Controller managed by the current Enterprise Console instance.

    To upgrade using the CLI:


    Step 1: Prepare the Upgrade

    1. Download the latest release from AppDynamics Download Center.
      If you prefer to use the Linux shell, see Downloading from the Linux Shell.

    2. Install the Enterprise Console.

    1. Download the latest release from AppDynamics Download Center.
      If you prefer to use the Linux shell, see Downloading from the Linux Shell.

    2. Upgrade the Enterprise Console.


    Step 2: Perform a Failover and Check the HA Pair State

    To begin the upgrade, perform a failover to the secondary Controller. Since the primary Controller is known to be working, this provides a stable configuration to fail back to in case of upgrade issues.

    To perform a failover and check the HA pair state:

    1. Run the HA Toolkit failover.sh script on the secondary Controller to initiate failover.
      This kills the watchdog process, starts the app server on the secondary, and makes the database on the secondary the replication master.

    2. Run the Linux service utility on either node to report the current state:

      service appdcontroller status
    3. Shut down the HA Toolkit (HATK) implementation.
    1. Run the following command on the Enterprise Console host to initiate a failover:

      bin/platform-admin.sh submit-job --service controller --job ha-failover --platform-name <name_of_the_platform>
    2. Run the following command on the Enterprise Console host to check the state:

      bin/platform-admin.sh list-platform-service --platform-name <name_of_the_platform>

      The desired output is "Controller: provisioned".

    3. Run the following command on the Enterprise Console host to check the Controller versions:

      bin/platform-admin.sh list-node --service controller --platform-name <name_of_the_platform>

      The output will display the host versions.

      Sample Output
      Available nodes in the controller service:
      	Controller: role Primary, host 172.12.0.1, version 4.5.0.2, state running
               MySQL: role Primary, host 172.12.0.1, version 5.7.21 state running      
      	Controller: role Secondary, host 172.12.0.2, version 4.5.0.2, state stopped
               MySQL: role Secondary, host 172.12.0.2, version 5.7.21, state running

      The primary Controller's Appserver and MySQL should be running before upgrading. The secondary Controller's MySQL must also be running.

    Step 3: Upgrade the Primary Controller

    You can upgrade the primary Controller using one of the following commands:

    To discover and upgrade the primary Controller, run the following command on the Enterprise Console host:

    bin/platform-admin.sh submit-job --service controller --job  discover-and-upgrade-primary --platform-name <name_of_the_platform> --target-version=<version> --args  controllerPrimaryHost=<primary_host> controllerSecondaryHost=<secondary_host> destinationDirectory=<controller_home_dir> controllerRootUserPassword=<controller_root_password> mysqlRootPassword=<mysql_root_password> controllerDBPassword=<controller_db_user_password>

    To upgrade the primary Controller, run the following command on the Enterprise Console host:

    bin/platform-admin.sh submit-job --service controller --job upgrade-primary --platform-name <name_of_the_platform> --target-version=<version> --args controllerRootUserPassword=<controller_root_password> mysqlRootPassword=<mysql_root_password>
    See Troubleshooting the Upgrade if the primary upgrade fails.

    Step 4: Verify the Primary Upgrade

    A primary upgrade success message displays if the upgrade is successful. However, AppDynamics recommends that you perform your own in-house verification on the Controller before proceeding to the secondary upgrade. For example, you can check that your agents are continuing to report to your Controller.

    The following describes the expected state in the Enterprise Console.

    Run the following commands on the Enterprise Console host:

    bin/platform-admin.sh list-platform-service --platform-name <name_of_the_platform>

    The desired output is Controller: primary_upgraded.

    bin/platform-admin.sh list-node --service controller --platform-name <name_of_the_platform>

    The output will display the host versions.

    Sample Output
    Available nodes in the controller service:
    	Controller: role Primary, host 172.12.0.1, version 4.5.5.0, state running
             MySQL: role Primary, host 172.12.0.1, version 5.7.24 state running      
    	Controller: role Secondary, host 172.12.0.2, version 4.5.0.2, state stopped
             MySQL: role Secondary, host 172.12.0.2, version 5.7.21, state running

    Check that all hosts show the new versions, that the Controller is running on the primary and stopped on the secondary, and that MySQL is running on both hosts. If you are not satisfied with the upgrade, see Verify the Primary Upgrade is Unsatisfactory.

    Step 5: Upgrade the Secondary Controller

    To upgrade the secondary Controller, run the the upgrade-secondary command on the Enterprise Console host:

    bin/platform-admin.sh submit-job --service controller --job upgrade-secondary --platform-name <name_of_the_platform> --args controllerRootUserPassword=<controller_root_password> mysqlRootPassword=<mysql_root_password>

    If the secondary Controller upgrade fails, see Upgrade the Secondary Controller Fails for possible recovery options. 

    Step 6: Verify the Secondary Upgrade

    A secondary upgrade success message displays if the upgrade is successful. However, AppDynamics recommends that you perform your own in-house verification on the Controller before completing the upgrade.

    The following describes the expected state in the Enterprise Console.

    Run the following commands on the Enterprise Console host:

    bin/platform-admin.sh list-platform-service --platform-name <name_of_the_platform>

    The desired output is "Controller: provisioned".

    bin/platform-admin.sh list-node --service controller --platform-name <name_of_the_platform>

    The output will display the host versions.

    Sample Output
    Available nodes in the controller service:
    	Controller: role Primary, host 172.12.0.1, version 4.5.5.0, state running
             MySQL: role Primary, host 172.12.0.1, version 5.7.24 state running      
    	Controller: role Secondary, host 172.12.0.2, version 4.5.5.0, state stopped
             MySQL: role Secondary, host 172.12.0.2, version 5.7.24, state running

    Ensure that the secondary host version has been upgraded. Also, the primary should be in a running state, while the secondary Controller appserver should remain stopped. However, its MySQL process should be running.

    Troubleshooting the Upgrade

    Failover Issues

    If you experience failover issues before upgrading, determine the condition of the secondary Controller. You may need to fix a broken secondary Controller before you attempt an upgrade.  

    The Primary Controller Upgrade Fails

    If the primary Controller failed for discover and upgrade, try the following recovery options:

    • Option 1: Retry the discover and upgrade by entering the following command on the Enterprise Console host:

      bin/platform-admin.sh submit-job --service controller --job  discover-and-upgrade-primary --platform-name <name_of_the_platform> --args  controllerPrimaryHost=<primary_host> controllerSecondaryHost=<secondary_host> destinationDirectory=<controller_home_dir> controllerRootUserPassword=<controller_root_password> mysqlRootPassword=<mysql_root_password> useCheckpoint=true
    • Option 2: Perform a manual failover using the HA Toolkit (HATK). Due to the recently performed failover, the current secondary host is a known-good host because it had been functioning as the primary host. You can quickly restore service by failing over to it, and then repairing the host (which experienced the failed upgrade) by rebuilding it as a secondary host.

    If you receive the following error confirming a failed primary upgrade:

    Controller: primary_upgrade_error

    You can try the following recovery options:

    •  Option 1: Retry the upgrade by entering the following command on the Enterprise Console host:

      bin/platform-admin.sh submit-job --service controller --job upgrade-primary --platform-name <name_of_the_platform> --args controllerRootUserPassword=<controller_root_password> mysqlRootPassword=<mysql_root_password> useCheckpoint=true
    • Option 2: Perform a failover and rebuild on the secondary host. Due to the recently performed failover, the current secondary host is a known-good host because it had been functioning as the primary host. You can quickly restore service by failing over to it, and then repairing the host (which experienced the failed upgrade) by rebuilding it as a secondary host.
      From the Enterprise Console host:

      1. Enter the ha-failover command to revert the primary host to the older version:

        bin/platform-admin.sh submit-job --service controller --job ha-failover --platform-name <name_of_the_platform> --args forceFailover=true
      2. Enter the incremental-replication command to reduce downtime. When dealing with large data, this is recommended because replication time and downtime depend on data size.

        bin/platform-admin.sh submit-job --service controller --job incremental-replication --platform-name <name_of_the_platform>
      3. Enter the finalize-replication command to rebuild the secondary host:

        bin/platform-admin.sh submit-job --service controller --job finalize-replication --platform-name <name_of_the_platform>

    Verify the Primary Upgrade is Unsatisfactory

    If the primary upgrade succeeded, but you discovered problems during manual verification, you can roll back the upgrade by performing a failover and rebuilding the secondary. 

    If the downtime maintenance window is closing, you need to restore the older deployment version and service. Due to the recently performed failover, the current secondary host is a known-good host because it had been functioning as the primary host. You can quickly restore service by failing over to it, and then repairing the host (which experienced the failed upgrade) by rebuilding it as a secondary host. 

    From the Enterprise Console host:

    1. Enter the ha-failover command to revert the primary host to the older version:

      bin/platform-admin.sh submit-job --service controller --job ha-failover --platform-name <name_of_the_platform> --args forceFailover=true
    2. Enter the incremental-replication command to reduce downtime. When dealing with large data, this is recommended because replication time and downtime depend on data size.

      bin/platform-admin.sh submit-job --service controller --job incremental-replication --platform-name <name_of_the_platform>
    3. Enter the finalize-replication command to rebuild the secondary host:

      bin/platform-admin.sh submit-job --service controller --job finalize-replication --platform-name <name_of_the_platform>

    Upgrade the Secondary Controller Fails

    If you receive the following error confirming a failed secondary upgrade:

    Controller: secondary_upgrade_error

    You can try the following recovery options:

    • Option 1: If the failure is recoverable, you can retry the upgrade by entering the following command on the Enterprise Console host: 

      bin/platform-admin.sh submit-job --service controller --job upgrade-secondary --platform-name <name_of_the_platform> --args controllerRootUserPassword=<controller_root_password> mysqlRootPassword=<mysql_root_password> useCheckpoint=true
    • Option 2: If retrying the upgrade does not work, you can run the incremental-replication and finalize-replication jobs. This involves downtime on the primary. Enter the following commands on the Enterprise Console host:

      bin/platform-admin.sh submit-job --service controller --job incremental-replication --platform-name <name_of_the_platform>
      bin/platform-admin.sh submit-job --service controller --job finalize-replication --platform-name <name_of_the_platform>

      For more details, see Initiate Controller Database Incremental Replication.

    • Option 3: If the machine dies or cannot be fixed by running the incremental-replication and finalize-replication jobs, you can remove the secondary Controller. This is a unique fix for an uncommon situation. Enter the following command on the Enterprise Console host:

      bin/platform-admin.sh submit-job --service controller --job remove --platform-name <name_of_the_platform> --args entireCluster=false removeBinaries=true

      The flag, removeBinaries=true, is optional. The use of this flag depends on the situation of your deployment.
      If removeBinaries=false then the Enterprise Console forgets the Controller without impacting or uninstalling the Controller.

    How to Upgrade the HA Modules Without Upgrading the Controller

    Using the Enterprise Console UI or the CLI, you can upgrade the HA modules without having to upgrade the Controller. When both Controllers are managed by the Enterprise Console, the HA module is automatically upgraded when you upgrade your HA Controllers.

    The following procedure describes an example scenario:

    1. You are currently running version 4.5.13 of both Enterprise Console and Controller.
    2. You activate the HA modules
    3. You then perform an upgrade only for the Enterprise Console to the latest version (4.5.15 or later).
    4. You can upgrade the HA modules from either the Enterprise Console UI or the CLI:

      • From the Enterprise Console UI:

        1. Log in to the Enterprise Console and access the Controller page.

        2. From the More menu, select Upgrade HA Modules.

      • From the CLI, run the following command on the Enterprise Console host to upgrade the HA modules: 

        bin/platform-admin.sh submit-job --job upgrade-ha-modules --service controller

    The HA modules are upgraded to the latest version without upgrading the Controller. When you upgrade the HA modules, no downtime is required on the Controller, and all HA settings are preserved. 

    • No labels