Upgrade the HA Controller Pair Using the Upgrade Wizard

You can use the HA Controller Upgrade Wizard once both of your Controllers (primary and secondary) have been onboarded into the Enterprise Console.

If you are using the HA ToolKit (HATK) and made any customizations to it, we recommend that you review your particular situation and determine if you should proceed with the migration. For more details, see Migrate to the New HA Module Using Enterprise Console.

Upgrading the HA Controller pair consists of different procedures based on your deployment:

Follow Option 1 - Discover and Upgrade for deployments that are not managed by the Enterprise Console.
Use the discover and upgrade job to onboard your HA Controller pair to the Enterprise Console before upgrading the primary Controller.
Follow Option 2 - Upgrade for HA pair deployments that are managed by the Enterprise Console.
Use the upgrade option to upgrade your primary and secondary Controllers managed by the current Enterprise Console instance.

To upgrade using the Upgrade Wizard:

Step 1: Prepare the Upgrade
Step 2: Enter Controller Credentials
Step 3: Perform a Failover
Step 4: Upgrade the Primary Controller
Step 5: Verify the Primary Controller Upgrade
Step 6: Upgrade the Secondary Controller

Step 1: Prepare the Upgrade

Download the latest release from Download Portal.
If you prefer to use the Linux shell, see Download Software.
Install the Enterprise Console.
Before you perform an upgrade, you must ensure that your secondary Controller is in a healthy state by performing a failover. We recommend that you log in to your secondary Controller host and run the <controller-home>/HA/failover.sh command to perform a Controller failover.

Download the latest release from Download Portal.
If you prefer to use the Linux shell, see Download Software.
Upgrade the Enterprise Console.

Step 2: Enter Controller Credentials

Select Discover and Upgrade.
Enter the credentials for the Controller, installation directory path, and determine the primary and secondary hosts.
Select Continue. The system verifies the credentials you entered for the Controller.
After the credentials have been successfully verified, the verification summary displays:
- Select Continue to upgrade the primary Controller. Continue with Step 5: Upgrade the Primary Controller.
- If you wish to stop the upgrade process, select Abort.

Select Upgrade.
Enter the credentials for the Controller.
Select Continue. The system verifies the credentials you entered for the Controller.
After the credentials have been successfully verified, the verification summary displays:

- Select Continue to perform a failover to the secondary Controller. Continue with Step 4: Perform a Failover.
- If you wish to stop the upgrade process, select Abort.

Step 3: Perform a Failover

Step 3: Perform a Failover does not apply to Option 1- Discover and Upgrade. Continue to Step 4: Upgrade the Primary Controller.

Before you upgrade the Controller, we recommend that you perform a failover to determine whether the other Controller is functional.

Because the primary Controller is known to be working, this step provides a stable configuration to fail back to in case of upgrade issues. Once the Controller HA failover job completes successfully, the following verification message displays:

Verification summary
Select Continue to upgrade the primary Controller.

Step 4: Upgrade the Primary Controller

Select Discover Upgrade.
After the primary Controller has been successfully discovered and upgraded, the following message displays:

Select Continue to verify the primary Controller.

Select Upgrade.
After the primary Controller has been successfully upgraded, the following message displays:

Select Continue to verify the primary Controller.

See Troubleshooting the Upgrade if the primary upgrade fails.

Step 5: Verify the Primary Controller Upgrade

This step pauses the upgrade process and allows you to manually verify the new Controller version.

Before you proceed with upgrading the secondary Controller, we recommend that you verify if Controller is working by logging in to the Controller and checking that metrics have been received within the last five minutes. If you are not satisfied with the upgrade, you can roll back to the older version from which you started.

If you are satisfied with the upgrade and want to continue:

Select Yes, I want to continue the upgrade.
Select Continue to upgrade the secondary Controller.

If you are not satisfied with the upgrade and want to stop:

Select No, I am not ready to continue.
Select OK to confirm that you want to abort the process and exit the Upgrade Wizard; or select Cancel to return to the upgrade process.
If you abort the process and exit from the Upgrade Wizard, select Fail Over and Rebuild Secondary and follow Option 1- Discover and Upgrade to continue the process.

If you are satisfied with the upgrade and want to continue:

Select Yes, I want to continue the upgrade.
Select Continue to upgrade the secondary Controller.

If you are not satisfied with the upgrade and want to roll back:

Select No, I want to failover & rebuild secondary.
Select Failover & Rebuild Secondary to start the process. Continue to Fail Over and Rebuild Secondary.

Step 6: Upgrade the Secondary Controller

Enter the credentials for the Controller.
Select Continue.
After the secondary Controller has been successfully discovered and upgraded, the following verification message displays:

Select Done to complete the Upgrade Wizard.

While the secondary Controller is being upgraded, the primary Controller is available to serve agent traffic and the downtime period is completed. Select Continue to complete the Upgrade Wizard.

Troubleshooting the Upgrade

Fail Over and Rebuild Secondary

When you want to quickly restore the service, you can manually fail over to the previous Controller version by running the following script on the current secondary Controller host:

<controller-home>/HA/failover.sh -f

CODE

This script kills the watchdog process, starts the app server on the current secondary Controller, and restores one of the Controllers to a previous version.
To rebuild the other Controller in the HA pair, you must finalize the replication. To do this, we recommend that you run the incremental-replication and finalize-replication jobs in HATK.

From the HA ToolKit (HATK), enter the replicate commands to rebuild the current secondary Controller.
```
./replicate.sh -s <secondary_hostname> -e <external_vip> -i <internal_vip>
```
where the command options are as follows:
-s <secondary_hostname> – The hostname of the current secondary you configured in /etc/hosts-e <external_vip> – The hostname and port for the Controller pair as exposed to clients at the load balancer or other type of reverse proxy. This is the address at which the App Agents and the Controller UI clients would typically address the Controller. For example: http://controllervip.example.com:80.
-i <internal_vip> – If your Controllers reside within a protected network (relative to the load balancer) and cannot address the virtual IP directly, use this value to indicate an internal virtual IP address for the Controllers to use. For example: http://controllervip.internal.example.com:80
From the HATK, enter the finalize replication commands:

This step involves downtime on the current primary Controller.
```
./replicate.sh -s <secondary_hostname> -f -w -e <external_vip> -i <internal_vip>
```
where the command options are as follows:-f – Causes the script to finalize setup and restarts the Controllers
-w – Starts the watchdog process on the current secondary

After starting the fail over and rebuild process to rebuild the current secondary Controller, the following message displays if the Controller HA failover job was successful:

If using HATK, you can check the current state of the primary and secondary Controllers by running this script on each server: <controller_home>/HA/appdstatus.sh
1. Select Continue to begin incremental replication for the rebuild and continue to Step 2.
2. Otherwise, if you need to cancel the Failover & Rebuild Secondary process, select Abort to exit. The following confirmation message displays:
3. Select OK to confirm cancellation.
If the Controller incremental replication job was successful, the following message displays:

Select Continue to finalize the replication to the secondary Controller.
If the Controller finalize replication job completed successfully, the following message displays:

Select Done.

This completes rebuilding the current secondary server so both Controllers in the HA pair are now replicating data.

The Secondary Controller Upgrade Fails

When the secondary Controller upgrade fails:

If you have identified the cause of failure and manually resolved the issue, select Resume to retry the upgrade.
If you already tried to manually resolve the issue, and selecting the Resume option did not resolve the issue, select Rebuild Secondary to rebuild your secondary Controller. Based on the size of your Controller data, this may take some time.

Upgrade failed message

From the More menu, select Incremental Replication. The incremental replication job does not involve any downtime on the current primary Controller.
Before selecting Finalize Replication, allow the incremental replication job to run and complete at least three times. The finalize replication job involves downtime on the current primary Controller.