Related pages:

You can upgrade a Controller instance using the Enterprise Console. The Enterprise Console simplifies the upgrade process by allowing you to discover and upgrade single Controllers and HA-pairs.

About the Upgrade

If your current Controller version is < 21.x, see Upgrade the Controller Using the Enterprise Console.

  • The Enterprise Console supports standalone and HA-pair Controller upgrades. Use the following table to determine your course of action based on your circumstances:

    If your current Controller version is...Controller version you want to upgrade to...Actions to take...
    >= 21.x Controller version 21.x or the latest version
    1. Upgrade MYSQL to 5.7 version.
    2. Upgrade to the supported OS version. See Controller Compatibility with Operating Systems and Components.
    3. Access the downloads portal to download the Controller version.
    4. Use the Enterprise Console to upgrade the Controller.

  • You can choose which version you would like to upgrade the Controller to as long as the Enterprise Console is aware of that version. This means that you can upgrade the Controller to any intermediate version or to the latest version as long as the Enterprise Console installer has been run for those versions. However, you cannot upgrade the Controller to an older version.

     From Enterprise Console >=23.11, the Controller uses Jetty Application server instead of GlassFish. Therefore, Enterprise Console >=23.11 allows only the Controller upgrade to >= 23.11.x.

  • If you have a license for the older version, the license should work when upgrading the Controller to a new version. However, if you have a temporary license for the old version and now have a new license, the new license will not work on the old Controller. In this case, you should upgrade the Controller to the latest version before applying the license. 
  • An upgrade results in Controller downtime, but it is not necessary to stop agents during the Controller upgrade.

  • The Enterprise Console expects a .passwordfile file to be present in the Controller home directory. The Enterprise Console reads this password and validates it against the Controller. Once the upgrade is complete, the Enterprise Console removes the file, and stores the password in its encrypted database.

    If you change the Glassfish Admin Password manually, you also need to update it in the Enterprise Console Controller Settings.

Before Upgrading

  • Before you start upgrading the Controller, make sure that you are using the correct update order.
  • Review the latest Release Notes and the release notes for any intermediate versions between the current version of your instance and the version you are targeting to learn about issues and enhancements in those releases. 

  • Check the most recent Controller System Requirements and Troubleshoot Controller Issues to review your Controller's current workload and determine whether you need to change your performance profile and increase your hardware resources, if necessary.

    You may change your Controller profile on the Platform Configurations pages of the Enterprise Console GUI, either before or after you upgrade your Controller. This process is not reversible, and you cannot move from a larger to a smaller profile size. 

  • Check the Controller's database.log for any errors. You can find the log at <controller_home>/db/logs/database.log. There should not be any InnoDB: Error lines in the log. If any errors are found, please reach out to customer support before attempting the upgrade. Upgrading the Controller with a corrupt database may put the Controller in a bad state, with high recovery time.
  • If you changed any Glassfish settings that are not JVM options, note your changes. You may need to configure them after an upgrade. The Enterprise Console recognizes and retains many common customizations to the domain.xml, db.cnf, and other configuration files, but is not guaranteed to retain them all. If you have made manual configuration changes to the files, verify the configuration after updating. See Retaining Configuration Changes to learn how to preserve changes.
  • If you uninstall the Enterprise Console that was previously managing the Controller and use a new Enterprise Console instance to discover and upgrade the Controller, you need to first manually create the .passwordfile file in order for the Enterprise Console to continue with the discover and upgrade process. You can create the file in the Controller home directory, and add the AS_ADMIN_PASSWORD=<controllerRootUserPassword> value in it.
  • When performing an upgrade while enabling HTTPS, check that the hostname does not start with a digit. The upgrade will fail if the Enterprise Console hostname starts with a digit due to a JDK limitation with the DNS name in CN/SAN.
  • Back up the existing Controller following the steps in the next section.

Modify the Controller Database

MYSQL versions <=5.7 have the Compact Row_Format. Therefore, you must modify the Controller database tables to change the Row_format to Dynamic before upgrading MYSQL to 8.0 version.

Before you upgrade the Controller MYSQL to version 8.0, upgrade the Enterprise Console and the Controller.

Complete the following steps to modify the database table:

  1. Stop the Controller application server.

    cd ~<controller_home>/bin/
./controller.sh stop-appserver
    CODE
  2. Run the following commands to create the SQL file that has ALTER commands.

    1. Navigate to the database bin directory.

      cd ~<controller_home>/db/bin/
      CODE

    2. Create the the Alter.sql file.

      ./mysql -N -uroot -p<database_root_user_password> --socket=~<MySql_Socket_Path> -e'SELECT CONCAT("ALTER TABLE ", tables.TABLE_SCHEMA, ".", tables.TABLE_NAME, " ROW_FORMAT=DYNAMIC;") FROM (SELECT TABLE_SCHEMA, TABLE_NAME, ROW_FORMAT FROM information_schema.TABLES WHERE ROW_FORMAT = "Compact") tables;' > Alter.sql
      CODE

      To fetch the MYSQL socket path, run this command:

      ~<MySql_Socket_Path> can be fetch from o/p of command -> ps -ef | grep mysqld
      CODE

  3. Log in to the Controller database to run the ALTER commands.

    cd ~<controller-home>/bin/
./controller.sh login-db
source Alter.sql
    CODE

  4. Verify that the tables have Dynamics Row_format.

    SELECT TABLE_SCHEMA, TABLE_NAME, ROW_FORMAT FROM information_schema.TABLES WHERE ROW_FORMAT = 'Compact';
    SQL

    If the output is NULL, the table has no Compact Row_Format.

Back up the Existing Controller

The Enterprise Console retains agent data, reports, configuration, and other types of data through an upgrade, including a copy of commonly modified configuration files under <controller_home>/backup. Nevertheless, to mitigate the risk of data loss in the event of an unexpected failure, be sure to back up the existing installation directory before applying an upgrade to the Controller.

The Enterprise Console does not back up MySQL data. You need to back up the data before upgrading standalone installations.

If the upgrade does not finish successfully for any reason, see Troubleshooting the Upgrade for more information. 

To back up the Controller instance
  1. Stop the Controller application server and database.

    • For Linux, run: 

      platform-admin.sh stop-controller-appserver

    • For Windows, run:

      platform-admin.exe cli stop-controller-appserver --with-db
  2. Back up the Controller home by copying the entire Controller home directory to a backup location. Note the following points: 
    • If the data home for the Controller is not under the Controller directory, be sure to back up the database directory as well. 
    • If it's not possible to back up the entire data set, you can selectively back up the most important tables. Use the Metadata Backup SQL script described and attached to the Controller Data Backup and Restore page. 
  3. Restart your Controller after completing the backup and before upgrading.

After Upgrading

  • Enterprise Console will take care of moving the configuration from glassfish to jetty.

    For the Controller upgrades from older versions (<23.11) to >= 23.11, it backs up the GlassFish directory (<controller_home>/appserver_gf_bkp). If you face any configuration issues, contact customer support.

  • The Enterprise Console preserves several recommended customizations. After the upgrade, you can find backup copies of common configuration files in the <controller_home>/backup directory.
  • As an optional step, your MySQL version can be upgraded after you upgrade the Controller, through the Enterprise Console GUI or by using the mysql-upgrade job. See Bundled MySQL Database Version for more information.

Troubleshooting the Upgrade

If the upgrade does not succeed for any reason, the Enterprise Console does not roll back changes on disk. This gives you an opportunity to diagnose and troubleshoot the issue before reattempting the installation or upgrade. 

To troubleshoot the issue, check the installation log at platform-admin/logs/platform-admin-server.log. The Controller server.log file may contain additional information. 

Resuming from Checkpoint

The Enterprise Console provides a feature to resume the upgrade from the last point of failure (checkpoint). The application creates a checkpoint at several stages during installation. If the installation fails, resuming from checkpoint would skip all the prior successfully completed stages and restart the installation from the beginning of the specific stage where the last point of failure occurred.

You can also resume a failed Controller job from the CLI by passing the flag useCheckpoint=true as an argument after --args in your command.

If for some reason, the upgrade from the last checkpoint is not successful, you may retry the upgrade from the beginning. Simply click Retry instead of Resume from the checkpoint. However, please note that retrying from the beginning after a failed upgrade may overwrite your customizations to db.cnf and domain.xml.

Timing Out

A common upgrade issue involves the upgrade process timing out. The Enterprise Console attempts to restart the Controller and database after applying an update or installation. For large databases and depending on the system resources, this can take a considerable amount of time. If the Enterprise Console cannot finish starting up the Controller within a set time-out period (30 minutes by default), the operation will fail.

You can increase the default time out period for system startup. The timeouts are defined in platform-admin/archives/controller/<version>/playbooks/controller.groovy. You can update the controllerStartRetryTimeout = 10 * 60 seconds = 10 minutes, and then retry the upgrade from the checkpoint.

When the Controller upgrade is complete, audit reports may not work immediately. The audit database table is getting migrated only after the upgrade process and the migration takes at least an hour to complete. If audit reports are run before completing the migration process, audit table migration messages are logged in the server.log file. No actions are required, try running the audit reports again after an hour.