Performing a full Controller upgrade, an upgrade of MySQL, or changing the database configuration in db.cnf
(through the Enterprise Console) may cause mysqld
to crash at shutdown, potentially resulting in data corruption and/or loss.
The crash is due to a known MySQL bug, MySQL Bug #95285 which results in mysqld_safe performing an unexpected restart of mysqld
. Processes performing a MySQL shutdown expect it to be down and subsequent operations based on that expectation can cause MySQL corruption. For example:
Reboots and other service-level operations usually initiate a MySQL shutdown and then proceed after a timeout without checking for a clean complete shutdown
Enterprise Console performs log-file management operations after a shutdown and prior to a MySQL upgrade or db.cnf
setting change
On-premises Controllers with MySQL v5.7.26 or higher (see the database.log banner)
You must apply the Oracle workaround, particularly if you plan on performing an upgrade. |
For a Standalone Controller
Before you start this procedure, you must take the system down because a Controller restart is required.
Perform a backup of <Controller Home>/db/db.cnf
, enter:
cp <Controller Home>/db/db.cnf <Controller Home>/db/db.cnf.pre-oracle-patch |
<Controller Home>/db/db.cnf
.internal_tmp_disk_storage_engine=INNODB
to internal_tmp_disk_storage_engine=MYISAM
.Shut down the Controller App Server, enter:
<Controller Home>/bin/controller.sh stop-appserver |
Shut down the Controller database, enter:
<Controller Home>/bin/controller.sh stop-db |
Verify that the database is stopped by checking the process, enter:
ps -ef | grep mysqld |
mysqld"
or "mysqld_safe
" process should be running from the <Controller Home>/db
directory.mysqld
" and "mysqld_safe
" processes running from your <platform>/mysql
directory. You can ignore them.mysqld
" or "mysqld_safe
" processes running from the <Controller Home>/db
directory, re-run the "Shutdown the Controller database" command until the processes are shut down.<Controller Home>/db/db.cnf
.internal_tmp_disk_storage_engine=INNODB
to internal_tmp_disk_storage_engine=MYISAM
.For an HA Controller
Before you start this procedure, you must take the system down because a Controller restart is required.
Perform a backup of <Controller Home>/db/db.cnf
on both servers, enter:
cp <Controller Home>/db/db.cnf <Controller Home>/db/db.cnf.pre-oracle-patch |
<Controller Home>/db/db.cnf
.internal_tmp_disk_storage_engine=INNODB
to internal_tmp_disk_storage_engine=MYISAM
.Restart the secondary database. To stop the secondary database, enter:
<Controller Home>/bin/controller.sh stop-db |
Verify that the secondary database is stopped by checking the process, enter:
ps -ef | grep mysqld |
mysqld"
or "mysqld_safe
" process should be running from the <Controller Home>/db
directory.mysqld
" and "mysqld_safe
" processes running from your <platform>/mysql
directory. You can ignore them.mysqld
" or "mysqld_safe
" processes running from the <Controller Home>/db
directory, re-run the "Shutdown the Controller database" command until the processes are shut down.Start the secondary database, enter:
<Controller Home>/bin/controller.sh start-db |
If you manage HA using the HA Toolkit, then enter:
[[ -z $(pgrep -f "[w]atchdog.sh") ]] || kill -9 $(pgrep -f "[w]atchdog.sh") ps -eaf | grep "[w]atchdog.sh" |
No "HA/watchdog.sh
" process should be running. To verify, enter:
<Controller Home>/HA/appdstatus.sh |
On the primary, shut down the Controller App Server, enter:
<Controller Home>/bin/controller.sh stop-appserver |
Shut down the primary database. To stop the primary database, enter:
<Controller Home>/bin/controller.sh stop-db |
Verify that the primary database is stopped by checking the process, enter:
ps -ef | grep mysqld |
mysqld"
or "mysqld_safe
" process should be running from the <Controller Home>/db
directory.mysqld
" and "mysqld_safe
" processes running from your <platform>/mysql
directory. You can ignore them.mysqld
" or "mysqld_safe
" processes running from the <Controller Home>/db
directory, re-run the "Shutdown the Controller database" command until the processes are shut down.<Controller Home>/db/db.cnf
.internal_tmp_disk_storage_engine=INNODB
to internal_tmp_disk_storage_engine=MYISAM
.AppDynamics is actively working on wrapper code for the shutdown procedure as a fix until Oracle issues a long-term solution.
05/05/2020, v1 (initial publication of this advisory)