Related pages:

AppDynamics strongly recommends that you perform routine data backups of the Controller.

One method of maintaining backups of the Controller is to implement high availability. With high availability, the database on the secondary Controller keeps a replicated copy of the data on the primary Controller. A secondary Controller also makes it practical to take cold copies of the Controller data, since you can shut down the secondary to copy its data without affecting Controller availability. For information on HA, see Controller High Availability (HA). 

Other approaches include using a disk snapshot mechanism or using database backup tools. The BackupTools section describes tools that support each approach. In addition to regular backups, back up the Controller and Enterprise Console before upgrading or migrating them from one server to another.

This page provides an overview of the tasks and considerations related to backing up the Controller. Note that your Controller should be shut down before performing any import functions.

It is to be noted that controller versions 4.3 and later will work only on restoring and backing up the <Controller Home>/.appd.scskeystore file.

Best Practices for Backups

Backing up the entire system each night may not be feasible when dealing with the large amount of data typically generated by a Controller deployment. To balance the risk of data loss against the costs of performing backups, a typical backup strategy calls for backing up the system at different scopes at different times. That is, you may choose to perform partial backups more frequently and full backups less frequently.

The scope of a Controller backup can be categorized into these levels:

  • Level 1: A light backup of the installation environment only
  • Level 2: A metadata backup involving all metadata associated with the installation except for big data tables.
  • Level 3: Backs up all data, either by performing a cold backup of the /data directory or a hot backup using a third-party tool.

A possible backup strategy may be to perform a level 1 and level 2 backup very frequently, say nightly, and a level 1 and level 3 backup about once a week. In addition to performing a level 1 or 2 backup, you should also back up the data for the Enterprise Console with mysqldump on a regular basis. A level 3 backup also backs up the Enterprise Console data. See Enterprise Console Back Up and Restore for more information.

Light Backup (Level 1)

A light backup targets Controller configuration files like db.cnf and domain.xml. This type of backup lets you avoid having to reconfigure the Controller in case of machine failure.

To perform this type of backup, simply copy everything in the Controller installation directory EXCEPT the data directory.

While it is recommended that you copy the entire Controller home except for the data directory when performing a light backup, particularly before performing a Controller upgrade, there are scenarios in which you may wish to copy only site-specific configuration files. This may be the case if you are migrating an existing Controller configuration to a new Controller installation, for example. For a list of those files, see Migrate the Controller.   

Metadata Backup (Level 2)

A metadata backup exports the data that encapsulates the environment monitored by the Controller. Metadata defines the applications monitored by the Controller, business transactions, policies, and so on. It does not include what can be thought of as "runtime data", the big data tables that contain the metrics, snapshots, events, and top summary stats (top SQL, top URLs, and so on) generated in the monitored environment. By backing up metadata, you can avoid having to reconfigure monitored applications in the Controller in the event of a failure.

To perform this type of backup:

  1. Run the script described in Using mysqldump to back up the Controller.
  2. Then augment it with a copy of:
    1. The Controller Java keystore (600 byte file): <controller install>/.appd.scskeystore

    2. And the Enterprise Console keystore: platform-admin/.appd.scs

Complete Backup (Level 3)

A complete backup saves all runtime data associated with the Controller installation. It captures the actual metrics data, snapshots, and so on.

Some third-party backup tools, such as Percona XtraBackup, do not rely on transactions so you can perform a hot backup of your system (that is, back up the Controller database while it is running).

You can perform a complete backup as either:

  • A cold backup of the /data directory. A cold backup means that, with the Controller app server and database shut down, you create an extra copy of the Controller database using, for example, the cp -r command, the tar utility, rsync, or others.  
  • A hot backup, which means the Controller is running. For a hot backup, use a third-party tool such as Xtrabackup.

This type of backup allows you to skip the Level 2 backup and the Enterprise Console backup. You still need to perform the Level 1 backup to back up the Controller home directory.

Backup Tools

This section lists a few third-party tools that you can use to back up Controller data. The list is not exhaustive; you can use any tool capable of backing up MySQL data with the Controller. It is up to you to test your backup and restore process. However, the tool you decide on should back up the data as binary data.  

For Linux systems:

For Windows systems:

An alternative to using a database backup tool is to use a disk snapshot tool to replicate the disk or partition on which the Controller data resides. Options include:

Details for performing this type of backup are beyond the scope of this documentation. For more information, refer to administration documentation applicable to your specific operating system.

Back Up the Controller with mysqldump

The mysqldump utility is a MySQL backup tool that is included with the Controller instance of MySQL.

While mysqldump is not recommended for use on large data tables, such as the Controller metric data tables, it is useful for backing up Controller metadata. Metadata defines the monitored domain for the Controller, including applications, business transactions, alert configurations, and so on.

The following instructions assume that the binary path for the Controller's MySQL instance is in the PATH variable. The path to the Controller's instance of MySQL must precede any other MySQL path on your system. This prevents conflicts with other database management systems on your machine, such as a MySQL instance included by default with Linux.

The database binary files for the Controller database are in <controller_home>/db/bin.

Before using mysqldump, first ensure that the Controller app server is stopped. If you attempt to run mysqldump while the app server is running, it will severely degrade the performance and stability of the Controller. 

To use mysqldump, run the mysqldump executable, passing the root username, password, and output file. The executable is located in the following directory:

<controller_home>/db/bin 

The command should be in the form:

mysqldump -u root -p<password> <ignore-table_statements> > /tmp/metadata_dump.sql
CODE

For a full example that shows which tables to exclude for a metadata backup, see the contents of the metadata backup script described in the next section.

Sample mysqldump Script

The following script illustrates how to use mysqldump to export Controller metadata while excluding runtime data tables by script. 

Backing up the Controller with a custom MySQL data directory location using mysqldump will result in an incomplete and unusable metadata dump. The default location for the Controller's MySQL data directory is appd_install_dir/db/data. If you do not see the data directory here then that means an alternate directory or mount point was selected and configured for your MySQL data directory.

To fix this issue:

  1. Examine <Controller Home>/db/db.cnf
    1. Locate the datadir parameter. It contains the path to the MySQL data directory on your Controller host.
  2. Edit the metadata backup script
    1. Replace $appd_install_dir/db/data with the datadir path you located in db.cnf

To use the script:

  1. Download the version appropriate for your operating system.
  2. Rename the file to remove the .txt extension.
  3. Modify the contents of the file as described in the script comments. 

Import Controller Data with mysql

When you restore or migrate the Controller, you can import the data you exported with mysqldump.

Shut down your Controller before using the following command to import the data into a database:

$install/db/bin/mysql -u controller -p<ControllerDBpassword> < metadata_dump.sql

It will overwrite the tables. You can clone your installation to another host and test your restoration there.

A Controller that is installed with a custom MySQL data directory location requires additional flags.

Sample Data Backup Script

The following script uses Percona XtraBackup to back up Controller data. To use it, you need the percona-xtrabackup or xtrabackup and qpress packages. For information on installing XtraBackup, see the Percona installation documentation. 

To use the script, download the following file: 

Rename the script (by removing the .txt extension). In the script:

  • Verify or edit the values of the CONTROLLER_HOME and DESTINATION variables at the beginning of the script for your environment.
  • Edit the if/then/else clause at the end of the script if you want to implement backup file rotation, call your enterprise backup system to pick up the compressed Controller database image, or send an alert if the backup fails for any reason.

The following commands demonstrate how to restore a compressed backup image:

mkdir /path/to/big/staging/folder

# unpack the compressed backup archive
cd /path/to/big/staging/folder && xbstream -xv < /path/to/backups/dir/controller-yyyymmdd.xbstream
 
# decompress the backup image and apply the log taken during backup
CONTROLLER_HOME=/path/to/AppDynamics/Controller && cd /path/to/big/staging/folder \
&& innobackupex --decompress --parallel=16 . && innobackupex \
--defaults-file=$CONTROLLER_HOME/db/db.cnf --use-memory=1GB --apply-log --parallel=16 .

# Move a prepared backup into an empty controller data directory
CONTROLLER_HOME=/path/to/AppDynamics/Controller && cd /path/to/big/staging/folder \
&& innobackupex --defaults-file=$CONTROLLER_HOME/db/db.cnf --move-back .
BASH

For more information on these options, see the Percona innobackupex option reference.

Using a Backup to Migrate to a New Physical Server

You can use either a hot or cold backup procedure to migrate Controller data to a new server. However, we recommend performing cold backups. While a hot backup does not bring down the Controller for an extended amount of time, it does introduce the possibility of data loss, since hot backups capture the state of the data only when the hot backup starts.

To perform a cold backup, simply shut down the Controller and back up the data directory located in <controller_home>/db.