Skip to end of metadata
Go to start of metadata

About Controller Backups

AppDynamics strongly recommends that you perform routine data backups of the Controller installation environment. Backups help to improve the disaster recovery preparedness of your organization. Any data generated since the last backup may be lost.

There are two general approaches to backing up the system, using a disk snapshot mechanism, such as Linux Volume Manager, or using database backup tools. The BackupTools section describes tools that support each approach.

While most backups are taken to guard against data loss in the event of a failure, they can also be used when upgrading the Controller and when migrating the Controller from one server to another.

Before Starting

Before attempting to perform database operations as described on this page, be sure to add the binary path for the Controller's MySQL instance in the PATH variable ahead of any other MySQL path on your system. This prevents conflicts with other database management systems on your machine, such as a MySQL instance included by default with Linux.

The database binary files for the Controller database are in <controller_install_dir>/db/bin.

Best Practices for Backups

Backing up the entire system each night may not be feasible when dealing with the large amount of data typically generated by a Controller deployment. To balance the risk of data loss against the costs of performing backups, a typical backup strategy calls for backing up the system at different scopes at different times. That is, you may choose to perform partial backups more frequently and full backups less frequently, relatively speaking.

The scope of a Controller backup can be categorized into these levels:

  • Level 1: A light backup of the installation environment only
  • Level 2: A metadata backup involving all metadata associated with the installation except big data tables.
  • Level 3: Backs up all data, either by performing a cold backup of the /data directory or a hot backup using a third-party tool.

Given these levels, a possible backup strategy would be to perform a level 1 and level 2 backup very frequently, say nightly, and a level 1 and level 3 backup about once a week. The following sections provide more details about these levels.

Light Backup (Level 1)

A light backup targets Controller configuration files like db.cnf, domain.xml, and so on. This type of backup lets you avoid having to reinstall and reconfigure the Controller in case of machine failure.

To perform this type of backup, simply shut down the Controller and copy everything in the Controller installation directory EXCEPT the data directory.

For a list of files that may contain customizations and therefore require backing up, see Migrate the Controller between Machines

Metadata Backup (Level 2)

A metadata backup exports the data that encapsulates the environment monitored by the Controller. Metadata defines the applications monitored by the Controller, business transactions, policies, and so on. It does not include what can be thought of as "runtime data", the big data tables that contain the metrics, snapshots, events, and top summary stats (top SQL, top URLs, and so on) generated in the monitored environment. By backing up metadata, you can avoid having to reconfigure monitored applications in the Controller in the event of a failure.

To perform this type of backup, run the script described in Using mysqldump to back up the Controller.

Complete Backup (Level 3)

A complete backup saves all runtime data associated with the Controller installation. It captures the actual metrics data, snapshots, and so on.

The AppDynamics Controller uses the default storage engine for MySQL, InnoDB. Since InnoDB supports transactions (in contrast to MyISAM), you cannot simply copy data files directly while the Controller is running. You need to stop the Controller before copying data files. However, some third-party backup tools, such as Percona XtraBackup, do not rely on transactions so you can perform a hot backup of your system (that is, back up the Controller database while it is running).

You can perform a complete backup either by performing a cold backup of the /data directory or a hot backup using a tool such as Xtrabackup, LVM snapshotting, or other third-party tool.

This type of backup allows you to skip the Level 2 backup, since it will include the metadata tables, but you would still need to perform the Level 1 backup to back up the Controller home directory.

For information on performing this type of backup, see instructions on using XtraBackup below or refer to the documentation provided with your backup tool.

Backup Tools

This section lists a few third-party tools that you can use to back up Controller data. The list is not exhaustive; you can use any tool capable of backing up MySQL data with the Controller. However, the tool should back up the data as binary data. Performing a complete backup with a tool that converts the data to text format, such as mysqldump, can put a significant burden on the system, especially if you have gigabytes of data in your system.

For Linux systems:

For Windows systems:

Disk snapshot tools

An alternative to backing up the data with a tool that works at the database level is to use a snapshot mechanism to back up the disk or partition on which the Controller data resides. Options are:

Details for performing this type of backup are beyond the scope of this documentation. For more information, refer to administration documentation applicable to your specific operating system.

Using mysqldump to Back Up the Controller

The mysqldump utility is a MySQL backup tool that is included with the Controller instance of MySQL.

While mysqldump is not recommended for use on large data tables, such as the Controller metric data tables, it is useful for backing up Controller metadata. Metadata defines the monitored domain for the Controller, including applications, business transactions, alert configurations, and so on.

Export Data with mysqldump

To use mysqldump to export data:

  1. Get the root user password for the Controller database. You can get the password for the root user from the following file: <Controller_install_directory>/db/.rootpw 

  2. Run the mysqldump executable, passing the root username, password, and output file. The executable is located in the following directory:

    <Controller_install_directory>/db/bin

    The command should be in the form:

    mysqldump -u root -p<password> <ignore-table_statements> > /tmp/metadata_dump.sql
    

For a full example that shows which tables to exclude when performing the backup, see the contents of the metadata backup script described next.

Sample mysqldump Script

The following script illustrates how to use mysqldump to export Controller metadata while excluding runtime data tables by script. 

To use the script, download the version appropriate for your operating system and modify it as described in the script comments.  

Import Data with mysqldump

After generating a mysqldump export file, you can import it into an empty database as follows:

$install/db/bin/mysql -u controller -psingcontoller controller < metadata_dump.sql

Use XtraBackup to Back Up Controller Data

Percona XtraBackup is a database backup tool that supports hot or cold backups.

To use XtraBackup with the Controller, you need to have version 1.6.4-314 or greater. See the Percona website at http://www.percona.com for information on getting and installing the tool.

After you install XtraBackup, use it by running the following two commands each time you want to back up the system. Note that the commands assume that the XtraBackup binary files (i.e., <xtrabackup_dir>/bin) are in the environment PATH variable.

innobackupex-1.6.4 --user=root --password=<mysql_root_user_password> --defaults-file=$INSTALL_DIR/db/db.cnf <backup_dir> --no-lock --socket=<path_to_mysql.sock>

innobackupex-1.6.4 --user=root --password=<mysql_root_user_password> --use-memory=1GB --defaults-file=$INSTALL_DIR/db/db.cnf --socket=<path_to_mysql.sock> --apply-log <backup_dir>/<time_stamped_dir>

You must run both commands to complete a successful backup. This first command generates the initial backup files and puts them in a timestamped directory in the location specified with <backup_dir>. The second command makes the data files internally consistent by resolving transactions started after the backup started to run.

Note: See Sample XtraBackup Script for a scripted version of the procedure in which the time-stamped name of the directory is parameterized for the second command.  

When running the commands:

  • Replace <mysql_root_user_password> with the password for the root user account in the Controller database. This password is stored in the <controller_install_dir>/db/.rootpw file or as the MYSQL_ROOT_PASSWD environment variable.
  • For the <path_to_mysql.sock> value, enter the fully qualified file name for the socket file (mysql.sock) on your system. You can get its location from the database log file, logs/database.log, in your Controller home directory. Look for the line that lists the MySQL version, as in the following example:

    Version: '5.5.16-log' socket: '/tmp/mysql.sock' port: 3388 MySQL Community Server (GPL)
    
  • In the first command, specify the location where you want to put the backup files as the <backup_dir>. In the second command, specify the full path to the backup directory generated by the first command as the final argument.

Sample XtraBackup Script

The following script illustrates how to use XtraBackup to back up Controller data in a Linux environment by script. Notice that the first innobackupex command generates a time-stamped directory with the backup artifacts. To be scriptable, the name of the generated directory needs to be parameterized for the script. The script illustrates how to do this.  

To use the script, download the script and set the parameter values in the script as appropriate for your environment, including the database user password (MYSQL_PASS) and the locations of the MySQL socket file (MYSQL_PASS), MySQL configuration file (MYSQL_CONF), and backup directory (BACKUP_DIR). 

#! /bin/sh
MYSQL_USER=root
MYSQL_PASS=
MYSQL_CONF=/etc/my.cnf
MYSQL_SOCK=/var/lib/mysql/mysql.sock
BACKUP_DIR=/opt/backup
innobackupex --user=$MYSQL_USER --password=$MYSQL_PASS \
        --defaults-file=$MYSQL_CONF --socket=$MYSQL_SOCK \
        --no-lock $BACKUP_DIR || exit 1
latest=`ls -dt $BACKUP_DIR/* | head -1`
innobackupex --user=$MYSQL_USER --password=$MYSQL_PASS \
        --defaults-file=$MYSQL_CONF --socket=$MYSQL_SOCK \
        --use-memory=1GB --apply-log $latest

You can download the script here: 

Once you have configured the script with parameter values, it can be set to run in an automated fashion, for example, as a Cron job. 

To restore Data using XtraBackup

  1. Shut down the MySQL database.
  2. Copy back the files in the data directory located at <Controller_Installation_Directory>/db. For example:

    innobackupex-1.6.4 --copy-back <Controller_Installation_Directory>/db
    
  3. Restart MySQL.

Backing Up and Restoring in High Availability (HA) Environments

When running the Controller in high availability mode, we recommend using XtraBackup or an LVM snapshot to perform a hot backup of the secondary database. This minimizes the impact of backing up on the primary Controller.

Hot backups are normally preferable to cold in this scenario because restarting the secondary after a cold backup typically imposes a performance burden on the primary controller, as the secondary controller tries to synchronize its data with the primary.

To create a backup in an HA environment, simply follow the instructions for using XtraBackup.

For information on recovering from a backup in a high availability environment, see HA failover and failback.

Using a Backup to Migrate to a New Physical Server

You can use either a hot or cold backup procedure to migrate Controller data to a new server. However, we recommend performing cold backups. While a hot backup does not bring down the Controller for an extended amount of time, it does introduce the possibility of data loss, since hot backups capture the state of the data only when the hot backup starts.

To perform a cold backup, simply shut down the Controller and back up the data directory located in <Controller_Installation_Directory>/db. 

Learn More

2 Comments

  1. Unknown User (anzianojackson@ldschurch.org)

    Look for the "set" statement for the mysql_root_user_password parameter, as in the following example:

    Shouldn't this be updated for 3.8.3 to something like "Navigate to <INSTALL DIR>db/.rootpw to verify password" ?

    1. Thanks for the catch, Colin! Fixed.