Database Size and Data Retention

Information Stored by the Controller

The Controller stores configuration data, event data, transaction snapshots, metric data, and incident data (such as policy violations and very slow transactions) in its database. The amount of data stored and the amount of disk space it consumes is controlled by data retention settings.  

These retention settings can be tuned in an on-premises deployment in the Administration Console. For example, lowering the snapshot retention (snapshots.retention.period) and hourly metric retention (metrics.retention.period) settings can significantly reduce disk consumption.

Before Starting

Increasing the retention periods for the settings for incident, event data, and transaction snapshots can significantly affect Controller performance and disk consumption. AppDynamics recommends recording your current retention settings before editing the default values if you need to roll back modified settings.

It is important to be cautious when modifying these settings. For example, cutting the default value of metrics.min.retention.period in half reduces the disk space required to keep metrics at the one minute level of granularity. However, doubling the default value can double the amount of disk space consumed by that type of metric.   

The Administration Console imposes allowed values in the input fields for metric retention settings based on theoretical limits regardless of the actual capacity of the current machine. For a given deployment, practical limits for the settings depend on hardware resources available to your Controller, application activity, features being used, and other factors. 

The Controller machine sizing guidelines assume that default values are used for retention settings. Therefore, increasing default values can affect the assumption on which those guidelines are based. Increasing the retention settings, even within the allowed values in the UI, can have adverse effects on performance or can result in disk space exhaustion. 

AppDynamics recommends that you monitor your system carefully if you change any of these settings from the default, and be prepared to roll back your changes if necessary.

Modify the Retention Settings

You can modify the default retention periods for events, snapshots, and incidents data.

Modifying retention settings can have significant effects. See Before Starting.

To change the retention period for event, snapshot, and incident data:

1. Log in to the Administration Console using the root account password. Use the root user password to access the Administration Console when the Controller is installed in single- or multi-tenant mode. This password is set during installation. See Access the Administration Console.

   http://<controller-installer-host>:<port>/controller/admin.jsp

2. Click Controller Settings.

Retention Settings

The data retention period for incident and event data and transaction snapshots are determined by using the following parameters: 

After you lower a data retention period value, the Controller discards the data in the database that is outside the scope of the new data retention period. As a result, the size of the database is reduced to 30–60 minutes after you make the change. If it does not reduce in size, see Troubleshooting Controller Database Growth Issues.  

Modify Metric Data Retention Periods

You can modify the default metric data retention periods, which control how long data is retained at 1-minute, 10-minute, and 1-hour resolution. See Metric Data Resolution over Time.

There are a few important points to note before changing metric data retention periods:

Setting longer retention periods for metric data can quickly and significantly affect database size and Controller performance.

• If you need to review details for an issue that occurred during a period for which you have only 10-minute or 1-hour data, AppDynamics provides access to diagnostic data at a more detailed level than might be visible on a graph, so increasing the default retention periods is not often necessary.
• While metric retention periods apply to the data kept in the Controller database, some of the data that appears in the Controller UI — in particular, data at the 1 and 10-minute granularity level that appears in the tier and application dashboards — is fetched from the Controller cache rather than from the database. If you set the 1 and 10-minute metric retention periods to an overall retention period that exceeds the cache retention period, the performance of the Controller UI will be adversely affected. This is because the Controller has to retrieve the data from the database rather than from the cache. The cache retention period is determined by the caches.retention.period setting, which you can modify in the Administration Console. 
• Changing the data retention periods in the Controller settings does not affect the values shown as time ranges in the Controller UI. 

Change Data Retention Period

To change the data retention period for metrics:

1. Log in to the Administration Console. 

2. Click Controller Settings.

3. Change the data retention period by modifying the values for the following settings:

   Property name	About the property	Default
   appdynamics.controller.sim.purge.machine.using.metric	Enable purging using only metadata information. Purging in this method does not use metrics data.	false
   metrics.min.retention.period	The number of hours 1-minute data is retained. This value should always be less than the value for metrics.ten.min.retention.period. Note that this property will vary in a non-timescale environment.	4 hours
   metrics.ten.min.retention.period	The number of hours 10-minute data is retained. This value should always be greater than the value for metrics.min.retention.period and less than metrics.retention.period.	48 hours
   metrics.retention.period	The number of days 1-hour data is retained. This value should always be greater than the value for metrics.ten.min.retention.period.	365 days
   purger.bts.shortlived.expected.baseline	The expected number of short-lived business transactions to be purged in one purging attempt.	100
   purger.metrics.shortlived.expected.baseline	The number of entities the short-lived metric purger expects to be purged on every purging attempt.	1000
   shortlived.metric.purger.time.interval.in.min	The interval frequency in minutes for the short-lived stale metric purger timer task to run.	30
   shortlived.metric.purging.enabled	Enable the short-lived metric purger. When set to true, the short-lived metric purger timer task purges all stale metrics which are short-lived. For example, container metrics.	true
   shortlived.stale.metric.duration.in.hours	The duration of data in an hour, in which a metric that has not reported any data, is considered stale.	2

4. Click Save.

For example, if you change the metrics.min.retention.period property to 3, metric data displayed for all time ranges less than or equal to 3 hours are shown at one-minute resolution. Metric data for time ranges greater than 3 hours and less than metrics.ten.min.retention are shown at 10-minute resolution, and metric data for older time ranges are shown at 1-hour resolution.

As another example, if you change metrics.ten.min.retention to 72, all the time ranges less than or equal to 72 hours (3 days), and greater or equal to min.retention.period are displayed at 10-minute resolution.

Troubleshoot Controller Database Growth Issues

If changing the data retention settings does not improve the rate of database growth for your system, ensure you restart the Controller after making the configuration changes.

If working with the support team on troubleshooting, provide a list of your data directory along with a description of the problem. Generate the list by running the following command on the Controller machine:  

ls <Controller_Home>/db/data/controller/ -lS > controller.output

The command writes the output to the controller.output file, which you can attach to your support ticket. 

The limits for a SaaS Controller are different from those of an on-premises Controller. For the data retention settings applicable to your SaaS Controller, refer to the terms of your account.

Data Retention Settings

The properties below are the default retention periods for events, snapshots, and incidents data for SaaS Controllers.

Metric Data Retention Periods

The following properties are the default retention periods of metric data for SaaS deployments: