Database Size and Data Retention

This topic explains the default data retention periods for data stored by the AppDynamics On-Premises Controller.

Information Stored by the On-Premises Controller

The Controller stores a variety of information in its database, including configuration data, event data and transaction snapshots, metric data, and incident data (such as policy violations and very slow transactions). The amount of data stored by the database, and accordingly, the amount of disk space it consumes, is controlled by data retention settings.

You can tune the retention settings in the Administration UI. In particular, you may need to reduce retention settings to reduce the amount of data stored by the Controller. As a good starting point, lowering the snapshot retention (snapshots.retention.period) and hourly metric retention (metrics.retention.period) settings can significantly reduce disk consumption. In general, those settings are a good place to start when tuning your system to reduce disk space consumption.

Before Proceeding

It is important to understand that increasing the retention periods for the settings described on this page can significantly affect Controller performance and disk consumption. To understand the impact of changing a retention period, consider the example of cutting the minute metric retention period from the default of four hours to two. This would cut in half the amount of disk space required to keep metrics at the one minute level of granularity. On the other hand, doubling the default from four to eight would double the amount of disk space consumed by that type of metric.

The AppDynamics administration UI imposes allowed values on the input fields for the metric retention settings. However, the allowed values are based on theoretical limits and do not take into account the actual capacity of the current machine. For a given deployment, the practical limits for the settings depend on the hardware resources available to your Controller, the amount of application activity, the features you are using, and other factors.

The Controller machine sizing guidelines assume that default values are used for retention settings. Increasing the default values, therefore, can affect the assumption on which those guidelines are based. Increasing the retention settings, even within the allowed values permitted by the UI, could have adverse effects on performance or result in disk space exhaustion. AppDynamics strongly recommends that you monitor your system carefully if you change any of these settings from the default, and be prepared to roll back your changes if necessary.

Note that the practical limits for a SaaS Controller are different from those shown here. For the data retention settings applicable to your SaaS Controller, refer to the terms of your account.

Modifying Event, Transaction Snapshot, and Incidents Data Retention Settings

You can only change these settings if your Controller is on-premises.

You can modify the default retention periods for events, snapshot, and incidents data.

Before following these steps to modify retention settings, see Before Proceeding.

To change the retention period for event, snapshot, and incident data

Log in to the Controller administration console using the root account password. (See Access the Administration Console.)
```
http://<controller-installer-host>:<port>/controller/admin.jsp
```
Use the root user password to access the Admin console when the Controller is installed in single- or multi-tenant mode. This password is set at installation time.
Click Controller Settings.

Change the data retention period for incident and event data and transaction snapshots are determined by the following parameters:

Property Name	About the property	Default
events.retention.period	The retention time for events in hours.	Up to 336 hours (2 weeks), inclusive
snapshots.retention.period	Determines the retention time for transaction snapshot data in hours.	Up to 336 hours (2 weeks), inclusive
incidents.retention.period	Determines the data retention time for incidents, including policy violations or very slow transactions, in hours.	Up to 336 hours (2 weeks), inclusive

After you lower one of the data retention period settings, data in the database that falls outside the scope of the new data retention period is discarded. As a result, the size of the database should drop anywhere from 30 to 60 minutes after you make the change. If it does not, see Troubleshooting Controller Database Growth Issues.

Modifying Metric Data Retention Periods

You can only change these settings if your Controller is on-premises.

You can modify the default metric data retention periods. The metric retention periods control how long data is retained at 1-minute, 10-minute, and 1-hour resolution (see Metric Data Resolution over Time).

There are a few important points to note before changing metric data retention periods:

Setting longer retention periods for metric data can quickly and significantly affect database size and Controller performance. If you need to review details for an issue that occurred during a period for which you have only 10-minute or 1-hour data, AppDynamics provides access to diagnostic data at a more detailed level than might be visible on a graph, so increasing the default retention periods is not often necessary.
While metric retention periods apply to the data kept in the Controller database, some of the data that appears in the Controller UI — in particular, data at the 1 and 10-minute granularity level that appears in the tier and application dashboards — is actually drawn from the Controller cache rather than from the database. If you set the 1 and 10-minute metric retention periods to an overall retention period that exceeds the cache retention period, the performance of the Controller UI will be adversely affected, since the UI will have to retrieve the data from the database rather than from the cache. The cache retention period is determined by the caches.retention.period setting, which you can modify in the Administration Console, as described below.
Changing the data retention periods in the Controller Settings do not affect the values shown as time ranges in the Controller UI.

See Before Proceeding for more information about increasing data retention settings.

To change the data retention period for metrics

Log in to the Controller Administration Console.
Click Controller Settings.

Change the data retention period by modifying the values for the following settings and saving. The settings are:

Property name	About the property	Default
appdynamics.controller.sim.purge.machine.using.metric	Enable purging using only metadata information. Purging in this method does not use metrics data.	false
metrics.min.retention.period	The number of hours 1-minute data will be retained in hours. This value should always be less than the value for metrics.ten.min.retention.period.	4 hours
metrics.ten.min.retention.period	The number of hours 10-minute data will be retained in hours. This value should always be greater than the value for metrics.min.retention.period and less than metrics.retention.period.	48 hours
metrics.retention.period	The number of days 1-hour data will be retained in days. This value should always be greater than the value for metrics.ten.min.retention.period.	365 days
purger.bts.shortlived.expected.baseline	Expected number of short-lived Its to be purged in one purging attempt	100
purger.metrics.shortlived.expected.baseline	Short-lived metric purger's expected number of entities to be purged on every purging attempt.	1000
shortlived.metric.purger.time.interval.in.min	The interval frequency in minutes for the short-lived stale metric purger timer task to run.	30
shortlived.metric.purging.enabled	Enable the short-lived metric purger. When set to true, the short-lived metric purger timer task will purge all stale metrics which are short-lived. For example, container metrics.	true
shortlived.stale.metric.duration.in.hours	The duration of data in an hour, in which a metric that has not reported any data, is considered stale.	2

For example, if you change the metrics.min.retention.period property to 3, metric data displayed for all time ranges less than or equal to 3 hours are shown at one-minute resolution. Metric data for time ranges greater than 3 hours and less than metrics.ten.min.retention are shown at 10-minute resolution, and metric data for older time ranges are shown at 1-hour resolution.

As another example, if you change metrics.ten.min.retention to 72, all time ranges less than or equal to 72 hours (3 days), and greater or equal to min.retention.period are displayed at 10-minute resolution.

Troubleshooting Controller Database Growth Issues

If changing the data retention settings does not improve the rate of database growth for your system, first make sure you have restarted the Controller since making the configuration changes.

If working with support on troubleshooting, along with a description of the problem, provide a listing of your data directory. To generate the listing, run the following command on the Controller machine:

ls <Controller_Home>/db/data/controller/ -lS > controller.output

The command writes the output to the controller.output file, which you can attach to your support ticket.