A large scale AppDynamics deployment typically requires some additional performance tuning, both to the Controller configuration and to the host environment. A large scale deployment is one in which the Controller monitors 250 or more nodes. This matches the Extra Large performance profile, as defined in Controller Performance Profiles.

The recommendations on this page are provided as general guidelines only. For details on the recommendations and for advice specific for your Extra Large profile deployment, consult with your AppDynamics Support representative.

In addition to the guidelines and configuration settings described here, it is especially important that any machine that hosts the Controller in high workload environments meets Controller System Requirements.

Linux Settings

Use the following settings for the operating system on which the Controller runs:

Use the Deadline scheduler.
Set swappiness to 0. See Configure Swappiness on Linux for more information.
Set the open file limit to 819200 or greater. See Configure File Descriptor Limits on Linux for more information.
Set the per-process open file limit for soft and hard limits to 819200 or greater.
Allow Web server to retry longer during stalls by setting higher TCP timeouts.
If using the Splunk extension, increase the maximum number of user processes to 2048.

File System and RAID Recommendations

Recommended file system: XFS.
Recommended RAID version: There are only minor performance differences between RAID 5, 6, and 10. The reasons to use one over another are purely for level of safety and space requirements. The more disks you have in any RAID configuration, the better the performance, since you have more stripes to work concurrently in doing disk I/O.
Recommended RAID configuration:
- Use a RAID Controller with a Battery Backup Unit (BBU) to allow the database to use O_DIRECT sync mode as the flush method for faster disk writes. Never set O_DIRECT without a BBU. You can specify this setting using the innodb_flush_method variable in <Controller_install_directory>/db/db.cnf. When not specified in the file (as is the case by default), MySQL uses the default flush method, fdatasync. See the MySQL documentation for more information.
- Disable individual hard drive from using onboard cache. All caching is to be done by RAID Controller.
For Storage Area Network (SAN) deployments, use multipath I/O for optimal performance and availability.
For either SAN or RAID controller systems, use the following file system mount options for your database storage mountpoint: "noatime,nodiratime,nobarrier,data=writeback"
On board disks are the easiest option to assure performance.
When you use remote disks (such as SAN or NAS) the configuration must be able to support very heavy disk I/O with very low latency. Requirements for the Controller database are similar to any production database that you run.
LVM partitions can be useful especially for doing quick hot backups with LVM snapshots.

Glassfish Configuration

New in 3.7 The Glassfish server used by the Controller is now version 3.1.2.2, and the path to the domain.xml file changed. If you have previously applied Glassfish settings as described in this document, you need to update those settings as per the instructions below.

The Controller uses Glassfish version 3.1.2.2 as its embedded application server. For better scalability and performance, you should tune Glassfish with settings based on your final hardware sizing.

For more information about configuring Glassfish see the Glassfish documentation.

To configure Glassfish

Shut down the Controller.
Edit the following file under the Controller home directory: /appserver/glassfish/domains/domain1/config/domain.xml
The settings in the file are described in To update the Glassfish settings.
Restart the Controller.

To update the Glassfish settings

Set the thread count (XX) to a value of 12 X # of CPU cores.
In the tag <thread-pools> change the line containing "http-thread-pool" to:

<thread-pool name="http-thread-pool" max-thread-pool-size="XX" min-thread-pool-size="16" max-queue-size="32768"></thread-pool>

If your Controller instance was upgraded from an earlier version, delete the following element if present, as it applies to Glassfish v2 only:

<request-processing header-buffer-length-in-bytes="8192" initial-thread-count="16" request-timeout-in-seconds="300" thread-count="XX" thread-increment="1"/>

Replace the TCP transport element under server-config with the following. Here we've set the depth of the connection pool queue to 32K to allow for connections to queue up instead of being dropped during peak load bursts. Replace the XX value in the acceptor-threads attribute with the number of cores in your system.
```
<transport buffer-size-bytes="32768" max-connections-count="32768" name="tcp" keep-alive="true" acceptor-threads="XX"></transport>
```
Note that there are two transport elements with the name value of tcp in the file. Be sure to modify the first one, below the server-config element.

If your Controller instance was upgraded from an earlier version, delete the following element if present, as it applies to Glassfish v2 only:
```
<connection-pool max-pending-count="32768" queue-size-in-bytes="32768" receive-buffer-size-in-bytes="32768" send-buffer-size-in-bytes="32768"/>
```

Increase the buffer size. In the <protocols> section under <protocol name="http-listener-1">, change the next line to:

<http request-timeout-seconds="300" max-connections="-1" request-body-buffer-size-bytes="32768" header-buffer-length-bytes="32768" timeout-seconds="300" default-virtual-server="server" send-buffer-size-bytes="32768" compressable-mime-type="text/html, text/javascript, text/css" compression="on">

Set the Heap size to approximately 20 to 40% of the RAM size on your machine: higher Heap size for lower RAM sizes, lower Heap for high-end RAM sizes. Set Xmn to 1/3 of the Xmx setting.
```
<jvm-options>-Xmx30g</jvm-options>
<jvm-options>-Xms30g</jvm-options>
<jvm-options>-Xmn10g</jvm-options>
```

Replace the garbage collection settings with these specially tuned settings:

<jvm-options>-XX:+UseConcMarkSweepGC</jvm-options>
<jvm-options>-XX:+UseParNewGC</jvm-options>
<jvm-options>-XX:+ScavengeBeforeFullGC</jvm-options>
<jvm-options>-XX:TargetSurvivorRatio=80</jvm-options>
<jvm-options>-XX:SurvivorRatio=6</jvm-options>
<jvm-options>-XX:+UseBiasedLocking</jvm-options>
<jvm-options>-XX:MaxTenuringThreshold=15</jvm-options>
<jvm-options>-XX:ParallelGCThreads=16</jvm-options>
<jvm-options>-XX:+OptimizeStringConcat</jvm-options>
<jvm-options>-XX:+UseStringCache</jvm-options>
<jvm-options>-XX:MaxPermSize=256m</jvm-options>
<jvm-options>-XX:+UseCompressedOops</jvm-options>
<jvm-options>-XX:+UseCMSInitiatingOccupancyOnly</jvm-options>
<jvm-options>-XX:CMSInitiatingOccupancyFraction=95</jvm-options>

MySQL Configuration

The Controller uses MySQL as its the database server. You should tune MySQL for scalability and better performance.

To configure MySQL

Shut down the Controller.

Configure the settings in the database configuration file, <Controller_Installation_Directory>/db/db.cnf, with these values. Since some of these name/value pairs are already in the db.cnf file, search in the file for each of the variables by name. If a variable already exists, simply replace the assigned value. Otherwise, add the name/value pair to the file.

thread_cache_size=120
table_definition_cache=500
open-files-limit=40960
innodb_open_files=3000
table_open_cache=4000
lock_wait_timeout=300
query_cache_size=0
long_query_time=4
innodb_flush_method=O_DIRECT # ONLY DO IF YOU HAVE BATTERY BACKED RAID !
innodb_buffer_pool_size= 11264M # This should be based on available RAM. This value should be approximately 40 to 60% of the RAM size.

Restart the Controller.
Your database should now be running with the new settings, but we suggest that you verify your database settings. To do so:
1. Log in to the database:
```
<Controller_installation_directory>/bin/controller.sh login-db
```
2. Enter the "SHOW VARIABLES;" command and verify that the variables are assigned the values you expect as reported in the command output.

Controller Configuration

You need to adjust some Controller settings for configurations that handle large amounts of traffic. This includes increasing the events, snapshots, and buffer sizes, increasing the read and write thread counts, decreasing the node retention and node permanent deletion periods, etc.

To change the Controller settings

Make sure Controller is running.
Log into the Controller Administration console at <host>:<port>/controller/admin.html using the root password.
See Access the Administration Console.
In the Administration Console, select Controller Settings.
Modify each of the following settings, clicking Save to save each update.
- disable.historic.transactional.flow = true
  Allows dashboards to draw quickly by using cache values from the cache setting above. For large deployments this should always be set this to true to avoid timeout errors.
- business.transaction.retention.period = 48
- events.buffer.size = 50
- snapshots.buffer.size = 200
- metrics.buffer.size = 600
- node.permanent.deletion.period = 168
- node.retention.period = 6
- read.thread.count = 3
  Number of threads to use simultaneously to do READS from the database. Set this to 20% of the number of CPU cores but not greater than 4.
- write.thread.count = 4
  Number of threads to use simultaneously to load data into the database. Set this to the same value as read.thread.count but not greater than 4.

Additional Setup for Environments with More than 500 Nodes

Additional recommendations apply for environments that consists of more than 500 nodes. For these environments, AppDynamics recommends that you split traffic from the UI and App Agents to different ports on the Controller, as described below.

In addition, AppDynamics recommends terminating SSL at a load balancer or HTTP proxy in front of the Controller. This alleviates the workload of SSL processing from the Controller. For information on configuring a reverse proxy, see Deploy with a Reverse Proxy.

Split Traffic on Two HTTP listeners

The Controller is preconfigured with two active HTTP listeners, one for SSL and the other for non-SSL. While you can create an additional HTTP listener, in most cases it will make sense to split traffic from agents and the UI between the two existing ports.

To split traffic on two HTTP listeners

Edit the domain.xml file. There are already two listeners (look for “http-listener” tags), one for SSL and another for non-SSL access. One of them is being used by your agents. Find the other one, and use it as your GUI listener.

Change the port number and remove the SSL line if needed. The GUI listener should look similar to the following example:

<http-listener acceptor-threads="1" address="0.0.0.0" blocking-enabled="false" default-virtual-server="server" enabled="true" family="inet" id="http-listener-1" port="8080" security-enabled="false" server-name="" xpowered-by="true">
<property name="proxiedProtocols" value="ws/tcp"/>
</http-listener>

Learn More

Tuning for Large Scale Deployments