On this page:
A High Availability (HA) Controller deployment helps you minimize the disruption caused by server or network failure, administrative downtime, or other interruptions. An HA deployment is made up of two Controllers, one in the role of the primary and the other as the secondary.
The High Availability (HA) Toolkit automates many of the configuration and administration tasks associated with a highly available deployment.
Even if you cannot use the toolkit directly (due to specific requirements or operating system compatibility) the toolkit can provide a model for your own tools and procedures. Essentially, to set up high availability for Controllers, you are configuring master-master replication between the MySQL instances on the primary and secondary Controllers.
An important operational point to note is that while the databases for both Controllers should be running, both Controller application servers should never be active (i.e., running and accessible by network) at the same time. Similarly, the traffic distribution policy you configure at the load balancer for the Controller pair should only send traffic to one of the Controllers at a time (i.e., do not use round-robin or similar routing distribution policy at the load balancer).
Overview of High Availability
Deploying Controllers in an HA arrangement provides significant benefits. It allows you to minimize the downtime in the event of a server failure and take the primary Controller down for maintenance with minimal disruption. It fulfills requirements for backing up the Controller data, since the secondary maintains an updated copy of the Controller data. The secondary can also be used to perform certain resource-intensive operations that are not advised to be performed on a live Controller, such as performing a cold backup of the data or accessing the database to perform long-running queries, say for troubleshooting or custom reporting purposes.
In HA mode, each Controller has its own MySQL database with a full set of the data generated by the Controller. The primary Controller has the master MySQL database, which replicates data to the secondary Controller's slave MySQL database. HA mode uses a MySQL Master-Master replication type of configuration. The individual machines in the Controller HA pair need to have an equivalent amount of disk space.
The following figure shows the deployment of an HA pair at a high level. In this scenario, the agents connect to the primary Controller through a proxy load balancer. The Controllers must be equivalent versions. The App Servers and Controllers may be distributed in different data centers for redundancy.
In the diagram, the MySQL instances are connected via a dedicated link for purposes of data replication. This is an optional but recommended measure for high volume environments. It should be a high capacity link and ideally a direct connection, without an intervening reverse proxy or firewall. See Load Balancer Requirements and Considerations on Using the High Availability (HA) Toolkit for more information on the deployment environment.
The HA toolkit provides a set of scripts that automate many of the tasks associated with high availability for Linux environments. Even if you are unable to use the toolkit (due to operating system incompatibility or site-specific requirements), the toolkit can provide a model for your own processes.
In a high availability deployment, it is important that only one Controller is the active Controller at one time. Only the database processes should be running on the secondary to that it can maintain a replicated copy of the primary database.
The Controller app server process on the HA secondary can remain off until needed. Having two active primary Controllers is likely to lead to data inconsistency between the HA pair.
When a fail over occurs, the secondary app server must be started or restarted (if it is already running, which clears the cache).
Connecting Agents to Controllers in an HA Scenario
Under normal conditions, the App Agents and Machine Agents communicate with the primary Controller. If the primary Controller becomes unavailable, the agents need to communicate with the secondary Controller instead.
AppDynamics recommends that traffic routing be handled by a reverse proxy between the agents and Controllers, as shown in the figure above. This removes the necessity of changing agent configurations in the event of a failover, or the delay imposed by using DNS mechanisms to switch the traffic at the agent.
If using a proxy, set the value of the Controller host connection in the agent configuration to the virtual IP or virtual hostname for the Controller at the proxy, as in the following example of the setting for the Java Agent in the controller-info.xml file:
For the .NET Agent, set the Controller high availability attribute to true in config.xml. See .NET Agent Configuration Properties.
If you set up automation for the routing rules at the proxy, the proxy can monitor the Controller at the following address:
An active node returns an HTTP 200 response to GET requests to this URL, with
<available>true</available> in the response body. A passive node returns 503, Service Unavailable, with a body of
For more information, see Use a Reverse Proxy.