Instrument JVMs in a Dynamic Environment

Application environments are increasingly dynamic. For example, in AWS or Rackspace hosted environments, an administrator can online or offline nodes quickly because of  workload resulting in an environment where there can be many, short-lived JVMs.  

Dynamic Environment Considerations

The AppDynamics app agent operates substantially the same way in a dynamic environment as it does in a traditional data center. However, these considerations apply when deploying agents in dynamic environments:

• Ensure that the agents can reach your Controller. Particularly if your Controller operates on-premises while the agents run on hosted servers, you need to ensure that the agents can access the Controller through the firewalls in your environment. 

• Configure custom node expiration handling. By default, the Controller manages the lifecycle for a node based on default timeout settings. You will likely want to reduce the default expiration times for nodes, as described in Historical and Disconnected Nodes. To manage this directly, rather than relying on activity timeouts, a strategy would be to run a script in the JVM before it shuts down that invokes the mark-nodes-historical resource in the AppDynamics REST API to declare itself historical to the Controller. 

• Depending on the nature of your environment, it may not make sense to track each JVM instance as a distinct node in AppDynamics, since the set of nodes, in this case, would be boundless. Also, since the JVMs are identical, representing them as different nodes in AppDynamics does not best reflect the logical model of the environment. 

Configure the Agent for Dynamic Environments

You can configure AppDynamics to reuse node names for dynamic environments. With node name reuse enabled, after a node is shut down and marked as historical in AppDynamics, the name of the node becomes available for reuse in subsequently registered nodes. You can also specify a node name prefix when reusing node names to indicate the association between all concurrently running JVMs for the dynamic application. 

To configure the agent for this environment, use these properties:  

• Reuse Node Name Property - To enable re-use of node names.
• Auto Node Name Prefix Property - To set the prefix used for automatically-named nodes.

When enabled, the Controller manages name assignments for the agents. It forms node names by appending a sequentially incremented number as the suffix to the prefix you specify. Be sure to avoid specifying the node name using any other mechanism (such as in controller-info.xml file or through a system property).

This listing shows sample settings for the properties as they may appear in a startup script:

-Dappdynamics.agent.reuse.nodeName=true
-Dappdynamics.agent.reuse.nodeName.prefix=CloudActivator_

When you have enabled reuse node name, the Java Agent logs to standard output after starting up and before registration with the Controller. To troubleshoot agent registration failure, you may want to write the log files to a file instead.

To configure the agent to write logs to a file, edit the log4j-unknown.xml at <agent_home>/<version_number>/conf/logging and uncomment the AgentLog Appender:

<!-- To log files to an "unknown" folder before the agent registers 
                          with the Controller, uncomment the line below -->
<appender-ref ref="AgentLogAppender"/>

This change causes the log files of an agent prior to registration to be written to a directory named "unknown" rather than standard output.   

Mark Dynamic Nodes as Historical at Shutdown

As indicated in the Configuring the Agent for Dynamics Environments section, setting the node name reuse flag causes the JVM to report graceful shutdowns, allowing the Controller to mark the corresponding node as historical. If you are not reusing node names, you can use one of these alternative methods to mark a node as historical:

• Set this system property at JVM startup:

  -Dappdynamics.jvm.shutdown.mark.node.as.historical=true

  JAVA

• Set this property in app-agent-config.xml to true:

  <app-agent-configuration>
     ...
     <configuration-properties>
        ...
        <property name="appdynamics.jvm.shutdown.mark.node.as.historical" value="true"/>

  CODE

  Note

  If your application environment abortively terminates JVMs rather than allowing them to shut down gracefully, arrange for your orchestration framework to use the controller API (Configuration API > Mark Nodes as Historical) to mark the node as historical at the same time as it terminates the node.

Controller Handling of Historical Nodes

If a node has been out of contact with the Controller for a certain amount of time, the Controller marks the node as a historical node. The Controller suspends certain types of processing activities for the node, such as rule evaluation. See Historical and Disconnected Nodes.