Infrastructure Monitoring in a Java Environment
A Java application environment has multiple functional subsystems. These are usually instrumented using JMX (Java Management Extensions) or IBM Performance Monitoring Infrastructure (PMI). AppDynamics automatically discovers JMX and PMI attributes.
JMX uses objects called MBeans (Managed Beans) to expose data and resources from your application. In a typical application environment, there are three main layers that use JMX:
- JVMs provide built-in JMX instrumentation, or platform-level MBeans that supply important metrics about the JVM.
- Application servers provide server or container-level MBeans that reveal metrics about the server.
- Applications often define custom MBeans that monitor application-level activity.
MBeans are typically grouped into domains to indicate where resources belong. Usually in a JVM there are multiple domains. For example, for an application running on Apache Tomcat there are “Catalina” and “Java.lang” domains. “Catalina” represents resources and MBeans relating to the Tomcat container, and “Java.lang” represents the same for the JVM Hotspot runtime. The application may have its own custom domains.
JVM Key Performance Indicators
There are often thousands of attributes, however, you may not need to know about all of them. By default, AppDynamics monitors the attributes that most clearly represent key performance indicators and provide useful information about short and long term trends. The preconfigured JVM metrics include:
- Total classes loaded and how many are currently loaded
- Thread usage
- Percent CPU process usage
- On a per-node basis:
- Heap usage
- Garbage collection
- Memory pools and caching
- Java object instances
You can configure additional monitoring for:
- Automatic leak detection
- Custom memory structures
Memory Usage and Garbage Collection
Monitoring garbage collection and memory usage can help you identify memory leaks or memory thrash that can have a negative impact application performance.
To view heap usage, garbage collection, and memory pools
- In the left navigation pane, click Servers - > App Servers -> <tier> -> <node>. The Node Dashboard opens.
- On the Node Dashboard, click the Memory tab.
- On the Memory tab, click the Heap & Garbage Collection subtab. The panels show data about the current usage.
See Node Dashboard for a complete description of the this dashboard.
The Heap panel shows data about the current usage.
Java garbage collection refers to how the JVM monitors the objects in memory to find any objects which are no longer being referenced by the running application. Unused objects are deleted from memory to make room for new objects. For details see the Java documentation for Tuning Garbage Collection.
Garbage collection is a well-known mechanism provided by Java Virtual Machine to reclaim heap space from objects that are eligible for garbage collection. The process of scanning and deleting objects can cause pauses in the application. Because this can be an issue for applications with large amounts of data, multiple threads, and high transaction rates, AppDynamics captures performance data about the duration of the pauses for garbage collection.
Below the Heap panel, the Garbage Collection - Time Spent panel shows how much time, in milliseconds, it takes to complete both minor and major collections.
The Garbage Collection - Minor Collections panel shows the number of minor collections per minute. The effectiveness of minor collections indicates better performance for your application.
The Garbage Collection - Major Collections panel shows the number of major collections per minute.
The Memory Pools panel shows usage and trends about the Java memory pools.
Classes, Garbage Collection, Memory, Threads, and Process CPU Usage Metrics
Information on JVM classes, garbage, threads and process CPU usage is available on the JMX Metrics subtab of the Node Dashboard JMX tab.
To view classes, garbage collection, memory, threads, and process CPU usage metrics
1. In the left navigation pane, click Servers - > App Servers -> <tier> -> <node>. The Node Dashboard opens.
2. In the Node Dashboard, click the JMX tab.
3. In the JMX Metrics subtab metric tree, click an item and drag it to the line graph to plot current metric data.
Alert for JVM Health
You can set up health rules based on JVM or JMX metrics. Once you have a health rule, you can create specific policies based on health rule violations. One type of response to a health rule violation is an alert. See Alert and Respond for a discussion of how health rules, alerts, and policies can be used.
You can also create additional persistent JMX metrics from MBean attributes. See Configure JMX Metrics from MBeans.
Monitor JVM Configuration Changes
The JVM tab of the Node Dashboard displays the JVM version, startup options, system options and environment properties for the node.
Changes to the application configuration generate events that can be viewed in the Events list.
For more information, see Monitor Application Change Events.
Detect Memory Leaks
By monitoring JVM heap utilization and memory pool usage you can identify potential memory leaks. Consistently increasing heap valleys may indicate either an improper heap configuration or a memory leak. You might identify potential memory leaks by analyzing the usage pattern of either the survivor space or the old generation. To troubleshoot memory leaks see Troubleshoot Java Memory Leaks.
Automatic Leak Detection
AppDynamics supports automatic leak detection for some JVMs as listed in JVM Support. By default this functionality is not enabled, because using this mode results in higher overhead on the JVM. AppDynamics recommends that you enable leak detection mode only when you suspect a memory leak problem and that you turn it off once the leak is identified and remedied.
Memory leaks occur when an unused object’s references are never freed. These are the most common occurrences in collections classes, such as HashMap. This is caused when an application code puts objects in collections but does not remove them even when they are not being actively used. In production environments with high workloads, a frequently accessed collection with a memory leak can cause the application to crash.
AppDynamics automatically tracks every Java collection (HashMap, ArrayList, and so on) that has been alive in the heap for more than 30 minutes. The collection size is tracked and a linear regression model identifies if the collection is potentially leaking. You can then identify the root cause of the leak by tracking frequent accesses of the collection over a period of time.
View this data on the Node Dashboard on the Automatic Leak Detection subtab of the Memory tab. See Node Dashboard for a complete description of this dashboard and its tabs.
To enable automatic leak detection
To enable automatic leak detection follow the instructions at Troubleshoot Java Memory Leaks.
Detect Memory Thrash
Memory thrash is caused when a large number of temporary objects are created in very short intervals. Although these objects are temporary and are eventually cleaned up, the garbage collection mechanism may struggle to keep up with the rate of object creation. This may cause application performance problems. Monitoring the time spent in garbage collection can provide insight into performance issues, including memory thrash. For example, an increase in the number of spikes for major collections either slows down a JVM or indicates potential memory thrash. To troubleshoot memory thrash, see Troubleshoot Java Memory Thrash.
Object Instance Tracking
The Object Instance Tracking subtab helps you isolate the root cause of possible memory thrash. By default,AppDynamics tracks the object instances for the top 20 core Java classes and the top 20 application classes. For the list of the supported JVMs see the Compatibility Matrix for Memory Monitoring.
The Object Instance Tracking subtab provides the number of instances for a particular class and graphs the count trend of those object in the JVM. It provides the shallow memory size (the memory footprint of the object and the primitives it contains) used by all the instances.
To monitor Java object instances
- Ensure the tools.jar file is in the jre/lib/ext directory.
- On the Node Dashboard, click the Memory tab.
- On the Memory tab, click the Object Instance Tracking subtab.
- Click On and then OK.
Monitor Long-lived Collections
AppDynamics automatically tracks long lived Java collections (HashMap, ArrayList, and so on) with Automatic Leak Detection. You can also configure tracking of specific classes using the Custom Memory Structures capability. You can use this capability to monitor a custom cache or other structure that is not a Java collection. Custom memory structures are used as caching solutions. For example, you may have a custom cache or a third party cache such as Ehcache. In a distributed environment, caching can easily become a prime source of memory leaks. In addition, custom memory structures may or may not contain collections objects that would be tracked using automatic leak detection. It is therefore important to manage and track these memory structures.
AppDynamics provides visibility into:
- Cache access for slow, very slow, and stalled business transactions
- Usage statistics (rolled up to Business Transaction level)
- Keys being accessed
- Deep size of internal cache structures
Ensure your custom memory structures are supported on your JVM, see JVM Support.
To view or configure custom memory structures
1. In the Node Dashboard, click the Memory tab.
3. On the Memory tab, click the Custom Memory Structures subtab.
For details see Configure and Use Custom Memory Structures for Java.