Trace Multithreaded Transactions for Java

The Java Agent can automatically track multithreaded applications. It correlates threads from classes that implement Runnable or Callable interfaces or extend the Thread class. Processes with such threads are identified in AppDynamics as asynchronous. 

Monitor Asynchronous and Thread Processing Activity

You can monitor performance around asynchronous calls and multithreaded activity in a variety of ways. It's helpful to understand what AppDynamics considers to be an asynchronous activity, what information it provides on that activity, and how that information relates to thread execution at the code level. 

These sections describe the type of information available given various thread and exit call execution scenarios. 

Asynchronous Exit Call

In the most common case, an asynchronous exit call is an exit call made by a thread that is spawned by a thread other than the primary business transaction thread.

The primary thread, in this case, is the one on which an entry point call is received. A common pattern for this case is when the main() function in the application spawns multiple threads as request handlers to receive and process requests. For example: 

Flow maps in the Controller UI show this connection as a dotted line (asynchronous connection).

AppDynamics considers the call to be an asynchronous call only due to the context in which it was invoked. If the exit call is made from thread 2, as shown in the following illustration, the call may be considered asynchronous from the perspective of the overall design of the application, but is not considered asynchronous in the context of the business transaction: 

Without a business transaction, AppDynamics does not track asynchronous activity.

AppDynamics does not differentiate between the two common patterns in asynchronous processing, Wait-for-Completion (in which the threads report back into the primary thread) and Fire-and-Forget (in which the primary thread may finish processing before spawned threads do). Both are represented the same way in the Controller UI, although they would usually be distinguished by their execution time, in which the primary thread for a Wait-for-Completion transaction would exceed any of the child processes, whereas Fire-and-Forget the primary thread may be shorter than child processes.   

Logical Asynchronous Exit Calls

An exception to the scenario described in the previous section exists for exit calls to backend systems which—from a conceptual point of view—are considered asynchronous in the AppDynamics model.

Message queue tiers such as JMS are considered logically asynchronous, so flow maps show connections to message queues as dotted lines (asynchronous connections) as well. Therefore, the following JMS exit call would be shown as an asynchronous call in a flow map, while the JDBC call would not be.  

Time on Tier

Exit calls are one consideration in understanding multithreaded application performance; another consideration is the actual time spent processing a request.

For example, consider the following processing scenario. This occurs on a single tier. In fulfilling a request, the entry point thread spawns multiple child threads that report back to the primary thread to assemble the response.    

For the transaction, the response time metric represents the time from when the request arrived at the entry point to the point the response was sent. However, while this may accurately represent the user experience of the transaction, it does not reflect the processing burden on the system, since many threads participate in the transaction. A slow performing thread may or may not be evident based on the user experience of the transaction.

To better understand the complete cost of the transaction in terms of time spent processing, you can use the time on tier metric. This metric shows the aggregate processing time for each of the threads involved in processing a request. In other words, the total execution time of threads 2, 3 and 4 from the figure, along with the processing time between the entry point and response for thread 1.

The value of the time on tier metric is indicated by the async tag in business transaction dashboards. 

End-to-End Processing

A response from an entry point thread may not best represent the logical end of a business transaction.

For example, consider an application in which an entry point method in a request handler spawns multiple threads, including one to serve as the final response handler. The request handler then returns a preliminary response to the client. By default, this stops the clock for purposes of measuring the response time for that business transaction. Meanwhile, the spawned threads continue to execute until completion, at which point the response handler generates the final response to the client.

In this case, the initial response time is much shorter than the full logical transaction execution time.

End-to-end metrics let you monitor logical transactions that use this programming pattern. End-to-end metrics include the end-to-end transaction time, the number of end-to-end transactions per minute, and the number of slow end-to-end message processing events. 

To enable end-to-end metrics, you configure an Asynchronous Transaction as described in Asynchronous Transaction Demarcators.

View Threads and Thread Tasks in the Metric Browser

In a multithreaded transaction, AppDynamics reports key business transaction performance metrics for individual threads in a Thread Tasks branch of the tier in the Metric Browser. The Thread Tasks branch is present in the Metric Browser only for multithreaded transactions. The Metric Browser path is Business Transaction Performance > Business Transactions  > tier-name > business-transaction-name > Thread Tasks. Thread Tasks also appear for tiers under Overall Application Performance, where you can see metrics on specific calls made by each thread in a node or in a tier.

Thread Metrics in Health Rules

For each asynchronous thread spawned in the course of executing a business transaction, AppDynamics collects and reports these metrics:

• Average response time
• Calls per minute
• Errors per minute

You can create custom health rules based on the performance metrics for a thread task. When you click the metric icon in the Health Rule Wizard, the embedded metric browser includes the Thread Tasks if the entity for which you are configuring the health rule spawns multiple threads. See Configure Health Rules.