On this page:
Multithreaded programming techniques are common in applications that require asynchronous processing. Although each thread has its own call stack, multiple threads can access shared data. This creates two potential problems for application performance:
- A visibility problem occurs if thread A reads shared data which is later changed by thread B, and thread A is not aware of the change.
- An access problem occurs if several threads are trying to access and change the same shared data at the same time.
Visibility and access problems can lead to:
- Liveness failure: Application performance becomes sluggish or stops processing also known as a deadlock.
- Safety failure: Race condition that results in difficult to discover programming errors.
Thread contention can occur when multiple threads attempt to access a synchronized method or block at the same time. If a thread remains in the synchronized method or blocks for a long time, the other threads must wait for access to shared resources. This situation has an adverse effect on application performance. Call graphs for multithreaded transactions enable you to trace thread creation in a business transaction and provide an aggregated view of the overall processing for transactions that spawn threads for concurrent processing.
Thread Visibility and Metrics
AppDynamics monitors asynchronous activities as first class entities with their own metrics to give you the information you need to see and act to correct these performance issues.
When applications spawn threads to perform concurrent tasks, you can monitor each thread as a separate entity, including exit calls and policies associated with a specific thread. By default, all Runnables, Callables and Threads are instrumented, except those that are explicitly excluded.
AppDynamics provides the flexibility to adjust the default monitoring. In some environments, if the default settings lead to too many classes being instrumented, you can create custom rules to exclude unnecessary classes. If you do not want to monitor any threads, you can completely disable asynchronous monitoring. This requires an agent restart. See Configure Multithreaded Transactions for Java.
AppDynamics provides thread visibility in dashboards, the metric browser and snapshots.
For each asynchronous thread spawned in the course of executing a business transaction, AppDynamics collects and reports metrics such as the following:
- Average response time
- Calls per minute
- Errors per minute
Asynchronous Activity in Dashboards
AppDynamics detects asynchronous calls in an application and labels them as "async" in the dashboards that display the asynchronous activity.
In the following Application Flow Map, you can see the calls per minute and average response time displayed on each flow line where asynchronous activities are detected. These metrics aggregate the metrics for asynchronous activities across all business transactions.
For improved visibility, async flow is labeled and displayed as dotted lines.
Business Transaction Dashboard
Asynchronous activity can be viewed in a hierarchical format with the originating activities encapsulating their respective spawned asynchronous activities. The tree view of a multithreaded business transaction flow shows the hierarchical view as well as the errors and time spent in asynchronous calls for separate business transactions. To expand the tree to see all the calls, right-click a tier name and select Expand All.
The following metrics are visualized in the Transaction Flow Tree View:
|Time Spent (ms)|
Average time spent by the specific activity and any spawned asynchronous activities. Percentage metrics are used to represent the fraction of time spent in a specific activity. Asynchronous activities do not have a percentage breakdown because each asynchronous activity is linked to the originating business transaction, but represents a separate logical entity throughout the execution of the business transaction.
|Calls||Number of calls made by a particular activity such as an asynchronous activity.|
Number of calls made per minute for a particular activity such as an asynchronous thread.
|Errors||Number of calls for a particular activity which resulted in errors.|
Number of calls made per minute for a particular activity which resulted in errors.
Trends for baselines are visualized using the data for the originating business transaction. Metrics for the asynchronous activities are not used in calculation of these trends.
The Transaction Scorecard reflects only the data for the originating business transaction. The scorecard metrics are not inclusive of the metrics for any asynchronous threads being spawned by the originating business transaction.
The Transaction Snapshot Flow Map for a transaction with asynchronous activity displays both synchronous time and time spent in asynchronous activity. The following are example screen shots of transactions with asynchronous activities:
AppDynamics displays the following metrics:On the transaction flow map,
Tier Response Time (ms)
Time spent processing at a particular tier for this business transaction. Only present for originating tier snapshots. (first in chain)
|2||Percentage of Time Spent (%)|
Percentage metric represents the fraction of time spent processing at a particular tier or in communication with other tiers/backends from the entire execution lifespan of a business transaction. Only present for "first in chain" snapshots. This metric does not include the processing time of the asynchronous activities
Asynchronous Activity Processing Time (ms)
Processing time of all asynchronous activities at this tier. This metric does not contribute to the overall tier response time because the activity is asynchronous by nature. This metric is calculated by adding the execution times of all asynchronous activities at a tier and the time spent in communication between other tiers and backends as follows:
Asynchronous Activity Processing Time = Asynchronous-activity-1-processing-time + Asynchronous-activity-2-processing-time + so on.
Execution Time (ms)
Time spent processing by the business transaction in all affected tiers and communication with other tiers and backends. This metric does not include processing time of the asynchronous activities. However, in the case of Wait-for-Completion, the originating business transaction will take a longer time processing the request due to blocking and waiting for all the activities to complete before proceeding.
The formula for this metric is calculated by summing up the processing times of a Business Transaction at a particular Tier/communication between Tiers/Backends as follows:
Execution Time = Time-spent-processing-in-Tier-1 + Time-spent-processing-in-Tier-2 + Time-spent-communicating-with-Tier-2 + so on.
|5||Call back to same tier (ms)||This label and corresponding value are not always present. When a tier makes an exit call and the call is received back by the same tier then this is displayed. The metric value corresponds to the time spent in the call from the moment the call went out of the tier until the point the call returned back to the caller. |
If the label contains "async" then the exit call was made asynchronously.
Threads and Thread Tasks in the Metric Browser
In a multithreaded transaction, AppDynamics reports key business transaction performance metrics for individual threads in a Thread Tasks branch of the tier in the Metric Browser. The Thread Tasks branch is created only for multithreaded transactions.
The Metric Browser path is Business Transaction Performance -> Business Transactions -> tier-name -> business-transaction-name -> Thread Tasks.
Thread Tasks are also reported in tiers under Overall Application Performance, where you can see metrics on specific calls made by each thread in a node or in a tier.
Threads in Call Graphs
When you drill down in a transaction snapshot for a tier with multiple calls, AppDynamics displays the list of calls that you can drill down into.
Select a call from the list and double-click or click Drill Down into Call to access the call graph.
Diagnostic sessions are automatically triggered based on the average response time of the originating thread of a business transaction. See Capture Details with Diagnostic Sessions.
To configure snapshots based on KPIs of an asynchronous thread, use a custom health rule based on the thread KPI of interest and set up a policy to trigger a diagnostic session on the business transaction. See Capture Details with Diagnostic Sessions.
Drill Down into Downstream Calls on a Thread
If the call graph indicates Async Activity in the Exit Call/Threads column, you can drill down further into the downstream call on the thread:
- Click Async Activity in the Exit Calls/Threads Column for the call that you want to drill down from.
- At the bottom of the Exit Calls and Async Activities window, click Drill Down into Downstream Call.
A call graph for the downstream call opens.
Thread Metrics in Health Rules
You can create a custom health rule based on the performance metrics for a thread task.
When you click the metric icon in the Health Rule Wizard, the embedded metric browser includes the Thread Tasks if the entity for which you are configuring the health rule spawns multiple threads.