You can use the Machine Agent to monitor application nodes running inside Docker containers and to identify container issues that impact application performance. By viewing and comparing APM metrics with the underlying container and server/machine metrics, you can easily answer the question: Is my application problem purely an application problem, or is the root cause in the container or the server?
Container monitoring requires a Server Visibility license and version 4.3.3 or higher of both the Controller and the Machine Agent.
The Machine Agent should be deployed inside a Docker container. The Machine Agent collects metrics for Docker containers on the same host, in addition to server and machine metrics for the host itself. The Controller shows all monitored containers, for each host, and the container and host IDs for each container.
In a BRIDGE networking mode, the containers will take on the container ID as the host name. If networking is in host mode, then the containers take on the node name of the host ID. This means every container on that node will have the same host ID. In this case you need to use the unique host ID settings. See Register the Container ID as the Host ID.
By default, the Machine Agent only monitors containers that have a running APM Agent, but this can be changed by setting the
sim.docker.monitorAPMContainersOnly property on the Controller. See Controller Settings for Server Visibility.
To deploy a Machine Agent on a host outside a Docker container, create a symbolic link:
ln -s / /hostroot on the host. This symbolic link enables the Machine Agent to collect host metrics along with Docker container metrics. When a Machine Agent is deployed inside a Docker container for monitoring, the symbolic link is automatically created when the volume mounts. If you want to grant more restrictive permissions you can use the following command to create symbolic links:
ln -s /proc /hostroot/proc;
ln -s /sys /hostroot/sys;
ln -s /etc /hostroot/etc. These links can be made read-only, the AppDynamics agent does not need write privileges to these directories.
The following diagram illustrates how to deploy container monitoring:
- Install the Machine Agent ( ) in a standalone container. The Machine Agent collects hardware metrics for each monitored container, as well as Machine and Server metrics for the host ( ), and forwards the metrics to the Controller.
- The Machine Agent is able to monitor all containers that are running on that host, subject to the limits outlined below, and will report runtime metrics and metadata for every container. In addition, if any of the containers have an APM Agent installed ( ), then the Machine Agent also correlates the container metadata and runtime metrics with the associated APM Node.
Before You Start
Note the following:
- AppDynamics recommends that you use Docker CE/EE v17.03 or Docker Engine v1.13 with this product. Some data might be missing if you are using previous versions of Docker.
- Container Monitoring is not supported on Docker for Windows or Docker for Mac.
- AppDynamics strongly recommends that you follow these guidelines, Instrument JVMs in a Dynamic Environment when monitoring container-based applications.
- The Machine Agent can monitor up to 120 running containers per host. The default maximum number of containers per host is 100. With the cgroup enabled flag set to true, the Machine Agent can monitor up to 600 containers per host. See Configuring Docker Visibility.
- The Machine Agent collects metrics from containers with one or more running processes whose command line matches a configurable regex. By default, this regex matches all processes (*). You can override this behavior by editing the following regex in
- The maximum number of containers you can monitor in one Controller depends on the Controller size, the total number of App Agents, and the current load. If you are using a SaaS Controller, please work with your Account Manager.
- Please review the following:
- Best Practices for setting up App Agents and Standalone Machine Agents
- Docker Visibility issues in the latest Release Notes
Enabling Container Monitoring
To enable Container Monitoring:
- On the Controller, log in to the Administration Console and verify that sim.docker.enabled is set to true.
- On the agent, make sure that Server Visibility Enabled and Enable Docker Visibility are both set to true.
Container Monitoring Setup
The quickest and easiest way to run the Machine Agent with Container Monitoring enabled is to use one of the official images from the Docker Store: https://store.docker.com/images/appdynamics. These images are produced by AppDynamics, based on certified base images from the Docker Community, and these can either be run directly or used as base images for your own application containers. For full details of how to download and run containers based on these official images, please see the documentation posted on the Docker Store. If you prefer to build your base images, the full source code for building these images is posted to GitHub. You can use this as a pattern for your own builds: https://github.com/Appdynamics/appdynamics-docker-images.
You should set the following for the Machine Agent to monitor containers running on the server. See Deploy Using Containers.
- Server Visibility Enabled – Enable Server Visibility
- Docker Enabled – Enable Docker Visibility
- Volume Mounts – Specify the following:
- Volume mounts to allow read-only access to the underlying file system (
/sys). This allows the Server Agent to collect host-level metrics for containers running on the server.
- The UNIX domain socket on which the Docker daemon is configured to listen for API calls.
- Volume mounts to allow read-only access to the underlying file system (
Viewing Container Details
To view container metadata and metrics in the Controller:
- In the Applications Dashboard, go to Containers to see all monitored containers used by the application.
- In the Servers Dashboard, go to Containers tab to see all monitored containers on that host.
- To open the Container Dashboard, right-click on the container name, and choose View Details.
The Container Details view has the following tabs, which provide an overview of the health and resource usage for the container:
- Overview - Container metadata, Tags (name-value pairs derived from Docker/Kubernetes) plus AWS tags where applicable, and single chart views for CPU, memory, network and disk usage.
- CPU - CPU Usage and Throttled Time metrics
- Memory - Memory Usage and Memory Fault metrics
- Network - Network Usage metrics (only available when running in Diagnostic mode). For more information on Diagnostic mode, see "Enable Dynamic Monitoring Mode (DMM)" under Standalone Machine Agent Configuration Properties.
- The Node Dashboard also includes a Containers tab for the container in which that node is running.
Viewing Container Metrics using the Metric Browser
To view time-series metric data for containers, double-click on one of container metric graphs (CPU, Memory, Network, Disk). This will open the Metric Browser with the displayed metric selected. The Metric Browser tree displays the full set of metrics available for that container, and you can add these to the Metric Browser display by double-clicking on the metric you wish to select. See Container Metrics.
Troubleshooting Containers using Dynamic Monitoring Mode
You can selectively control the number of Container metrics reported for individual containers, using Dynamic Monitoring Mode (DMM). Rather than have all agents report all metrics for all containers all the time, you can configure individual agents to collect
- Key Performance Indicator metrics only (KPI mode)
- KPI and Diagnostic metrics (Diagnostic mode)
- All available metrics (Advanced Diagnostic mode)
This provides the flexibility to report KPI metrics only on most containers and then increase the metric level on specific containers where you need deeper visibility to diagnose problems. You can increase scalability on the Controller and conserve metric bandwidth on the network with no sacrifice in visibility. Every Docker Visibility metric has a default DMM class (KPI, Diagnostic, and Advanced Diagnostic). Consult the Container Metrics table for details: the Default Monitoring Mode column indicates the default category of each metric when Dynamic Monitoring Mode is enabled.
When you notice a container that requires additional analysis or troubleshooting, change the DMM on that container to collect more metrics:
- Open the Container Detail window for the container of interest: Open the Containers table (Containers, left menu) and double-click on the container.
- Click the Edit (pen) button in the top-right corner, change the mode to Diagnostic or Advanced Diagnostic, and click the checkmark to apply the changes.
- Collect additional metrics for your analysis or troubleshooting.
- When you no longer need to collect additional metrics for that container, set the DMM on that container back to KPI.
Note the following:
- The Controller has three different DMM defaults, depending on the type of metrics collected by the agent. These defaults are all set to KPI (the lowest setting).
- The default for both settings is KPI (the lowest setting).
- AppDynamics recommends that you leave these global settings at their defaults. If you need to collect advanced metrics for a specific server or container, increase the DMM on an "as-needed" basis as described in the following section.
- When a container monitored by a Java App Agent shuts down and restarts, any overridden DMM specified for the shut-down container is lost. The DMM for the restarted container automatically resets to the global default specified by
- If you switch the monitoring mode in the Controller from a more-inclusive to a less-inclusive mode, the Metric Browser will show values for the newly-excluded metrics for one hour after the switch. For example: suppose you switch from Diagnostic to KPI mode. For any Diagnostic metric, the Metric Browser will report a steady line (at 0 or the last-reported value) for one hour after the switch; then the line will disappear. This is standard behavior in the Metric Browser for an agent when it stops reporting a specific metric.