Monitoring Containers with Docker Visibility

You can use the Machine Agent to monitor application nodes running inside Docker containers and to identify container issues that impact application performance. By viewing and comparing APM metrics with the underlying container and server/machine metrics, you can easily answer the question: Is my application problem purely an application problem, or is the root cause in the container or the server? 

Container monitoring requires a Server Visibility license and version 4.3.3 or higher of both the Controller and the Machine Agent.

The Machine Agent should be deployed inside a Docker container. The Machine Agent collects metrics for Docker containers on the same host, in addition to server and machine metrics for the host itself. The Controller shows all monitored containers, for each host, and the container and host IDs for each container. 

In a BRIDGE networking mode, the containers will take on the container ID as the host name. If networking is in host mode, then the containers take on the node name of the host ID. This means every container on that node will have the same host ID. In this case you need to use the unique host ID settings. See Register the Container ID as the Host ID.

By default, the Machine Agent only monitors containers that have a running APM Agent, but this can be changed by setting the sim.docker.monitorAPMContainersOnly property on the Controller. See Controller Settings for Server Visibility.

To deploy a Machine Agent on a host outside a Docker container, create a symbolic link: ln -s / /hostroot on the host. This symbolic link enables the Machine Agent to collect host metrics along with Docker container metrics. When a Machine Agent is deployed inside a Docker container for monitoring, the symbolic link is automatically created when the volume mounts. If you want to grant more restrictive permissions you can use the following command to create symbolic links: ln -s /proc /hostroot/proc; ln -s /sys /hostroot/sys; ln -s /etc /hostroot/etc. These links can be made read-only, the AppDynamics agent does not need write privileges to these directories.

The following diagram illustrates how to deploy container monitoring: 

• Install the Machine Agent () in a standalone container. The Machine Agent collects hardware metrics for each monitored container, as well as Machine and Server metrics for the host (), and forwards the metrics to the Controller.
• The Machine Agent is able to monitor all containers that are running on that host, subject to the limits outlined below, and will report runtime metrics and metadata for every container. In addition, if any of the containers have an APM Agent installed (), then the Machine Agent also correlates the container metadata and runtime metrics with the associated APM Node.

Before You Start

Note the following:

1. AppDynamics recommends that you use Docker CE/EE v17.03 or Docker Engine v1.13 with this product. Some data might be missing if you are using previous versions of Docker.
2. Container Monitoring is not supported on Docker for Windows or Docker for Mac. 
3. AppDynamics strongly recommends that you follow these guidelines, Instrument JVMs in a Dynamic Environment when monitoring container-based applications.
4. The Machine Agent can monitor up to 120 running containers per host. The default maximum number of containers per host is 100. With the cgroup enabled flag set to true, the Machine Agent can monitor up to 600 containers per host. See Configuring Docker Visibility. 
5. The Machine Agent collects metrics from containers with one or more running processes whose command line matches a configurable regex. By default, this regex matches all processes (*). You can override this behavior by editing the following regex in <machine_agent_home>/extensions/DockerMonitoring/conf/DockerMonitoring.yml:
   containerMonitoringConfig:containerProcessSelectorRegex: ".*"
6. The maximum number of containers you can monitor in one Controller depends on the Controller size, the total number of App Agents, and the current load. If you are using a SaaS Controller, please work with your Account Manager.
7. Please review the following:
   1. Best Practices for setting up App Agents and Standalone Machine Agents 
   2. Docker Visibility Known Issues in the latest Release Notes

Enabling Container Monitoring

To enable Container Monitoring:

1. On the Controller, log in to the Administration Console and verify that sim.docker.enabled is set to true. 
2. On the agent, make sure that Server Visibility Enabled and Enable Docker Visibility are both set to true. 

Container Monitoring Setup

The quickest and easiest way to run the Machine Agent with Container Monitoring enabled is to use one of the official images from the Docker Store: https://store.docker.com/images/appdynamics. These images are produced by AppDynamics, based on certified base images from the Docker Community, and these can either be run directly or used as base images for your own application containers. For full details of how to download and run containers based on these official images, please see the documentation posted on the Docker Store. If you prefer to build your base images, the full source code for building these images is posted to GitHub. You can use this as a pattern for your own builds: https://github.com/Appdynamics/appdynamics-docker-images.  

You should set the following for the Machine Agent to monitor containers running on the server. See Deploy Using Containers.

• Server Visibility Enabled – Enable Server Visibility
• Docker Enabled – Enable Docker Visibility
• Volume Mounts – Specify the following:
  • Volume mounts to allow read-only access to the underlying file system (/proc, /etc and /sys). This allows the Server Agent to collect host-level metrics for containers running on the server.
  • The UNIX domain socket on which the Docker daemon is configured to listen for API calls. 

Viewing Container Details

To view container metadata and metrics in the Controller:

• In the Applications Dashboard, go to Containers to see all monitored containers used by the application.

• In the Servers Dashboard, go to Containers tab to see all monitored containers on that host.

• To open the Container Dashboard, right-click on the container name, and choose View Details.

The Container Details view has the following tabs, which provide an overview of the health and resource usage for the container:

1. 1. Overview - Container metadata, Tags (name-value pairs derived from Docker/Kubernetes) plus AWS tags where applicable, and single chart views for CPU, memory, network and disk usage.
   2. CPU - CPU Usage and Throttled Time metrics
   3. Memory - Memory Usage and Memory Fault metrics
   4. Network - Network Usage metrics (only available when running in Diagnostic mode). For more information on Diagnostic mode, see "Enable Dynamic Monitoring Mode (DMM)" under Machine Agent Configuration Properties.

• The Node Dashboard also includes a Containers tab for the container in which that node is running.

Viewing Container Metrics using the Metric Browser

To view time-series metric data for containers, double-click on one of container metric graphs (CPU, Memory, Network, Disk).  This will open the Metric Browser with the displayed metric selected.  The Metric Browser tree displays the full set of metrics available for that container, and you can add these to the Metric Browser display by double-clicking on the metric you wish to select. See Container Metrics.