Use the Machine Agent to monitor application nodes running inside Docker containers and to identify container issues that impact application performance. By viewing and comparing APM metrics with the underlying container and server/machine metrics, you can easily answer the question: Is my application problem purely an application problem, or is the root cause in the container or the server? 

Container monitoring requires a Server Visibility license >=4.3.3, for both the Controller and the Machine Agent.

You should deploy the Machine Agent inside a Docker container. The Machine Agent collects metrics for Docker containers on the same host, and server and machine metrics for the host itself. The Controller shows all monitored containers, for each host, and the container and host IDs, for each container. 

In a BRIDGE networking mode, the containers take on the container ID as the host name. If networking is in host mode, then the containers take on the node name of the host ID. This means every container on that node has the same host ID. In this case you need to use the unique host ID settings. See Register the Container ID as the Host ID. When using Docker Visibility, if the unique host ID setting is not configured to use container ID in host network mode, the Machine Agent automatically registers the container using the container ID as the host ID. If you have an older version of the Controller or Machine Agent, AppDynamics recommends that you upgrade to Machine Agent version 20.7 or later. See Using Docker Visibility with Kubernetes.

With Controller >=20.11.0:

  • If the Machine Agent >=20.7.0, the Machine Agent automatically registers the container using the container ID as the host ID. No further action is needed.
  • If the Machine Agent <=20.6.0 is configured incorrectly, the Controller rejects the misconfigured containers registration.

By default, the Machine Agent only monitors containers that have a running APM Agent. You can change this by setting the sim.docker.monitorAPMContainersOnly property on the Controller. See Controller Settings for Server Visibility.

To deploy a Machine Agent on a host outside a Docker container, create a symbolic link: ln -s / /hostroot on the host. This symbolic link enables the Machine Agent to collect host metrics with Docker container metrics. When you deploy a Machine Agent inside a Docker container for monitoring, the symbolic link is automatically created when the volume mounts. To grant more restrictive permissions, enter this command to create symbolic links: ln -s /proc /hostroot/proc; ln -s /sys /hostroot/sys; ln -s /etc /hostroot/etc. You can make these links read-only because the AppDynamics Agent does not need write privileges to these directories.

This diagram illustrates how to deploy container monitoring: 

container monitoring

  • Install the Machine Agent () in a standalone container. The Machine Agent collects hardware metrics for each monitored container, as well as Machine and Server metrics for the host (), and forwards the metrics to the Controller.
  • The Machine Agent can monitor all containers that are running on that host, subject to established limits, and will report runtime metrics and metadata for every container. Additionally, if any of the containers have an APM Agent installed (), then the Machine Agent also correlates the container metadata and runtime metrics with the associated APM Node.

Before You Start

  1. AppDynamics recommends that you use Docker CE/EE 17.03 or Docker Engine 1.13 with this product. Some data might be missing if you use previous versions of Docker.
  2. Container Monitoring is not supported on Docker for Windows, or Docker for Mac. 
  3. AppDynamics strongly recommends that you follow these guidelines, Instrument JVMs in a Dynamic Environment when monitoring container-based applications.
  4. The Machine Agent can monitor up to 120 running containers per host. The default maximum number of containers per host is 100. With the cgroup enabled flag set to true, the Machine Agent can monitor up to 600 containers per host. See Configure Docker Visibility.
  5. The Machine Agent collects metrics from containers with one or more running processes whose command line matches a configurable regex. By default, this regex matches all processes (*). To override this behavior, edit this regex in <machine_agent_home>/extensions/DockerMonitoring/conf/DockerMonitoring.yml:
    containerMonitoringConfig:containerProcessSelectorRegex: ".*"
  6. The maximum number of containers you can monitor in one Controller depends on the Controller size, the total number of App Agents, and the current load. If you are using a SaaS Controller, please work with your Account Manager.
  7. Please review:
    1. Best Practices for setting up App Agents and Machine Agents.
    2. Docker Visibility issues in the latest Release Notes.

Enable Container Monitoring

To enable Container Monitoring:

  1. On the Controller, log in to the Administration Console and verify that sim.docker.enabled is set to true. 
  2. On the Agent, enable Server Visibility and Docker Visibility.

Container Monitoring Setup

The quickest and easiest way to run the Machine Agent with Container Monitoring enabled is to use one of the official images from the Docker Store: https://store.docker.com/images/appdynamics. These images are produced by AppDynamics, based on certified base images from the Docker Community, and can either be run directly or used as base images for your own application containers. For full details of how to download and run containers based on these official images, see the documentation posted on the Docker Store. To build your own base images, the full source code for building these images is posted to GitHub. You can use this as a pattern for your own builds: https://github.com/Appdynamics/appdynamics-docker-images.  

For the Machine Agent to monitor containers running on the server, configure these settings (See Configure Installation Options):

  • Server Visibility Enabled – Enable Server Visibility
  • Docker Enabled – Enable Docker Visibility
  • Volume Mounts – Specify:
    • Volume mounts to allow read-only access to the underlying file system (/proc, /etc and /sys). This allows the Server Agent to collect host-level metrics for containers running on the server.
    • The UNIX domain socket on which the Docker daemon is configured to listen for API calls. 

View Container Details

To view container metadata and metrics in the Controller:

  • In the Applications Dashboard, go to Containers to see all monitored containers used by the application.

Applications Dashboard

  • In the Servers Dashboard, go to Containers to see all monitored containers on that host.

Servers Dashboard

  • To open the Container Dashboard, right-click on the container name, and choose View Details.

Container Dashboard

The Container Details view contains these tabs, which provide an overview of the health and resource usage for the container:

    • Overview - Container metadata, Tags (name-value pairs derived from Docker/Kubernetes) plus AWS tags where applicable, and single chart views for CPU, memory, network and disk usage
    • CPU - CPU Usage and Throttled Time metrics
    • Memory - Memory Usage and Memory Fault metrics
    • Network - Network Usage metrics (only available when running in Diagnostic mode), see "Enable Dynamic Monitoring Mode (DMM)" under Machine Agent Configuration Properties.
  • The Node Dashboard also includes a Containers tab for the container in which that node is running.

Containers tab

View Container Metrics Using the Metric Browser

To view time-series metric data for containers, double-click one of container metric graphs (CPU, Memory, Network, Disk) to open the Metric Browser with the displayed metric selected. The Metric Browser tree displays the full set of metrics available for that container, and you can add these to the Metric Browser display by double-clicking the metric you wish to select. See Container Metrics.

Container Metrics


Troubleshooting Containers Using Dynamic Monitoring Mode

Using Dynamic Monitoring Mode (DMM), you can selectively control the number of Container metrics reported for individual containers. Rather than have all Agents report all metrics for all containers all the time, you can configure individual Agents to collect:

  • Key Performance Indicator metrics only (KPI mode)
  • KPI and Diagnostic metrics (Diagnostic mode)
  • All available metrics (Advanced Diagnostic mode)

This provides the flexibility to report KPI metrics only on most containers, and then increase the metric level on specific containers when you need deeper visibility to diagnose problems. You can increase scalability on the Controller and conserve metric bandwidth on the network with no sacrifice in visibility. Every Docker Visibility metric has a default DMM class (KPI, Diagnostic, and Advanced Diagnostic). For details, consult the Container Metrics table: the Default Monitoring Mode column indicates the default category of each metric when Dynamic Monitoring Mode is enabled.

When you notice a container that requires additional analysis or troubleshooting, change the DMM on that container to collect more metrics:

  1. Open the Container Detail panel for the container of interest: Open the Containers table (Containers, left menu) and double-click the container.
  2. Click the Edit (pen) button in the top-right corner, change the mode to Diagnostic or Advanced Diagnostic, and click the checkmark to apply the changes. 
  3. Collect additional metrics for your analysis or troubleshooting.
  4. When you no longer need to collect additional metrics for that container, set the DMM on that container back to KPI.

Edit (pen) button

Note:

  • Depending on the type of metrics collected by the Agent, the Controller has three different DMM defaults. These defaults are all set to KPI (the lowest setting): 
  • The default for both settings is KPI (the lowest setting). 
  • AppDynamics recommends that you leave these global settings at their defaults. If you need to collect advanced metrics for a specific server or container, increase the DMM on an "as-needed" basis.
  • When a container monitored by a Java App Agent shuts down and restarts, any overridden DMM specified for the shut-down container is lost. The DMM for the restarted container automatically resets to the global default specified by sim.machines.dmm.container.defaultMode. 
  • If you switch the monitoring mode in the Controller from a more-inclusive to a less-inclusive mode, the Metric Browser will show values for the newly-excluded metrics for one hour after the switch. For example, if you switch from Diagnostic to KPI mode. For any Diagnostic metric, the Metric Browser will report a steady line (at 0, or the last-reported value) for one hour after the switch; then the line will disappear. This is standard behavior in the Metric Browser for an Agent when it stops reporting a specific metric.