PDFs

On this page:

Related pages:

Your Rating:
Results:
PatheticBadOKGoodOutstanding!
59 rates

This topic describes the basic hardware metrics collected by the Machine Agent and the additional metrics collected by the Standalone Machine Agent for Server Visibility. Not all of the metrics that appear in the Hardware Resources branch of the Metric Browser tree are provided by the Machine Agent; some are collected by the Database Agent and are used in the Database Monitoring UI, while others might be custom metrics added by another extension run by your Standalone Machine Agent. 

For most metrics in the Metric Browser, you can click any of the points in the graph to view more information about the metric observed at that point in time. The information includes the metric identifier, date and time of the observation, along with any of the following values relevant to the metric:

  • Obs (observed value): the average of all data points seen for that interval. For a cluster or a time rollup, this represents the weighted average across nodes or over time. 
  • Min: the minimum data point value seen for that interval
  • Max: the maximum data point value seen for that interval
  • Sum: the sum of all data point values seen for that interval. For the Percentile Metric for the App Agent for Java, this is the result of the percentile value multiplied by the Count.
  • Count: The number of data points generated for the metric in that interval. This is information can be useful because the collection interval for infrastructure metrics varies by environment. For example: 
    • For AIX, HP-UX, Mac OS X and Z/OS, CPU and memory metrics are gathered every two seconds and averaged over a period of one minute. Machine agent network and disk metrics are gathered at one-minute intervals.
    • For Windows, Linux, and Solaris, all metrics are collected on one-minute intervals and aggregated over one-minute intervals.

CPU Metrics

Metric NameDescriptionBasic or Server VisibilityWindowsLinux1SolarisAIXDefault Monitoring Mode 4
%IdlePercentage of time the CPU was idle; the CPU had completed its tasks and has nothing to do.Basic(tick)(tick)(tick)(tick)Diagnostic
%Busy

Percentage of time the CPU was busy processing system or user requests; this metric includes CPU Stolen time.

Basic(tick)(tick)(tick)(tick)KPI
%Stolen

Usually stolen time is the percentage of time a virtual CPU waits for a real CPU while the hypervisor is servicing another virtual processor. 

Basic(tick) 2(tick)(tick)(error)KPI
%NicePercentage of CPU time spent on low priority processes.Server Visibility(tick)(tick)(error)
(error)
Advanced
System

Percentage of time the CPU was busy processing kernel code.

Server Visibility(error)
(tick)(tick)(tick)KPI
User

Percentage of time the CPU was busy processing non-kernel code.

Server Visibility(error)
(tick)(tick)(tick)Diagnostic
IOWait

Percentage of CPU time spent waiting for an I/O request.

Server Visibility(error)
(tick)(tick)

(error)

 

KPI
%IrqPercentage of CPU time spent servicing hardware interrupts.Server Visibility(error)
(tick)(error)
(error)
Advanced
%SoftIrqPercentage of CPU time spent servicing software interrupts.Server Visibility(error)
(tick)(error)
(error)
Advanced
%Busy 95th Percentile 3

The CPU %Busy percentage was at this level or lower 95% of the time.

Server Visibility(error)
(tick)(error)
(error)
KPI
IOWait 95th Percentile 3The CPU %IOWait percentage was at this level or lower 95% of the time.Server Visibility(error)
(tick)(error)
(error)
KPI

 See Linux Kernel Processes

 Reported on VMWare only.

 See Percentile Metric Reporting must be Enabled.

 See Default Monitoring Mode

Memory Metrics

 

Metric NameDescriptionBasic or Server VisibilityWindows Linux1  SolarisAIXDefault Monitoring Mode3
Total (MB)

The total amount of memory.

Basic(tick)(tick)(tick)(tick)KPI
Used %The percentage of memory used.Basic(tick)(tick) 4(tick)(tick)KPI
Used (MB)

The amount of memory used.

Basic(tick)(tick) 4(tick)(tick)Advanced
Free %Percentage of free or unused memory available for processes.Basic(tick)(tick) 4(tick)(tick)Advanced
Free (MB)The total amount of free or unused memory available for processes.Basic(tick)(tick) 4(tick)(tick)Advanced
Swap Free (MB)The total amount of free swap space. Reported for each hierarchical group.Server Visibility(tick)  (tick)(tick)(tick)Advanced
Swap Total (MB)The total amount of allocated swap space. Reported for each hierarchical group.Server Visibility(tick)  (tick)(tick)(error)
KPI
Swap Used (MB)The amount of swap space used. Reported for each hierarchical group.Server Visibility(tick)  (tick)(tick)(error)
Advanced
Swap Used %

The percentage of available swap space used. Reported for each hierarchical group.

Server Visibility(tick)  (tick)(tick)(tick)KPI
Swap Free %Percentage of free swap space. Reported for each hierarchical group.Server Visibility(tick)  (tick)(tick)(tick)Advanced
Pages Paged In 95th Percentile 2The paging-in rate to memory was at this level or lower 95% of the time.Server Visibility(error)
(tick)(error)
(error)
KPI
Pages Paged Out 95th Percentile 2The paging-out rate from memory was at this level or lower 95% of the time.Server Visibility(error)
(tick)(error)
(error)
KPI
Pages Swapped In 95th Percentile 2The swapping rate of pages from disks was at this level or lower 95% of the time.Server Visibility(error)
(tick)(error)
(error)
KPI
Pages Swapped Out 95th Percentile 2The swapping rate of pages to disks was at this level or lower 95% of the time.Server Visibility(error)
(tick)(error)
(error)
KPI

See Linux Kernel Processes.

See Percentile Metric Reporting must be Enabled.

 See Default Monitoring Mode

 See Configure Free/Used Memory Metric Calculation on Linux

 

Disk and Partition Metrics

The agent reports metrics for each disk partition and for disks in aggregate. Only mounted partitions are reported. Only local partitions are reported.

Metric NameDescriptionBasic or Server VisibilityWindows LinuxSolarisAIXDefault Monitoring Mode1
KB read/secThe number of KB per second read from all disks and partitions.Basic(tick)(tick)(tick)

(tick)

Diagnostic
KB written/secThe average amount of data per second written to all disks and partitions.Basic(tick)(tick)(tick)(tick)Diagnostic
Reads/secNumber of read operations per second performed on all disks and partitions.Basic

(tick)

(tick)(tick)(error)
KPI
Writes/secNumber of write operations per second performed on all disk and partitions.Basic(tick)(tick)(tick)(tick)5KPI
Avg IO Utilization (%)The average time spent processing read/write requests on all disks and partitions as a percentage of the total reported time window. Databases often report high disk I/O utilization due to frequent read/write requests. For example, if the agent detects read/write processing in 55 out of 60 seconds, the Avg IO Utilization for that minute is 92%. This metric does not measure the amount of available disk space or read/write request sizes.Server Visibility(tick)

(tick) 

(tick)(tick)KPI
<partition> %CPU TimeThe percentage of CPU processing consumed by a partition during read and write operations. Basic(error)
(tick)(error)
(tick)Diagnostic
<partition> Avg Service Time (ms)Time in milliseconds spent performing read and write operations across one partition. Basic(error)
(tick)(tick)(tick)Diagnostic
<partition> Avg Read Time (ms)Time in milliseconds spent performing read operations across one partition. Basic(error)
(tick)(tick)(tick)KPI
<partition> Avg Write Time (ms)Time in milliseconds spent performing write operations across one partition. Basic(error)
(tick)(tick)(tick)KPI
<partition> Avg Queue Time (ms)Time in milliseconds that a read or write request is in the queue before it gets processed across one partition.Basic(error)
(tick)(error)
(tick)KPI
<partition> KB read/secThe average amount of data per second read from one specific partition.Basic(tick) (tick)(tick)(tick)Diagnostic
<partition> KB written/secThe average amount of data per second written to one specific partition. Basic(tick) (tick)(tick)(tick)Diagnostic

<partition>

Reads/sec

Number of read operations per second performed on one specific partition.Basic(tick)(tick)(tick)(error)
KPI

<partition>
Writes/sec 

Number of write operations per second performed on one specific partition.Basic(tick)(tick)(tick)(tick)5KPI

<partition>
Space Available

The amount of unused or free disk space on a specific partition in KB.

Basic(tick)  

(tick) 4

(tick)(tick)KPI

<partition>
Space Used

The amount of used or unavailable disk space on a specific partition in KB.
Basic

(tick) 

 

(tick) 4 (tick)(tick)Advanced
 

<partition>

Avg IO Utilization (%)
The average time spent processing read/write requests as a percentage of the total reported time window. Databases often report high disk I/O utilization due to frequent read/write requests. For example, if the agent detects read/write processing in 55 out of 60 seconds, the Avg IO Utilization for that minute is 92%. This metric does not measure the amount of available disk space or read/write request sizes.Server Visibility(tick)

(tick)  

(tick)(error)
KPI
<partition>
Avg read time (ms)
The average time required to service a read request by all disks or by one specific disk or partition.Server Visibility(tick) 2(tick)(tick)(error)
KPI
<partition>
Avg write time (ms)
The average time required to service a write request.Server Visibility (tick) 2(tick)(tick)(error)
KPI
<partition>
Queue Time 95th Percentile 3
The queue time for read/write requests was this long or shorter 95% of the time for the reported time window.Server Visibility(error)
(tick)(error)
(error)
KPI

 See Default Monitoring Mode

For some versions of Windows Server 2008 and Windows Vista, the Hardware Resources|Disks|<partition>|Avg Read Time (ms) and Hardware Resources|Disks|<partition>|Avg Write Time (ms) metrics will be reported as 0. This is due to a known bug with Windows. To resolve this issue, download the hotfix: https://support.microsoft.com/en-us/kb/961435.

See Percentile Metric Reporting must be Enabled.

4 See Configure Disk Usage Metric Collection on Linux.

 The Standalone Machine Agent calculates the Disk metric Writes/sec on AIX machines differently, depending on whether the HardwareMonitor or the JavaHardwareMonitor extension is used. These extensions assume different block sizes when calculating these metrics. For HardwareMonitor, the block size is 4096 bytes. For JavaHardwareMonitor (based on SIGAR), the block size is 512 bytes. Because the number of writes/sec is calculated as <kb_written>/<block_size>, the HardwareMonitor results are lower than those calculated by JavaHardwareMonitor.  

Volume Metrics

AppDynamics Server Visibility gets the volume space metrics on POSIX systems using the df command. The volume metrics are reported across all listed volumes and for each volume at a specified mount point, such as "/boot". Only local volumes are reported.

Metric NameDescriptionBasic or Server VisibilityWindows Linux1Solaris2AIXDefault Monitoring Mode3
Total (MB)The amount of storage space available (used and free) across all listed volumes or at the specified mount point. On Linux, the space reserved for root is not counted in the available space.Server Visibility(tick)(tick)(tick)(tick)KPI
Free (MB)The amount of unused or free space across all listed volumes or on the selected volume at the specified mount point. On Linux, the space reserved for root is not counted in the available space.Server Visibility(tick)(tick)(tick)(tick)
  • Advanced (volume)
  • KPI (mount point)
Used (MB)The amount of storage space in use across all listed volumes or on the selected volume at the specified mount point.Server Visibility(tick)(tick)(tick)(tick)
  • KPI
    (volume)
  • Advance
    (mount point)
Used (%)

The percentage of storage space in use across all listed volumes or on the selected volume at the specified mount point.

Server Visibility(tick)(tick)(tick)(tick)KPI

 Only /dev volumes are monitored on Linux.

 Only /dev/dsk and /rpool volumes are monitored on Solaris.

 See Default Monitoring Mode


Load Metrics

The load metrics are reported for each machine. The CPU % (reported as part of the basic Machine Agent metrics) is the percentage of the CPU consumed by processes that are currently running. Load takes into account processes that are waiting to run. These metrics are shown as percentages in the Server Dashboard and are scaled by 100 in the Metric Browser.

Metric NameDescriptionBasic or Server VisibilityWindows LinuxSolarisAIXDefault Monitoring Mode1
Last 1 minute

CPU Load, presented as an average over the last 1 minute.

Server Visibility(tick)(tick)(tick)(error)
KPI
Last 5 minutes

CPU Load, presented as an average over the last 5 minutes.

Server Visibility(tick)(tick)(tick)(error)
KPI
Last 15 minutes

CPU Load, presented as an average over the last 15 minutes.

Server Visibility(tick)(tick)(tick)(error)
KPI

 See Default Monitoring Mode


Machine Metrics

Metric NameDescriptionBasic or Server VisibilityWindows LinuxSolarisAIXDefault Monitoring Mode1
Availability

The percentage of time the Machine Agent was reporting to the Controller. In the Server Visibility UI, this provides a percentage with 6 digits of precision (i.e. 100.0000), in other words measuring up to six digits of availability. The Standalone Machine Agent sends a heartbeat to the AppDynamics Controller once per minute to indicate the agent is reporting.

Reported for each machine.

Server Visibility(tick)(tick)(tick)(error)
KPI

 See Default Monitoring Mode

Network Metrics

The Standalone Machine Agent is configured to ignore virtual networks by default. For information about configuring the agent, see Machine Agent Settings for Server Visibility.

Metric NameDescriptionBasic or Server VisibilityWindows LinuxSolarisAIXDefault Monitoring Mode1
Incoming KB 2 The volume of data received by all monitored network devices.Basic(tick)(tick)(tick)(tick)Advanced
Incoming KB/secThe amount of data per second received by all monitored network devices.Basic(tick)(tick)(tick)(tick)KPI
Incoming packets 2 The number of packets received by all monitored network devices.Basic(tick)(tick)(tick)(tick)Advanced
Incoming packets/secThe number of data packets per second received by all monitored network devices.Basic(tick)(tick)(tick)(tick)Advanced
Outgoing KB 2 The volume of data sent by all monitored network devices.Basic(tick)(tick)(tick)(tick)Advanced
Outgoing KB/secThe volume of data sent per second by all monitored network devices.Basic(tick)(tick)(tick)(tick)KPI
Outgoing packets 2 The number of packets sent by all monitored network devices.Basic(tick)(tick)(tick)(tick)Advanced
Outgoing packets/secThe number of data packets sent per second by all monitored network devices.Basic(tick)(tick)(tick)(tick)Advanced
Incoming Errors/minThe number of incoming packet errors the network incurs every minute.Server Visibility(tick)(tick)
(tick)(tick)KPI
Outgoing Errors/minThe number of outgoing packet errors the network incurs every minute. Server Visibility(tick)(tick)
(tick)(error)
KPI
Avg Utilization (%)

The average network utilization as a percentage of the maximum possible throughput. This metric is not reported if the agent cannot determine the throughput (not supported for some devices and Linux versions). The percentage is rounded to the nearest integer. Therefore, very low utilizations might be reported as 0%. Reported for:

  • Servers
  • <group
  • <network interface>
Server Visibility(tick)(tick)(tick)(tick)KPI
<network interface> Incoming KBThe volume of data received by the selected network interface.Basic(tick)(tick)(tick)(tick)Advanced
<network interface> Incoming KB/secThe volume of data received per second by the selected network interface.Basic(tick)(tick)(tick)(tick)KPI
<network interface> Incoming packetsThe number of data packets received by the selected network interface.Basic(tick)(tick)(tick)(tick)Advanced
<network interface> Incoming packets/secThe number of data packets received per second by the selected network interface.Basic(tick)(tick)(tick)(tick)Advanced
<network interface> Outgoing KBThe volume of data sent by the selected network interface.Basic(tick)(tick)(tick)(tick)Advanced
<network interface> Outgoing KB/secThe volume of data sent per second by all monitored network interface.Basic(tick)(tick)(tick)(tick)KPI
<network interface> Outgoing packetsThe number of data packets sent per second by the selected network interface.Basic(tick)(tick)(tick)(tick)Advanced
<network interface> Outgoing packets/secThe number of data packets sent per second by the selected network interface.Basic(tick)(tick)(tick)(tick)Advanced

<network interface>
Incoming Errors/min

The number of incoming packet errors the network incurs every minute.

Server Visibility(tick)(tick)
(tick)(error)
KPI

<network interface>
Outgoing Errors/min

The number of outgoing packet errors the network incurs every minute.

Server Visibility(tick)(tick)
(tick)(error)
KPI

 See Default Monitoring Mode

2  The ServerMonitoring and JavaHardwareMonitor extensions calculate unidirectional throughput (packet/KB) metrics differently. ServerMonitoring reports the total number of packets/bytes in the last minute. JavaHardwareMonitor reports the total number of packets/bytes from the time the Standalone Machine Agent started.

Process Metrics

The following metrics are aggregated and reported for each process or process class (except Total Process Count, which measures all processes observed by the agent). To modify the default process monitoring, see Machine Agent Settings for Server Visibility.

Metric NameDescriptionBasic or Server VisibilityWindows Linux1  Solaris2AIXDefault Monitoring Mode3
CountThe number of processes in this class consuming CPU or memory resources or the total number of processes in this class.Server Visibility(tick)(tick)(tick)(tick)Diagnostic
CPU Used (%)Percentage of the CPU bandwidth used by all processes in a process class. A process using 100% CPU is executing on all processors on the system. If there are four cores on the machine and the process is executing four threads, each executing on one core, then the process can use up to 400% CPU. If there are four cores on the machine and the process is executing one thread on one core, then the process can use up to 25% CPU.Server Visibility(tick)(tick)(tick)(error)
KPI
Major Faults/sec

Number of major page faults caused by all processes in this class.

Server Visibility(tick)(tick)(error)
(error)
Diagnostic
Minor Faults/secNumber of minor page faults caused by all processes in this class.Server Visibility(tick) (tick)(error)
(error)
Diagnostic
Memory Used (%)Percentage of memory consumed by the top 10 consuming processes or the percentage of memory used by all processes in this class.Server Visibility(tick)(tick)(tick)(error)
KPI
Memory Used (KB)Amount of memory used by all processes in this class.Server Visibility(tick)(tick)(tick)(error)
Advanced
Memory Virtual (KB)

Current size of the virtual address space that the process is using. Use of virtual address space does not necessarily imply a corresponding use of either disk or main memory pages. Virtual space is finite and, by using too much, the process can limit its ability to load libraries.


Server Visibility(tick)(tick)(tick)(error)
Advanced
Thread Count

The number of kernel threads used by all processes in this class.

Server Visibility(tick)(tick)(tick)(error)
Diagnostic
Total Process CountAll individual processes observed by the Standalone Machine Agent, before it filters out unreported processes and groups the remaining processes into classes. This metric is useful for identifying anomalies in the number of processes running on a specific machine. For example: If the average Total Process Count on a machine is usually 200 to 300, and then suddenly rises to 2000 or more, this could indicate a problem on that machine. 
To collect this metric, the Machine Agent captures the number of processes every 30 seconds and calculates the average number of processes per minute. This means that 
  • Observed is the average number of processes per minute.
  • Min and Max are the 30-second buckets with the least and most processes.
  • Count and Sum are the number of process captures each minute and the sum of the processes in each capture for that minute (capture1 + capture2).
Server Visibility(tick)(tick)(tick)(error)
Diagnostic

1  See Linux Kernel Processes.

2  When collecting Process metrics on Solaris, the Standalone Machine Agent observes and captures only the first 80 characters of each process name and argument list. This means that the agent considers only the first 80 characters of each process string when it applies whitelists ("always monitor this process") and blacklists ("never monitor this process") defined in ServerMonitoring.yml.

 See Default Monitoring Mode

Service Availability Metrics

Reported for each service that is configured.

Metric NameDescriptionBasic or Server VisibilityWindows LinuxSolarisAIXDefault Monitoring Mode1
Response Time (ms)The elapsed time between sending a request and receiving a response from the monitored service.Server Visibility(tick)(tick)(tick)(tick)KPI
Response Size (bytes)

The size of the response received from the monitored service.


Server Visibility(tick)(tick)(tick)(tick)
KPI
Success Rate (%)The percentage of successful requests over all requests made to the service.Server Visibility(tick)(tick)(tick)(tick)KPI

 See Default Monitoring Mode

Enable Percentile Metric Reporting

Percentile metric reporting must be enabled on both the Controller and the Machine Agent. By default, reporting is disabled on the Controller and enabled on the agent.

Configure Disk Usage Metric Collection on Linux

You can configure the Linux Standalone Machine Agent to calculate the following metrics in a manner more consistent with the Linux df command:

  • Servers > Volumes > /opt > Disk Usage
  • Metric Browser > Disks and Partitions > <partition> > Space Available
  • Metric Browser > Disks and Partitions > <partition> > Space Used

Linux includes a mechanism for reserving some disk space to ensure that the system keeps functioning even if non-privileged users consume all other disk space. By default, these metrics do not include this reserved space. This means that the total disk space reported by the agent might differ from the total disk space reported by the Linux df command or other sources.
To override this default, and to ensure that these metrics include this reserved space, run the agent with the following command-line argument:
      -Dappdynamics.machine.agent.extensions.calcVolumeFreeAndUsedWithDfCommand=true
This feature is supported for Linux versions of the Standalone Machine Agent only.

Configure Free/Used Memory Metric Collection on Linux

You can configure how the Standalone Machine Agent calculates the amount of free and used memory on Linux machines. By default, the agent calculates any slab-reclaimable memory as used (not free) memory. 
To configure the agent to calculate slab-reclaimable memory as free (not used) memory, run the Standalone Machine Agent from the command line with the following argument: 
        -Dappdynamics.machine.agent.extensions.countSlabReclaimableAsFreeMem=true
This setting impacts the following metrics: 

  • Memory Usage (in Server Dashboard)
  • Memory Used %, Memory Used MB, Memory Free %, and Memory Free MB (in Metric Browser)

Viewing Server Visibility Metrics

Server Visibility metrics can only be viewed in the Server Visibility version of the Metric Browser. You see this when you access the Metric Browser from the Servers tab in the top navigation bar of the Controller UI.

Standalone Machine Agent vs. .NET Machine Agent

If a server has both Machine and .NET Agents installed, there might be differences in metric values reported by the Machine Agent and the .NET Agent due to different averaging rates and measurement methods. For more information, see .NET Compatibility Mode

Streamlined Browsing Mode

The Server Metric Browser includes a Streamlined Browsing option for node metrics. This option is useful when you are browsing metrics for tiers that contain multiple nodes. 

  • With Streamlined Browsing enabled, each node in the browser treeview shows only metrics reported by the Standalone Machine Agent on the corresponding node. 
  • With Streamlined Browsing disabled, each node in the browser treeview also shows metrics for other nodes in the same tier (with no indication of the node that corresponds to which tier group).

This streamlining option is disabled by default. To enable this option, go to http://<controller host>:<port>/controller/admin.jsp and set sim.metrics.metricBrowser.machineMetricMappings.enabled to true.  

 

Streamlined Browsing Enabled
Streamlined Browsing Disabled
      

 

Linux Kernel Processes

The Standalone Machine Agent has a processSelectorRegex setting that specifies the set of processes monitored by the agent. The default regex filters out most kernel processes on Linux machines. The agent considers monitored processes only when it calculates CPU, Memory, and Process metrics. Therefore, these metrics might be different from metrics reported by other sources such as Linux commands.

Default Monitoring Mode

The Default Monitoring Mode column indicates the default category of each metric when Dynamic Monitoring Mode (DMM) is enabled. When this mode is enabled, a Standalone Machine Agent reports metrics based on the DMM setting on that server:

  • KPI – Report Key Performance Indicator metrics only
  • Diagnostic – Report KPI and Diagnostic metrics
  • Advanced – Report all unfiltered metrics on the Machine Agent.

Load Average Percentages are Scaled by 100 in the Metric Browser

Load Average metrics are shown as float values (such as 0.70 or 1.05) in the Server Dashboard. In the Metric Browser, these metrics are multiplied by 100 to provide two decimal points of precision. If a server has an average load of 7.67, for example, the Server Dashboard shows the Load Average as 7.67 and the Metric Browser shows the Load as 767.  If you want to use one of the metrics in a health rule or custom dashboard, divide the metric by 100. 

  • No labels