Related pages:

You can write a monitoring extension script (also known as a custom monitor or hardware monitor) to add custom metrics to the metric set that AppDynamics already collects and reports to the Controller. Your script reports the custom metrics every minute to the Machine Agent. The Machine Agent passes these metrics to the Controller.

This page describes the steps for adding custom metrics using a shell script and includes an example.

Review Existing Extensions

Before creating your own extension, review the extensions that have been created and shared among members of the AppDynamics community. New extensions are added continuously. It is possible that someone has already created what you need or something close enough that you can download and use it after making a few modifications. 

See the AppDynamics Exchange for free downloads.

Agent Configuration Requirements

Confirm that you have correctly configured the Machine Agent in the controller-info.xml file and on the Agent start command on the command line. See Machine Agent Configuration Properties.

Create a Monitoring Extension

To create a monitoring extension using a script:

  1. Create your script. See Create the Script File.
  2. Create a monitor.xml configuration file. See Create the monitor.xml File.
  3. Create a subdirectory, <your_extension_dir>, in <machine_agent_home>/monitors. See Create a directory under the Machine Agent monitors directory.

  4. Copy your script file and the monitor.xml file into the new subdirectory.

  5. Restart the Machine Agent.

Define Your Metrics

Metric names must be unique within the same metric path but need not be unique for the entire metric hierarchy. AppDynamics recommends using short metric names so that the whole name is visible when displayed in the Metric Browser. Prepend the metric path to the metric name when you upload the metrics to the Controller.

Metric Processing Qualifiers

The Controller has various qualifiers for how it processes a metric regarding aggregation, time rollup, and tier rollup. There are three types of metric qualifiers:

  1. Aggregator qualifier
  2. Time roll-up qualifier
  3. Cluster roll-up qualifier

In the script, specify the metric qualifiers after the name-value pair for the metric. A typical metric entry in the script file has the following structure:

name=<metric name>,value=<long value>,aggregator=<aggregator type>, time-rollup=<time-rollup strategy>, cluster-rollup=<cluster-rollup strategy>

Aggregator Qualifier

The aggregator qualifier specifies how the Machine Agent aggregates the values reported during a one-minute period. Specify the aggregator qualifier as aggregator="aggregator type". This value is an enumerated type. If no value is reported during that minute, no data is reported to the Controller, and an UNCHANGED notice appears in the Machine Agent log for that metric. Valid values are:

Aggregator Type

Description

AVERAGE

(Default) Average of all reported values in that minute

SUM

Sum of all reported values in the minute, causes the metric to behave like a counter

OBSERVATION

Last reported value in the minute

Time Roll-Up Qualifier

The time-rollup qualifier specifies how the Controller rolls up the values when it converts from one-minute granularity tables to 10-minute granularity tables, and 60-minute granularity tables over time. The value is an enumerated type. Valid values are:

Roll Up Strategy

Description

AVERAGE

Average of all one-minute values when adding it to the 10-minute granularity table; the average of all 10-minute values when adding it to the 60-minute granularity table

SUM

Sum of all one-minute values when adding it to the 10-minute granularity table; the sum of all 10-minute values when adding it to the 60-minute granularity table

CURRENT

Last reported one-minute value in that 10-minute interval; the last reported 10-minute value in that 60-minute interval

Cluster Roll-Up Qualifier

The cluster-rollup qualifier specifies how the Controller aggregates metric values in a tier (a cluster of nodes). The value is an enumerated type. Valid values are:

Roll up Strategy

Description

INDIVIDUAL

Aggregates the metric value by averaging the metric values across each node in the tier

COLLECTIVE

Aggregates the metric value by adding up the metric values for all the nodes in the tier

For example, if a tier has two nodes, Node A and Node B, and Node A has three errors per minute and Node B has seven errors per minute, then the INDIVIDUAL qualifier reports a value of five errors per minute, and COLLECTIVE qualifier reports ten errors per minute. INDIVIDUAL is appropriate for metrics such as % CPU Busy, where you want the value for each node. COLLECTIVE is appropriate for metrics, such as Number of Calls, where you want a value for the entire tier.

Add a Monitoring Extension Script

To add a monitoring extension script:

Create a Subdirectory Under the Machine Agent Monitors Directory

The <machine_agent_home>/monitors directory is the repository for the Machine Agent extensions. For each new extension, create a subdirectory under the /monitors directory. The user running the Agent requires read, write, and execute permissions to this subdirectory.

For example to create an extension that monitors open files in the JVM, create a subdirectory named "openfiles" under <machine_agent_home>/monitors. The structure looks like:

Machine Agent Subdirectory

Create the Script File

A script writes data to STDOUT. The Machine Agent parses STDOUT and sends information to the Controller every minute. Use these instructions to create the script file:

For Windows custom metrics, PowerShell and VBScript are recommended over .bat files

To generate custom metrics on Windows, AppDynamics recommends that you use PowerShell and VBasic scripts instead of .bat files. 
When a standard Windows batch (.bat) script echoes metric names, it surrounds the names with quotes. The quotes will cause the Machine Agent to ignore these metrics. PowerShell and VBasic scripts do not have this issue.

  1. Specify a name-value pair for the metrics.
    Each metric has a name-value pair that is converted to a java 'long' value. A typical metric entry in the script file has this structure:

    name=<metric name>,value=<long value>,aggregator=<aggregator type>, time-rollup=<time-rollup strategy>, cluster-rollup=<cluster-rollup strategy>

    Use the following format:

    Form

    Format

    Standard Form

    Hardware Resources| Instrument Name=Instrument Value

    Fully Qualified Form

    Hardware Resources| <metric name>,value=<long value>

  2. Define the category of the metric, for example:
    1. Infrastructure (for the default hardware metrics, see Machine Agent)
    2. JVM
    3. Custom Metrics, where Custom Metrics must have the path prefixes:
      - Custom Metrics
      - Server|Component:<tier-name-or-tier-id>

  3. Metrics with the Custom Metrics prefix are common across all tiers in your application. Metrics with the Server|Component:<tier-name-or-tier-id> prefix appear only under the specified tier. 

    To find the component ID of a tier, open the dashboard for the tier and review the URL. The ID appears as the component value in the URL:

    The Machine Agent has to be associated with the target or destination for the metrics. If you attempt to publish metrics to a tier that is not associated with the Machine Agent, the metrics are not reported.


    The "|" character separates the branches in the metric hierarchy notifying the Controller where the metric should appear in the metric tree:

    Custom Metrics|Hardware Resources|Disks|Total Disk Usage %
Custom Metrics|Hardware Resources|Disks|Disk 1|Current Disk Usage %


    You can insert a custom metric next to an existing type of metric. For example, the following declaration causes the custom metric named pool usage to appear next to the JMX metrics:
    Server|Component:18|JMX|Pool|First|pool usage

  4. You can then use the metric in health rules as would other types of JMX metrics.
    To monitor multiple metrics with the same script file, edit the script to write a different line for each one to STDOUT. For example:  

    name=Custom Metrics|Hardware Resources|Disks|Total Disk Usage %, value=23
name=Custom Metrics|Hardware Resources|Disks|Disk 1|Current Disk Usage %, value=56

Copy the Script File to the Subdirectory Created in Step 1

Ensure that the Agent process has execute permissions for the script file and for the contents of the file.

Create the monitor.xml File

For each custom monitoring extension script, create a monitor.xml file. The monitor.xml file executes the script file created in Step 2. You can edit the following sample file to create your own file:

<monitor>
    <name>HardwareMonitor</name>
       <type>managed</type>
        <description>Monitors system resources - CPU, Memory, Network I/O, and Disk I/O.</description>
        <monitor-configuration>     </monitor-configuration>
        <monitor-run-task>
           <!--  Edit execution-style as needed. -->
           <execution-style>continuous</execution-style>
           <name>Run</name>
           <type>executable</type>
           <task-arguments></task-arguments>
            <executable-task>
                <type>file</type>
                <!--  Use only one file element per os-type. -->
                 <file os-type="linux">linux-stat.sh</file>
                 <file os-type="mac">macos-stat.sh</file>
                 <file os-type="windows">windows-stat.bat</file>
                 <file os-type="solaris">solaris-stat.sh</file>
                 <file os-type="sunos">solaris-stat.sh</file>
                 <file os-type="aix">aix-stat.sh</file>
           </executable-task>
         </monitor-run-task>
</monitor>


The os-type attribute is optional for the executable-task file element when only one os-type is specified. One monitor.xml file executes one script per os-type.

  1. Select the execution style: continuous or periodic.

    Execution Style

    Description

    Example

    continuous

    Select continuous if you want data collection averaged over time. For example, average CPU usage over a minute.
    For the monitor to be declared as 'continuous', the script should also run in an infinite loop. This ensures that the script keeps running until the Machine Agent process is terminated.

    while [ 1 ]; do
    ... the actual script goes here ...
    sleep 60
    done

    periodic

    Select periodic to report data from system performance counters periodically. The periodic task runs every minute by default, and the data is aggregated. 

    To specify a different frequency, use the execution-frequency-in-seconds element. The execution frequency must be less than 60. For periodic execution style, you can also specify the timeout setting (shown in the example).

    <monitor-run-task>

    ...

    <execution-style>periodic</execution-style> <execution-frequency-in-seconds>30</execution-frequency-in-seconds> <execution-timeout-in-secs>30</execution-timeout-in-secs>

    ...

    </monitor-run-task>

    Do not set the <execution-frequency-in-seconds> higher than 300 seconds (five minutes). The extension must collect metrics at least once every five minutes.

  2. Add the name of your script file to the <file> element in the monitor.xml file. Be sure to use the correct os-type attribute. The os-type value should match the value returned from calling System.getProperty("os.name"). 

    <file os-type="your-os-type">{script file name}</file>

    You can use either the relative or absolute path of the script.

Copy the monitor.xml file to the Subdirectory Created in Step 1

Restart the Machine Agent

Required Agent Properties

Ensure that you have correctly configured the Agent in the controller-info.xml file and on the Agent start command on the command line. See Database Agent Configuration Properties.

After restarting the Machine Agent, you should see following message in your log file:

Executing script [<script_name>] on the console to make sure your changes work with the machine agent.

Verify Execution of the Monitoring Extension Script

To verify the execution of extension, wait at least one minute and check the metric data in the Metric Browser.

You can now create alerts based on any of these metrics.

Example: Create a Monitoring Extension for Open Files in UNIX

To create a custom monitor for monitoring all the open files for JVMs:

  1. Create a new directory in the custom monitor repository.

  2. Create the script file by modifying this UNIX script for the specific process name (for example: Author, Publish, and so on).

    lookfor="<process name 1>"
pid=`ps aux | grep "$lookfor" | grep -v grep | tr -s " " | cut -f2 -d' '`
count1=`lsof -p $pid | wc -l | xargs`

lookfor="<process name 2>"
pid=`ps aux | grep "$lookfor" | grep -v grep | tr -s " " | cut -f2 -d' '`
count2=`lsof -p $pid | wc -l | xargs`

echo "name=JVM|Files|<process name 1>,value="$count1
echo "name=JVM|Files|<process name 2>,value="$count2

  3. Create the following monitor.xml file and point it to the UNIX script shown in Step 2.

    <monitor>
    <name>MyMonitors</name>
    <type>managed</type>
    <description>Monitor open file count </description>
    <monitor-configuration>
    </monitor-configuration>
    <monitor-run-task>
        <execution-style>continuous</execution-style>
        <name>Run</name>
        <type>executable</type>
        <task-arguments>
        </task-arguments>
        <executable-task>
            <type>file</type>
            <file>openfilecount.sh</file>
        </executable-task>
    </monitor-run-task>
</monitor>

Example: Create a Monitoring Extension to Check Process Status in Windows

  1. Create a new directory in the custom monitor repository.

  2. Create the script file by modifying this Windows PowerShell script for the specific process name (for example: Author, Publish, and so on). The following Windows PowerShell script checks if a process instance is running on the machine.

    To generate custom metrics on Windows, AppDynamics recommends that you use PowerShell and VBasic scripts instead of .bat files.
    When a standard Windows batch (.bat) script echoes metric names, it surrounds the names with quotes. The quotes will cause the Machine Agent to ignore these metrics. PowerShell and VBasic scripts do not have this issue.

    $procName='<PROCESS_NAME>'
$procStatus=Get-Process -Name $procName -ErrorAction SilentlyContinue
if($procStatus) {
  write-output "name=Custom Metrics|Process|$procName|Running, value=1"
} else {
  write-output "name=Custom Metrics|Process|$procName|Running, value=0"
}

  3. Create the following monitor.xml file and point it to the Windows PowerShell script shown in Step 2.

    <monitor>
    <name>MyMonitors</name>
    <type>managed</type>
    <description>Monitor open file count </description>
    <monitor-configuration>
    </monitor-configuration>
    <monitor-run-task>
        <execution-style>continuous</execution-style>
        <name>Run</name>
        <type>executable</type>
        <task-arguments>
             <argument name="" value="-Command"></argument>
             <argument name="" value="C:\\absolute\\path\\to\\powershell\\script\\mp.ps1"></argument>   
        </task-arguments>
        <executable-task>
           <type>command</type>                                                                            
		   <command>C:\\absolute\\path\\to\\powershell\\executable\\powershell.exe</command>
        </executable-task>
    </monitor-run-task>
</monitor>
    CODE

If you have provided the PowerShell program for the Machine Agent and are unable to see custom metrics, follow these steps:

  1. Create a .bat file in the same folder where you created the PowerShell script and monitor.xml file.

    Example .bat file

    process_checker.bat
    CODE

  2. Add the following lines of code into the .bat file. Ensure that you are using the correct PowerShell script file name. In this example, the PowerShell script file name is  process_status_checker.ps1.

    @echo off
Powershell.exe -executionpolicy remotesigned -File process_status_checker.ps1
    CODE

  3. Add the following lines of code to the monitor.xml file. Ensure that you are using the correct .bat file name. In this example, the .bat file name is process_checker.bat.

    <monitor>
  <name>MyMonitors</name>
  <type>managed</type>
  <description>Monitor process status</description>
  <monitor-configuration>
  </monitor-configuration>
  <monitor-run-task>
 <execution-style>periodic</execution-style>
 <execution-frequency-in-seconds>60</execution-frequency-in-seconds>
 <execution-timeout-in-secs>60</execution-timeout-in-secs>
    <name>MyMonitors</name>
    <type>executable</type>
    <task-arguments></task-arguments>
    <executable-task>
      <type>file</type>
      <file os-type=“windows”>process_checker.bat</file>
    </executable-task>
  </monitor-run-task>
</monitor>
    CODE