Related Pages:

This topic describes general techniques when troubleshooting AppDynamics Analytics deployment problems, along with a few specific scenarios you may encounter with workarounds. 

General Troubleshooting

If you encounter problems with your Analytics deployment, first check the logs for errors or warnings. The components write log information as follows:  

  • Analytics Dynamic Service: The Analytics Dynamic Service is built into the Java and .NET App Agents and writes logs to the same file as the App Agent. The logs are in the following location:
    <application_home>/<app_agent_home>/logs. 
    The primary log file to use for troubleshooting is the file named agent.<timestamp>.log file. Search the file for messages written by the Analytics Dynamic Service. These logs are useful for the following:
    • Determining if the Analytics Dynamic Service is enabled or disabled. 
    • Viewing the configurations used on startup.
    • Determining if the Analytics Dynamic Service is encountering errors sending messages to the Analytics Agent. For example, when the Analytics Dynamic Service is not able to communicate with the Analytics Agent due to invalid connection configuration.
    • Determining if messages are being dropped by the Analytics Dynamic Service because its internal buffers are full.
  • The Analytics Agent writes log messages to files in the following directory: 
    <analytics_agent_home>/logs 
  • The Events Service writes log messages to files in the following directory: 
    <events_service_home>/logs 
    In particular, the events-service-api-store.log file can help you with troubleshooting. 

Configuration Issues

Verify your configuration settings, particularly that they are properly configured with the required account name and key. Slashes in account names and key values need to be escaped, with the failure to do so a common source of configuration problems.

Transaction Analytics without an Analytics Agent (Agentless Analytics) Issues

This section discusses possible issues using agentless Transaction Analytics and how to disable the feature. 

Known Issues

AppDynamics has identified potential issues you may encounter while using agentless Transaction Analytics (see below for details). 

If you encounter any issues with agentless Analytics, AppDynamics recommends that you downgrade to Java Agent version 4.5.14. You should only disable Transaction Analytics without an Analytics Agent if you cannot downgrade the Java Agent. 

Firewall Configuration

Transaction Analytics without an Analytics Agent requires a connection between the Java Agent and the Events Service. If you have a firewall that blocks requests from the Java Agent to the SaaS or on-premises Events Service, you need to open the firewall. See Connection Ports and Agent-to-Controller Connections for more information.

AppDynamics recommends that you do not upgrade to Java Agent 4.5.15 and later if you cannot open the firewall. If you have already upgraded your deployment, downgrade to Java Agent 4.5.14. 

High CPU Usage

AppDynamics recommends that you downgrade to Java Agent 4.5.14 if you encounter high CPU usage. 

Disable Transaction Analytics without an Analytics Agent

The instructions to disable agentless Analytics apply to on-premises deployments only.

If you need to disable agentless Analytics for a SaaS deployment, contact AppDynamics Customer Support.

Transaction Analytics without an Analytics Agent is automatically enabled for:

  • Controller 4.5.17 and later (SaaS and on-prem)
  • Java Agent 4.5.15 and later

If you would like to remove this feature and use a local Analytics Agent:

  1. Navigate to http://<localhost:port>/controller/private/ControllerFlagsServlet.
  2. Set appdynamics.analytics.features.agentless.enabled to false. 

To disable the feature across Controller restarts, pass the property to the domain.xml file, as shown below:

<jvm-options>-Dappdynamics.analytics.features.agentless.enabled=false</jvm-options>
XML

If Transaction Analytics without an Analytics Agent was previously enabled for any applications, you need to manually remove and re-enable Transaction Analytics for the applications. 

  1. In the Controller UI, navigate to Analytics > Configuration. 
  2. Go to the Transaction Analytics - Configuration tab, and find the desired application. Uncheck and recheck the app, then select your business transaction(s).

Clock Management and Timestamps

AppDynamics recommends maintaining clock-time consistency throughout your monitored environment. If analytics metrics are always reporting zero, confirm that the clocks are synchronized across the application, Controller, and Events Service nodes. 

Understanding Analytics Data and Timestamps

There are potentially four time zones involved when dealing with log analytics, which you should be aware of:

  1. The timestamp and time zone from the log file.
  2. The event timestamp (and pickup timestamp) time zones can be different from that in the log for a number of reasons, such as the following:
    1.  When the time zone is overridden
    2. The time zone is not provided correctly in the log
    3. The timestamp and time zone parsing goes awry
    4. When no time zone is specified in the log timestamp, then local time is assumed
  3. The Events Service time zone, the Events Service stores all timestamps in UTC time.
  4. The browser used to view the analytics data in the Controller UI (such as event timestamp column displayed in the UI search results or the time picker widget) converts all timestamps to the browser's local time.

Limits for Business Transaction Events

The Analytics Dynamic Service sends messages to the Events Service where the request body is an array of event segments. A business transaction event consists of one or more segments that are related to each other by the business transaction requestGUID. There are ingestion limits related to messages:

  • Event (segment) size: The maximum size of an individual business transaction segment collected by the Analytics Dynamic Service is .1 MB. This limit is defined by the appdynamics.analytics.message.maxSizeBytes Java system property. To change this value, pass it on the command line as a system property when the Java Application Agent is started, for example: 

    '-Dappdynamics.analytics.message.maxSizeBytes=1024000'

  • Events per request: The maximum number of segments per request is defined by the appdynamics.analytics.agent.send.batch.items.max Java system property. The default value of this property is 16. To change this value, pass it on the command line as a system property when the Java Application Agent is started.  

  • Message Size: This limit refers to the size of a single request body sent to the Events Service, which is usually an array of event segments. Publish requests for all event types are limited to 1 MB. If the limit is exceeded, you will see exceptions in agent log file and messages in Events Service logs. 

Issues Starting the Analytics Agent

If an instance of the Analytics Agent terminates and leaves behind its process ID file (PID file), the next agent startup will fail with the following error:

java.lang.RuntimeException: Unable to create file [D:\AppDynamics\analytics-agent\analytics-agent.id] to store the process id because it already exists. Please stop any currently running process and delete the process id file

In Controller versions prior to 4.3, you needed to delete the old process id file and restart the agent to work around this issue.

In 4.3 and higher, you can use the -f option when starting the agent. This option causes a pre-existing process id file to be deleted. The flag is not required when you are starting the agent as a Windows service. 

Use the -f flag as follows:

  • UNIX type OS: start -f
  • Windows CLI: start -f

Log Analytics Missing Field Extractions

If you are missing fields in your Log Analytics data that you expect to see based on your source rule configuration or using regex (including grok patterns) in your field extraction, you may encounter a performance safeguard.

If a regex pattern takes more than five seconds to match against a log line, the attempt to extract the fields is terminated. No further processing that requires the extracted fields will occur. As a result, some fields may be missing when viewed on the controller for that log line. In this case, the following error message appears in the Analytics Agent log: 

[ERROR ] java.util.concurrent.TimeoutException: The current regex has spent 5 seconds attempting to match the log line, further processing has been stopped for this log line.

Another reason for missing fields is if the log line doesn't contain the field to be extracted as defined in the pattern. 

Custom Analytics Metrics

If you are having issues with metrics created from saved searches or with the alerts performance for those metrics, try increasing the query batch size. You can increase the size using the analytics.scheduledqueries.batch.size Controller setting in the Controller Administration Console. The default value for this setting is 5. 

See Access the Administration Console for information about accessing the setting.  

Firewall Considerations

For SaaS-based installations, you configure the Analytics endpoint by modifying the http.event.endpoint setting in the \conf\analytics-agent.properties file (as described in Installing Agent-Side Components). For example:

  • http.event.endpoint=http://analytics.api.appdynamics.com:443

If your firewall rules require you to use specific IP addresses, rather than hostnames, note the following information. If you are unable to see transaction analytics data collected as expected (even after configuring your firewall rules) and you see repetitive "Connection Reset" messages in the logs similar to the following, your firewall rules may not include the correct IP addresses.

[2015-12-18T16:08:39,907-06:00] [INFO ] [AD Thread Pool-Global0] [o.a.http.impl.execchain.RetryExec] I/O exception (java.net.SocketException) caught when processing request to {s}->https://analytics.api.appdynamics.com:443: Connection reset

Your firewall rules may not include the correct IP addresses.

In SaaS environments, both analytics.api.appdynamics.com, fra-ana-api.saas.appdynamics.com and syd-ana-api.saas.appdynamics.com are round-robin DNS aliases and may resolve to multiple DNS (54. vs 52.) such as in the following examples:

Name: analytics.api.appdynamics.com
Address: 54.213.173.141
Name: analytics.api.appdynamics.com
Address: 52.88.111.157
Name: fra-p-con-2.saas.appdynamics.com
Address: 52.28.42.67
Name: fra-p-con-2.saas.appdynamics.com
Address: 52.57.96.225
Name: syd-p-con-1.saas.appdynamics.com
Address: 54.153.248.179
Name: syd-p-con-1.saas.appdynamics.com
Address: 54.153.246.219

Amazon Web Services (AWS) controls the IPs used, so they may change from time to time. AWS publishes its current IP address ranges in JSON format, so if you are unable to open firewalls to hostnames, you can download the AWS IP address ranges. If you want to be notified whenever there is a change to the AWS IP address ranges, you can subscribe to receive notifications using Amazon SNS. 

See Analytics IP Ranges for the AWS regions for the SaaS Analytics environments. You can view the AWS IP ranges for each listed region.

Monitoring Health of the Analytics Agent

The check-health command returns the status of an Analytics Agent. You can specify the agent to check using a properties file or IP address and port. For example, using a properties file:

./bin/analytics-agent.sh check-health --properties conf/analytics-agent.properties

You can also use the -hp argument and pass the host IP address and the port number for the Analytics Agent. For example:

bin/analytics-agent.sh check-health -hp <ip-address>:9091

The default Analytics Agent port for the health check is 9091.

Usage for the check-health command is:

check-health
	-hp (--host-and-port) STRING[]
	(Optional. Application host name or IP:HTTP admin port (Multiple values separated by space))
	-p (--properties) TYPE_PATH
	(Optional. Path to the properties file (Multiple values separated by space))
	-v (--verbose) TYPE_BOOLEAN
	(Optional. Verbose output)

Known Issues

  • On Windows, you cannot delete a log file with the "del" command while the Analytics Agent is collecting log data from the file.
  • Avoid using robocopy/move commands to move files in Windows. Instead, it is recommended that you use the "move" command.

  • If you are running the Analytics Agent as an extension to the Machine Agent with JRE 1.8.0_171 or later, starting the Analytics Agent with encrypted credentials will fail. Disable encryption or use JRE 1.8.0_162 to run the Machine Agent.