Backend Detection Rules

In AppDynamics, databases and remote services (and in fact, any detected out of process components that are involved in Business Transaction processing) are collectively known as backends.

AppDynamics discovers backends from exit point instrumentation placed in the application code. An exit point is the precise location in the code where an outbound call is made from an instrumented node. AppDynamics automatically discovers a wide range of backends.

Default Automatic Backend Discovery

A discoverable backend is identified by its type and associated properties. The precise set of properties used to identify any given backend is dependent on its type. The type itself is defined in the built-in backend discovery rules for backend types supported out of the box. For exit calls to backends that are not instrumented with APM agents, AppDynamics uses the backend properties to identify and name the backend. Where a backend call is actually processed by a downstream tier also instrumented with AppDynamics, ensure that any given backend that is identified will always result in calls that will be processed by just one downstream tier.

This means, for example, that if tier A makes an HTTP call to an HTTP router on localhost:4040 which will forward the call to tier B or tier C depending on the URL, custom backend naming rules will be required to include enough segments of the request URL - as well as the host and port - such that requests destined for tier B will be associated with a backend with a different set of backend properties than those associated with tier C.

The Automatic Backend Discovery list in the Configuration > Instrumentation > Backend Detection tab shows the configurable backend discovery rules.

The automatic discovery rules vary according to the type of backend being identified, but generally include settings for enabling automatic discovery of the backend type, enabling correlation, and properties used to identify and name the backend. AppDynamics uses transaction correlation to track request processing across distributed tiers. Certain types of backends are automatically discovered as high volume exit points (sometimes called turbo exit points). These backend types include cache servers, EhCache, Danga Memcache, Memcached, and Oracle Coherence. For more about high volume exit points, see Exit Point Detection Rules.

When an exit point is identified for the first time, the exit call results in a backend discovery event. 

Service Proxy

Sometimes an inaccurate identification of the backend destination causes each call that is sent (based on configuration) to a single backend, to arrive at one of several different downstream tiers.

For example, this situation may occur if the naming of the backend is based on the backend host and port where a particular backend's host:port combination actually identifies an Apache server. The Apache server then forwards requests to different downstream tiers based on the URL acting as a service proxy. When the AppDynamics Controller detects this situation, it will automatically instruct the upstream agent to identify the backend as a service proxy. Once that occurs, a service proxy icon will display on the flow map between the upstream tier and all of the possible target tiers shown downstream. See Service Proxy.

Permissions

To modify backend detection, you need the Configure Backend Detection permission.

Modify Automatic Discovery

If you know that the application flow map should be showing a specific backend and you are not seeing it, you can modify the configuration to detect backends. You can:

• Edit the default backend discovery rules.
• Create new custom backend discovery rules.
• Create new custom exit points to provide visibility for backend types that are not automatically discovered.
• Disable automatic discovery of backends that are not of interest.
• Disable correlation for the backend.

In many cases, you can achieve the level of customization that you need by editing the default rules rather than creating new rules.

To edit a rule, in the Backend Detection tab, select the application and click Edit Automatic Discovery. 

For certain backend types—particularly those named by URL—the name of the discovered backend instances include a port number. If the app agent is unable to discover the port number for some reason, the port number is displayed in the UI as "-1". This can happen, for example, when a port is not a public port or the API for discovering the port is not available on the application environment (often the case for an RMI backend type). Despite having an unknown port number, the backends are nonetheless uniquely identified.

Configurable Properties and Naming

For many of the automatically discovered backend types, AppDynamics uses a set of configurable properties to identify and name the backend. For example, the properties for a JMS messaging queue are something like this:

• DestinationType (for example, QUEUE)
• Destination (for example, OrderQueue)
• Vendor (for example, Active MQ)

The following informational popup shows how the values are used to identify a discovered backend. In the following example, callout 1 shows the value that corresponds to the destination type, callout 2 to the vendor name, and callout 3 to the destination queue name: 

Aggregate or Split Backends

You can change the default configuration for the backend types shown in the UI to aggregate or split the backends. For example, you could disable the Vendor property for JMS backend detection to aggregate (join) all JMS backends with the same destination and destination type. The metrics for all JMS backends with the same destination and destination type would be aggregated as one backend.

For properties that you want to use, you can also configure how AppDynamics uses the property. For example, if the URL property is used to identify a backend and your URLs include a file name, the default configuration results in a separate backend for every file.

To reduce the number of backends identified using the URL property, configure which parts of the URL to use. For example, run a regular expression on the URL to discard the file name or any other parts of the URL that you do not want used for identification.

Add Custom Backend Discovery Rules

AppDynamics provides the additional flexibility to create new custom discovery rules for the automatically discovered backend types. 

To create a custom discovery rule for an automatically discovered backend type, select the backend type in the Automatic Backend Discovery list. Click Add (the + icon) to create a new rule. Specify the following:

• Name for the custom rule.
• Enabled to enable or disable the rule.
• Correlation Enabled to specify whether or not to send business transaction correlation data to the backend.
• Priority for this custom rule compared to other custom rules for this backend type. The higher the number, the higher the priority. A value of 0 (zero) indicates that the default rule should be used.
• Match Conditions used to identify which backends are subject to the custom naming rules. AppDynamics applies the default rules to backends that do not meet all the defined match conditions.
• Backend Naming Configuration used to name the backends matching the match conditions. The naming configuration must include the property used by the match condition.

See specific exit points for examples.

Backend Detection Rule Priorities

When a request matches more than one type of backend detection rule, AppDynamics agents apply backend detection rules in the following order:

Custom Exit Points > Custom Discovery Rules >  Automatic Backend Discovery.

If detection rules are of the same type, AppDynamics agents apply backend detection rules based on rule priority from highest to lowest. A priority of 0 is the lowest priority possible.

For example, consider an HTTP request that matches a priority 2 custom discovery rule and a priority 6 custom discovery rule. The agent applies the priority 6 rule.

To configure exit point detection:

1. Go to the Configuration > Instrumentation page.
2. Select the Backend Detection tab.
3. Select the application or tier on which you want to configure an exit point and the application type of the exit point, Java, .NET, and so on. From there you can add new exit points or modify existing ones. 

All Other Traffic Backends

AppDynamics limits the number of backend systems that can be registered and tracked. Once the limit is reached (databases and remote services combined), additional backend are put in the "All other traffic for <type>" category.

The "type" is the backend type such as HTTP or Web Service. HTTP calls and JMS queues are the most common types that may reach the limit with the default configuration rules. 

On a flow map, the category appears as follows:

Metrics for the backend calls grouped in the "All other traffic" category are aggregated. If the group includes backend instances that you want to be individually tracked, you can manage backend detection, as described in Manage Backend Discovery.   

Backend Registration Limits

The limits on the number of backends that can be registered:

• The Controller enforces a limit of 1000 backend registrations in the business application for each type of backend (that is, 1000 HTTP backends, 1000 JMS backends, and so on). This setting is controlled by the backend.registration.limit Controller Setting. This limit applies whether the backends are resolved to a tier or unresolved. 
• Agents apply a limit of 300 unresolved backends per application, regardless of the type. When determining if the limit has been reached, it's important to include all types. For example, agents, databases, remote services, etc., should all be included when totaling unregistered backends if it appear that the limit has not been reached but the logs have reported it has. To safeguard against reaching the limit, we recommend that you delete unnecessary backends.

When the backend count exceeds the limit, the "All Other Traffic" category is automatically created. The logs indicate that a limit has been reached by a log entry such as the following:

[pool-1-thread-2] 21 Mar 2013 11:28:01,702 DEBUG AFastBackendResolver - Not discovering backend call [Exit Point Type [WEB_SERVICE] Properties [{SERVICE_NAME=ZipCodeService002.wsdl}] Display Name [ZipCodeService002.wsdl]] because maximum limit of discovered backends [300] reached.

Manage Backend Discovery

After reaching the backend registration limit, you should make sure that the systems being tracked as first-class registered backends are the ones that are important to you relative to those in the "All Other Traffic" category.

Also make sure that application environmental conditions are not causing an excess number of backends to be registered. These conditions may include:

• JMS queues that use the session ID in the destination. This causes each session to be identified as a separate backend.
• Calls to message queues hosted on the same server. In this case, you might not be interested in seeing each queue separately, and instead, want to aggregate everything for the same host and port to the same backend.
• Dynamic generation of IP addresses, such as when you host a service on Amazon Elastic Compute Cloud (Amazon EC2).
• Different JDBC drivers connecting to the same database may cause many backends. This can happen when there are slight variations in the driver metadata that AppDynamics uses to identify the database. For example, if two different JDBC drivers are used to access the same database, something as simple as a different spelling or format of the database name (ORACLE, oracle, or ORACL) can generate multiple database names when they are the actually the same physical database.

To manage backend registrations, you can change the configuration of the backend detection rules. For example, if a property causes excessive unique identification, consider removing the property or modifying its use in the detection and naming rule. If you created custom exit points, consider refining or removing them. 

Delete Unnecessary Backends

Some backends may no longer have traffic. You can delete unused backends from the Remote Services List, any Remote Service dashboard, the Databases List,  and any Database dashboard. If the backend has new traffic in the future, the app agent rediscovers it and reregisters it with the Controller.

You can also configure the Controller to automatically remove stale backends. See Stale Remote Service Removal.

Organize Backends in Flow Maps

Once you have configured detection for all the backends you require, you may need to organize the way they appear in the UI. See Group Remote Services on Flow Maps.