Skip to end of metadata
Go to start of metadata

ADQL Reference:

Your Rating:
Results:
1 Star2 Star3 Star4 Star5 Star
37 rates

Overview

Selected fields in the analytics data are "analyzed". Analysis consists of tokenizing a block of text into individual terms suitable for use in an inverted index and normalizing these terms into a standard form to improve their searchability. The ADQL Comparison Operators behave differently for analyzed and non-analyzed fields. To construct queries on analyzed fields, you need to understand some concepts about how the tokens are built.

Uppercase letters in analyzed fields are all converted to lowercase to build tokens. 

Delimiters and other factors (such as CamelCase terms) affect how strings are tokenized. For a string such as "myname@company.com",  "@" is a delimiter, therefore myname and company.com are two separate tokens. Additionally company.com is two separate tokens. Note that all non-alphanumeric characters are delimiters. A term that uses CamelCase, such as VicePresident, is tokenized into separate tokens based on recognition of the CamelCase nature of the term resulting in the tokens: Vice and President as well as VicePresident.

For an example string such as:  <VicePresident:SalesAndMarketing> - EMEAAustraliaUSA94107, the tokens generated include the following:

  • vicepresident
  • vice
  • president
  • salesandmarketing
  • sales
  • and
  • marketing
  • emeaaustraliausa94107
  • emeaaustraliausa
  • emea
  • australia
  • usa
  • 94107

Analytics Analyzed Fields

The analyzed fields in analytics events are the following:

  • Logs: Message
  • Transactions: Error Detail and Errors
  • Mobile: stacktrace

Queries on Analyzed Fields

Full text search is supported on analyzed fields, including the message field for logs, using the LIKE operator. See Comparison Operators for details on using LIKE.

On analyzed fields, the REGEXP operator matches exactly only the analyzed and processed tokens, so you cannot query across the complete message.

Consider this log message: 

[2016-06-09 06:07:50,118] [INFO ] [org.springframework.jms.listener.DefaultMessageListenerContainer#0-1] [com.appdynamics.provision.OrderMessageListener] [AD_REQUEST_GUID[2b8ce807-986c-45f9-a5b4-2fe5a6fd90f3]] Received a message to process the order Order_3138 for the user myname@company.com

In this log message, myname and company.com are two separate tokens because "@" is a delimiter. To search a log message like this for results based on the email address requires searching across tokens. 

 A query such as the following using REGEXP, will fail because myname and company.com are two separate tokens. 

SELECT FROM logs WHERE appName = 'yourAppNameAND sourceType = 'yourLogFile' AND message REGEXP ‘myname@company.* ’

The LIKE operator is not affected by delimiters, so an alternative query using LIKE operator is a better choice.

SELECT FROM logs WHERE appName = 'yourAppNameAND sourceType = 'yourLogFileAND message LIKE ‘myname@company'

You can also use wildcards in the query because they work across tokens.

SELECT FROM logs WHERE appName = 'yourAppNameAND sourceType = 'yourLogFileAND message LIKE ‘myname@company*'

Example Queries for Analyzed Fields

Searching the following analytics log events:

Jun 7 11:27:45 appd sshd[1032]: Illegal user test from 110.49.183.11
Jun 7 11:27:46 appd sshd[1032]: Failed password for illegal user test from 110.49.183.11 port 9218 ssh2
Jun 7 11:27:46 appd sshd[1032]: error: Could not get shadow information for NOUSER

Note the results from the following queries:

QueryResults
SELECT * FROM logs WHERE sourceType='yourLogFile' AND message REGEXP 'illegal.+user'

Does not match any log events in the sample because the query string spans across multiple tokens. Use LIKE for an instance like this.

SELECT * FROM logs WHERE sourceType='yourLogFile' AND message REGEXP 'illegal.*'

Matches the first two log events

SELECT * FROM logs WHERE sourceType='yourLogFile' AND message REGEXP 'Failed*'Does not match any log events in the sample because the token has only lowercase ‘failed'

 To search for string : javaIOException

Query
SELECT * FROM transactions WHERE application = 'yourAppAND segments.errorList.errorCode REGEXP 'java[a-z][a-z][a-z][a-z][a-z][a-z][a-z][a-z][a-z][a-z][a-z]'
  • No labels