Selected fields in the analytics data are "analyzed". Analysis consists of tokenizing a block of text into individual terms suitable for use in an inverted index and normalizing these terms into a standard form to improve their searchability. The ADQL Comparison Operators behave differently for analyzed and non-analyzed fields. To construct queries on analyzed fields, you need to understand some concepts about how the tokens are built.
Uppercase letters in analyzed fields are all converted to lowercase to build tokens.
Delimiters and other factors (such as CamelCase terms) affect how strings are tokenized. For a string such as "email@example.com", "@" is a delimiter, therefore myname and company.com are two separate tokens. Additionally, company.com is two separate tokens. Note that all non-alphanumeric characters are delimiters. A term that uses CamelCase, such as
VicePresident, is tokenized into separate tokens based on recognition of the CamelCase nature of the term resulting in the tokens:
President as well as
For an example string such as:
<VicePresident:SalesAndMarketing> - EMEAAustraliaUSA94107, the tokens generated include the following:
The analyzed fields in analytics events are the following:
Full-text search is supported on analyzed fields, including the message field for logs, using the LIKE operator. See Comparison Operators for details on using LIKE.
On analyzed fields, the REGEXP operator matches exactly only the analyzed and processed tokens, so you cannot query across the complete message.
Consider this log message:
[2016-06-09 06:07:50,118] [INFO ] [org.springframework.jms.listener.DefaultMessageListenerContainer#0-1] [com.appdynamics.provision.OrderMessageListener] [AD_REQUEST_GUID[2b8ce807-986c-45f9-a5b4-2fe5a6fd90f3]] Received a message to process the order Order_3138 for the user firstname.lastname@example.org
In this log message, myname and company.com are two separate tokens because "@" is a delimiter. To search a log message like this for results based on the email address requires searching across tokens.
A query such as the following using REGEXP, will fail because myname and company.com are two separate tokens.
SELECT FROM logs WHERE appName = 'yourAppName' AND sourceType = 'yourLogFile' AND message REGEXP ‘myname@company.*
The LIKE operator is not affected by delimiters, so an alternative query using LIKE operator is a better choice.
SELECT FROM logs WHERE appName = 'yourAppName' AND sourceType = 'yourLogFile' AND message LIKE ‘myname@company'
You can also use wildcards in the query because they work across tokens.
SELECT FROM logs WHERE appName = 'yourAppName' AND sourceType = 'yourLogFile' AND message LIKE ‘myname@company*'
Searching the following analytics log events:
Jun 7 11:27:45 appd sshd: Illegal user test from 188.8.131.52 Jun 7 11:27:46 appd sshd: Failed password for illegal user test from 184.108.40.206 port 9218 ssh2 Jun 7 11:27:46 appd sshd: error: Could not get shadow information for NOUSER
Note the results from the following queries:
|SELECT * FROM logs WHERE sourceType='yourLogFile' AND message REGEXP 'illegal.+user'|
Does not match any log events in the sample because the query string spans across multiple tokens. Use LIKE for an instance like this.
|SELECT * FROM logs WHERE sourceType='yourLogFile' AND message REGEXP 'illegal.*'|
Matches the first two log events
|SELECT * FROM logs WHERE sourceType='yourLogFile' AND message REGEXP 'Failed*'||Does not match any log events in the sample because the token has only lowercase ‘failed'|
To search for string : javaIOException
|SELECT * FROM transactions WHERE application = 'yourApp' AND segments.errorList.errorCode REGEXP 'java[a-z][a-z][a-z][a-z][a-z][a-z][a-z][a-z][a-z][a-z][a-z]'|