Selected fields in the analytics data are "analyzed". Analysis consists of tokenizing a block of text into individual terms suitable for use in an inverted index and normalizing these terms into a standard form to improve their searchability. The ADQL Comparison Operators behave differently for analyzed and non-analyzed fields. To construct queries on analyzed fields, you need to understand some concepts about how the tokens are built.
Uppercase letters in analyzed fields are all converted to lowercase to build tokens.
Delimiters and other factors (such as CamelCase terms) affect how strings are tokenized. For a string such as "firstname.lastname@example.org", "@" is a delimiter, therefore myname and company.com are two separate tokens. Additionally, company.com is two separate tokens. Note that all non-alphanumeric characters are delimiters. A term that uses CamelCase, such as
VicePresident, is tokenized into separate tokens based on recognition of the CamelCase nature of the term resulting in the tokens:
President as well as
For an example string such as:
<VicePresident:SalesAndMarketing> - EMEAAustraliaUSA94107, the tokens generated include the following:
Analytics Analyzed Fields
The analyzed fields in analytics events are the following:
- Logs: Message
- Transactions: Errors and Error Detail
- Mobile: stacktrace
Queries on Analyzed Fields
Full-text search is supported on analyzed fields, including the message field for logs, using the LIKE operator. See Comparison Operators for details on using LIKE.
On analyzed fields, the REGEXP operator matches exactly only the analyzed and processed tokens, so you cannot query across the complete message.
Consider this log message:
In this log message, myname and company.com are two separate tokens because "@" is a delimiter. To search a log message like this for results based on the email address requires searching across tokens.
A query such as the following using REGEXP, will fail because myname and company.com are two separate tokens.
The LIKE operator is not affected by delimiters, so an alternative query using LIKE operator is a better choice.
You can also use wildcards in the query because they work across tokens.
Example Queries for Analyzed Fields
Searching the following analytics log events:
Note the results from the following queries:
|SELECT * FROM logs WHERE sourceType='yourLogFile' AND message REGEXP 'illegal.+user'|
Does not match any log events in the sample because the query string spans across multiple tokens. Use LIKE for an instance like this.
|SELECT * FROM logs WHERE sourceType='yourLogFile' AND message REGEXP 'illegal.*'|
Matches the first two log events
|SELECT * FROM logs WHERE sourceType='yourLogFile' AND message REGEXP 'Failed*'||Does not match any log events in the sample because the token has only lowercase ‘failed'|
To search for string : javaIOException
|SELECT * FROM transactions WHERE application = 'yourApp' AND segments.errorList.errorCode REGEXP 'java[a-z][a-z][a-z][a-z][a-z][a-z][a-z][a-z][a-z][a-z][a-z]'|