Server-side log parsing rules extract fields from incoming log messages at the time of ingestion, which has many benefits:

  • It speeds up root cause analysis and reduces mean-time-to-resolution (MTTR). Instead of manually sifting through log messages to find relevant information, you can quickly filter log messages based on extracted fields and focus on the relevant log messages.
  • It allows you to leverage pre-defined extracted fields for searching, alerting, and creating dashboards.
  • By saving parsing rules on the server, you can standardize field names, simplify searches, improve search performance, and apply the same rules to new log sources and hosts.

You must be logged into the Cisco Cloud Observability interface with the role of Configuration Manager in order to configure parsing rules. Other roles can view parsing rules but can't configure them.


  1. Click Configure > Log Processing.
  2. If you've never created a server-side processing rule, click Get started. Otherwise, click + Create rule.
  3. On the Create Processing Rule form:
    1. In Rule scope, select a log attribute from the pull-down menu. To find the attribute you want, start typing into the pull-down menu. After you select an attribute, select IN operator and type the value which this attribute must match in order to trigger this log processing rule. You can use && operator if you want to add more than one attribute along with its value to the rule scope. 
      For example, when you specify the rule scope filter as attributes(host.name)='appd_1' && attributes(service.name)='appd_2', the rule gets applied to the logs that include host.name as appd_1 and service.name as appd_2.
      However, when you specify the rule scope filter as attributes(host.name) IN ['appd_1', 'appd_2']  the rule gets applied to the logs that include host.name as appd_1 or appd_2.
      After adding attributes, click Apply.
      The form displays the first 200 log messages that match your rule scope.

      If any log messages match your rule scope, the form lists the 200 most recent log messages. 
    2. If no log messages are listed, repeat the previous step, but select a different attribute or value in Rule Scope. 

    3. In the list of log messages, select the radio button next to the one you want to use as your sample for this parsing rule, and click Next.

      You need to select a sample log message in order to validate the parsing rule you'll set up in the next step. To search for a log message to use as your sample, type into the search box.

    4. In Sample log, ensure that your sample message is displayed.

    5. In Parser type, select the parser which is most likely to match your sample log message in Sample log.

      The following parser types autopopulate the Parsing Pattern field with a predefined GROK pattern:

      • APACHE
      • ELB
      • NGINX
      • ALB
      • KAFKA
      • SQLSERVER

      • POSTGRESQL

      • MYSQL
      • ZOOKEEPER
      • REDIS
      • AWSS3ACCESS
      • HAPROXY

      If you select any of these parser types, you don't need to specify a parser pattern. Predefined patterns are not editable. If you need to create your own pattern, select parser type GROK, JSON, Log4j, Logback, or Timestamp.

    6. In Parsing Pattern, construct a pattern that is most likely to match your sample log message.

      • To get started, click Start with one of our examples and copy patterns from the examples shown in the right pane.
      • In addition to specifying the right pattern to match each field in your log message, you should also name the fields your pattern will extract. Naming conventions vary by parser type. For help, see the examples in Start with one of our examples and then click Next.
      • JSON log messages don't need a parsing pattern. If your Parser type is JSON, no pattern is necessary; click Next.

       

    7. To test your parsing pattern, click Preview. The Preview button is not visible if Parser type is JSON.
      Your parsed fields appear in the Extracted fields pane on the right.
    8. If you're satisfied with the parsed fields, click Next. Otherwise, repeat the previous steps.
    9. In Processing rule name, type a name for this rule. Names must contain alphanumeric characters, underscores, dashes, dots only. Maximum of 64 characters.
      Your new rule is now listed on the Processing Rules page, and is enabled by default. Parsing rules only apply to new, incoming logs. Existing logs are not parsed retroactively.
    10. Click Save.

Enable or Disable a Parsing Rule

Click the slider in a rule's Status column to enable or disable it. When you enable a processing rule, it takes about 5 minutes to go into effect.

Rule Precedence

  • If the Processing Rules page lists multiple rules that are applicable to any log message, the system applies the most recent rule to the log message. The list of rules on the Processing Rules are sorted in oldest-first order.
  • If there are applicable rules listed on the Processing Rules page as well as in collectors-values.yaml, the system applies the most recent rule on the Processing Rules page.
  • The Processing Rules page only supports editing of rules that you defined through this interface. If you also defined rules in collectors-values.yaml, you need to edit them in that file.