Creating a filtering rule

You can filter the logs to include relevant log data and exclude data that you do not require. Filtering rules enable you to configure the grep configurations that define the data that you want to collect. For example, you set up the following grep configurations:

You can also create a filtering rule while you create collection policies.

To create a filtering rule

1. Click the Collection menu and select Filtering Rules.
2. On the Filtering Rules page, click Create.
3. In the Rule Information section, perform the following actions:
   1. Enter a unique name and description for the rule.
   2. In the Collection Type field, select the log collection type.
4. In the Rule Configuration section, perform the following actions:
   1. If you selected Kubernetes as the collection type, in the Type field, select Agent or Aggregator.
      The filtering rule will apply to the logs collected by the Agent or aggregated by the Aggregator.
   2. From the Log Filter list, select Grep.
   3. From the Directive list, select Regex (to include logs) or Exclude.
   4. (Only for the Agent configuration type) From the Condition list, select AND or OR.
   5. In the Key field, enter the key from the log expression.
      • Get the keys from the log expression. For example, in the Apache expression, host, user, time, method, path, code, size, refer, and agent are keys.

      • For a CSV type of parser, keys are the column names (or field names) of the columns in the CSV file.

        Important

        For the same directive, do not configure the same key more than one time. 

   6. In the Pattern field, enter the value to be included or excluded.
      For Apache, Apache Error, Nginx, and Regexp, enclose the values within forward slashes (//).

   7. Click + to add another grep expression.

      Examples

      Example for Apache, Apache Error, Nginx, and Regexp

      
      Sample logs:
      The value of the message field contains cool.
      The value of the hostname field matches web<INTEGER>.example.com.
      The value of the message field does NOT contain uncool.
      The following logs are collected:
      {"message":"It's cool outside today", "hostname":"web001.example.com"}
      {"message":"That's not cool", "hostname":"web1337.example.com"}
      The following logs are excluded:
      {"message":"I am cool but you are uncool", "hostname":"db001.example.com"}
      {"hostname":"web001.example.com"}
      {"message":"It's cool outside today"}

      Example for Java multiline

      
      Sample logs:
      The value of the message field contains cool.
      The value of the message field does NOT contain uncool.
      The following logs are collected:
      {"message":"It's cool outside today"}
      The following logs are excluded:
      {"message":"I am cool but you are uncool"}
      

      Example for Json

      
      Sample log:
      {"time":1362020400,"host":"111.111.0.1","size":777,"method":"PUT"}
      {"time":1362020400,"host":"111.111.0.1","size":777,"method":"POST"}
      {"time":1362020400,"host":"111.111.0.1","size":777,"method":"GET"}
      
      The following logs are collected:
      {"time":1362020400,"host":"111.111.0.1","size":777,"method":"GET"}
      The following logs are excluded:
      {"time":1362020400,"host":"111.111.0.1","size":777,"method":"PUT"}
      {"time":1362020400,"host":"111.111.0.1","size":777,"method":"POST"}

      Example for CSV

      
      Sample CSV format:

      2013/02/28 12:00:00,111.111.0.1,111,user1
      2013/02/28 12:00:00,111.111.0.1,111,user2
      2013/02/28 12:00:00,111.111.0.1,111,user3

      The following logs are collected:
      2013/02/28 12:00:00,111.111.0.1,111,user2
      2013/02/28 12:00:00,111.111.0.1,111,user3
      The following logs are excluded:
      2013/02/28 12:00:00,111.111.0.1,111,user1

5. Click Save.
   You can see the filtering rule on the Filtering Rules page.

Where to go from here

Creating collection policies