Creating a filtering rule

Creating a filter rule is an optional step. You get the option to create filtering rules when you are creating collection policies.

After the logs are parsed, you can filter the logs to include relevant log data and exclude data that you do not require. Filtering rules enable you to configure the grep configurations that define the data that you want to collect. For example, you set up the following grep configurations:

Sample logs:
The value of the message field contains cool.
The value of the hostname field matches web<INTEGER>.example.com.
The value of the message field does NOT contain uncool.
The following logs are collected:
{"message":"It's cool outside today", "hostname":"web001.example.com"}
{"message":"That's not cool", "hostname":"web1337.example.com"}
The following logs are excluded:
{"message":"I am cool but you are uncool", "hostname":"db001.example.com"}
{"hostname":"web001.example.com"}
{"message":"It's cold outside today"}

To create a filtering rule

1. Click the Collection menu and select Filtering Rules.
2. On the Filtering Rules page, click Create.
3. Enter a unique name and description of the rule.
4. From the Log Filter list, select Grep.
5. From the Directive list, select Regex (to include logs) or Exclude.

6. In the Key field, enter the key from the log expression.
   Get the keys from the log expression. For example, in the Apache expression, host, user, time, method, path, code, size, refer, and agent are keys.
   For a CSV type of parser, keys are the column names (or field names) of the columns in the CSV file.

   Important

   For the same directive, do not configure the same key more than one time. 

7. In the Pattern field, enter the value to be included or excluded.
   For Apache, Apache Error, Nginx, and Regexp, enclose the values within forward slashes (//).

8. Click + to add another grep expression.

   Examples

   Example for Apache, Apache Error, Nginx, and Regexp

   
   Sample logs:
   The value of the message field contains cool.
   The value of the hostname field matches web<INTEGER>.example.com.
   The value of the message field does NOT contain uncool.
   The following logs are collected:
   {"message":"It's cool outside today", "hostname":"web001.example.com"}
   {"message":"That's not cool", "hostname":"web1337.example.com"}
   The following logs are excluded:
   {"message":"I am cool but you are uncool", "hostname":"db001.example.com"}
   {"hostname":"web001.example.com"}
   {"message":"It's cool outside today"}

   Example for Java multiline

   
   Sample logs:
   The value of the message field contains cool.
   The value of the message field does NOT contain uncool.
   The following logs are collected:
   {"message":"It's cool outside today"}
   The following logs are excluded:
   {"message":"I am cool but you are uncool"}
   

   Example for Json

   
   Sample log:
   {"time":1362020400,"host":"111.111.0.1","size":777,"method":"PUT"}
   {"time":1362020400,"host":"111.111.0.1","size":777,"method":"POST"}
   {"time":1362020400,"host":"111.111.0.1","size":777,"method":"GET"}
   
   The following logs are collected:
   {"time":1362020400,"host":"111.111.0.1","size":777,"method":"GET"}
   The following logs are excluded:
   {"time":1362020400,"host":"111.111.0.1","size":777,"method":"PUT"}
   {"time":1362020400,"host":"111.111.0.1","size":777,"method":"POST"}

   Example for CSV

   
   Sample CSV format:

   2013/02/28 12:00:00,111.111.0.1,111,user1
   2013/02/28 12:00:00,111.111.0.1,111,user2
   2013/02/28 12:00:00,111.111.0.1,111,user3

   The following logs are collected:
   2013/02/28 12:00:00,111.111.0.1,111,user2
   2013/02/28 12:00:00,111.111.0.1,111,user3
   The following logs are excluded:
   2013/02/28 12:00:00,111.111.0.1,111,user1

Where to go from here

Creating-collection-policies