Creating a parsing rule
A parsing rule consists of a regular expression that helps you to parse the data present in your log files. Logs are parsed based on the format present in the logs. For more information, see Fluentd documentation.
Example
Here are the expression and date formats for the Apache log format. These expressions are provided for all supported formats (wherever required) when you configure a parsing rule.
Expression (Apache): /^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>(?:[^\"]|\\.)*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>(?:[^\"]|\\.)*)" "(?<agent>(?:[^\"]|\\.)*)")?$/
Time Format: %d/%b/%Y:%H:%M:%S %z
Log entry: 192.168.0.1 - - [28/Feb/2013:12:00:00 +0900] "GET / HTTP/1.1" 200 777 "-" "Opera/12.0"
Parsed as:
time:
1362020400 (28/Feb/2013:12:00:00 +0900)
record:
{
"user" : nil,
"method" : "GET",
"code" : 200,
"size" : 777,
"host" : "192.168.0.1",
"path" : "/",
"referer": nil,
"agent" : "Opera/12.0"
}
Before you begin
Install the connector for log collection. For more information, see Installing-and-managing-connectors.
To create a parsing rule
- Click the Collection menu and select Parsing Rules.
- On the Parsing Rules page, click Create.
- In the Rule Information section, perform the following steps:
- Enter a unique name and description of the rule.
- From the Collection Type list, select the logs that you are collecting.
Based on the collection type, different options are made available in the Format list.
In the Rule Configuration section, perform the steps as described in the following table based on the log format:
- Click Save.
You can see the parsing rule on the Parsing Rules page.
Where to go from here