Parsing logs
Logs are parsed through a parser before collection and the parsed logs are displayed in the Discover tab in BMC Helix Log Analytics. Logs are parsed based on the format present in the logs. A log expression informs the parser what information is present in the logs.
For more information, see Fluentd documentation.
Supported parsing formats
The following table provides detailed information about the supported parsing formats:
Format | Supported in logs collection from files | Supported in logs collection from Amazon Web Services | Description |
|---|---|---|---|
Apache, Apache Error, Nginx, and Regexp | ✅️ | ✅️ | For these formats, expression and supported date format are displayed in the Expression and Time Format fields. Update the expression or date format based on the expression and date format present in your log files. To parse logs with a custom expression, use the Regexp format. ExampleSample log: [Mon Jan 10 02:13:55 2022] [necessitatibus:notice] [pid 5441:tid 6660] [client 11.111.111.111:2222] The TCP bus is down, override the wireless capacitor so we can connect the XML interface! Default expression: /^\[[^ ]* (?<time>[^\]]*)\] \[(?<level>[^\]]*)\](?: \[pid (?<pid>[^\]]*)\])? \[client (?<client>[^\]]*)\] (?<message>.*)$/ Updated expression to parse logs without port number: /^\[[^ ]* (?<time>[^\]]*)\] \[(?<level>[^\]]*)\](?: \[pid (?<pid>[^\]]*)\])? \[client (?<client>\d+\.\d+\.\d+\.\d+):\d+\].(?<message>.*)$/
|
Java multiline | ✅️ | ❌️ | Date, firstline, and time format expressions are displayed in the Format Firstline, Format 1, and Time format fields. To parse the following sample logs: 2021-09-07 14:19:17 INFO [main] Generating some log messages 0 Modify the default expression for multiline. Here is how you can modify the out-of-the-box expression (note the square brackets location in the expressions): Updated: /^(?<time>\d{4}-\d{1,2}-\d{1,2} \d{1,2}:\d{1,2}:\d{1,2}) (?<thread>.*) \[(?<level>[^\s]+)\](?<message>.*)/ To verify the expression, visit rubular or fluentular. |
Json | ✅️ | ✅️ | In the Time Key field, enter the key or field in which time value is present in the logs. In the Time Format field, enter the time format present in your logs. |
CSV | ✅️ | ✅️ | In the Keys field, enter the field names (separated by comma) that you want to provide to the values in the CSV file in the order they appear in the file. In the Time Format field, enter the time format present in your CSV file. For example, a CSV contains the following values: 2013/02/28 12:00:00,192.168.0.1,111,user1 2013/02/28 12:00:00,192.168.0.1,112,user2 2013/02/28 12:00:00,192.168.0.1,113,user3 For this example, enter time,host,req_ID,user. The CSV is parsed as:
|
No parser | ✅️ | ✅️ | Logs are collected without any parser. Each log line is collected as a separate record. The @timestamp field of these collected logs will contain the log collection time. |
