×
 
 
  •  
$helper.renderConfluenceMacro('{bmc-global-announcement:$space.key}')
BMC Helix Log Analytics 22.1 Collecting logs

Collecting logs by using Fluentd

Collect logs by using docker-based Fluentd that you can configure by using the Collect logs from file option added to BMC Helix Developer Tools. Set up parsing based on the log formats present in your log files and also set up filtering rules to include or exclude logs.

Logs parsing

Logs are parsed through a parser before collection and the parsed logs are displayed in the Discover tab in BMC Helix Log Analytics. A log expression informs the parser what information is present in the logs. You can also use the expression to filter logs for collection.

Let's look at an example to help understand parsing. Here are the expression and date format for Apache. These expressions are provided for all supported formats (wherever required) when you configure a log collection.

Expression (Apache): /^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>(?:[^\"]|\\.)*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>(?:[^\"]|\\.)*)" "(?<agent>(?:[^\"]|\\.)*)")?$/
Time Format: %d/%b/%Y:%H:%M:%S %z

Log entry: 192.168.0.1 - - [28/Feb/2013:12:00:00 +0900] "GET / HTTP/1.1" 200 777 "-" "Opera/12.0"

Parsed as:

time:
1362020400 (28/Feb/2013:12:00:00 +0900)
record:
{
"user" : nil,
"method" : "GET",
"code" : 200,
"size" : 777,
"host" : "192.168.0.1",
"path" : "/",
"referer": nil,
"agent" : "Opera/12.0"
}

To parse logs with different expressions, you can either update the default expressions or use a custom format.

For more information, see Fluentd documentation Open link .

Logs filtering

After the logs are parsed, you can filter the logs to include relevant log data and exclude data that you do not require. For example, you set up the following grep configurations

 

Sample logs:
The value of the message field contains cool.
The value of the hostname field matches web<INTEGER>.example.com.
The value of the message field does NOT contain uncool.
The following logs are collected:
{"message":"It's cool outside today", "hostname":"web001.example.com"}
{"message":"That's not cool", "hostname":"web1337.example.com"}
The following logs are excluded:
{"message":"I am cool but you are uncool", "hostname":"db001.example.com"}
{"hostname":"web001.example.com"}
{"message":"It's cool outside today"}

Before you begin

Perform the following actions before collecting logs:

To collect logs from files

  1. In BMC Helix Developer Tools, open the Available integrations tab if it is not open by default.

  2. On the Collect Logs from File tile, click Configure.


    On this tile, the build number of the product and the name of the entity that made the product available for configuration are displayed. In this example, the product is made available by BMC.
  3. Enter the integration name.

  4. Select the connector that you downloaded and saved.
    If no connector is available, click Add Connector to download and install a connector. 
  5. In the Customize Entity Configuration section, click Configure.

  6. In the Log Collection File Path field, enter the locations to collect logs.
    Enter only directory paths and an absolute file name with the path. Separate multiple entries with a comma. Ensure that all log files have the same format. Supported formats are available in the Format field. 

    Example 1: Log file is present in the logs folder of your connector (/opt/bmc/connectors/<connector_name>/logs/<log_file_name>), enter /fluentd/log/<log_file_name>. Replace /opt/bmc/connectors/<connector_name>/logs with fluentd/log.
    Example 2: Log file is present in a folder (other than the connector logs folder) on the same computer (or virtual machine) where the connector is installed. Mount the folder with log files to the connector log folder (such as, /opt/bmc/connectors/<connector_name>/logs). Then, enter /fluentd/log/<log_file_name>.
    Example 3: Log file location and connector are on different computers (or virtual machines). Mount the remote folder with log files to the connector log folder (such as, /opt/bmc/connectors/<connector_name>/logs). Then, enter /fluentd/log/<log_file_name>. 

  7. (Optional) If you have entered a path with multiple folders and you want to exclude some folders from collection, in the Exclude Paths field, remove those folders.
    For example, you have entered the log collection path as /opt/bmc/connectors/<connector_name>/logs/applicationLogs and this folder contains the following folders: app1, app2, app3. The app1, app2, and app3 folders are shown in the Exclude Paths field. To prevent log collection from the app3 folder, remove the app3 folder from the field.  

  8. From the Format field, select the format present in your logs and perform the appropriate steps to filter the logs.
    The following table lists the available formats and the steps to use them:

    FormatDescription

    Apache, Apache Error, Nginx, and Regexp

    For these formats, expression and supported date format are displayed in the Expression and Time Format fields. Update the expression or date format based on the expression and date format present in your log files. 

    Sample log:

    [Mon Jan 10 02:13:55 2022] [necessitatibus:notice] [pid 5441:tid 6660] [client 11.111.111.111:2222] The TCP bus is down, override the wireless capacitor so we can connect the XML interface!
    [Mon Jan 10 02:13:55 2022] [necessitatibus:info] [pid 9948:tid 2588] [client 22.222.222.22:3333] You can't bypass the program without programming the bluetooth HDD sensor!
    [Mon Jan 10 02:13:55 2022] [et:notice] [pid 4498:tid 4891] [client 111.111.111.1:4444] Programming the alarm won't do anything, we need to hack the 1080p EXE protocol!

    Default expression: /^\[[^ ]* (?<time>[^\]]*)\] \[(?<level>[^\]]*)\](?: \[pid (?<pid>[^\]]*)\])? \[client (?<client>[^\]]*)\] (?<message>.*)$/

    Updated expression to parse logs without port number: /^\[[^ ]* (?<time>[^\]]*)\] \[(?<level>[^\]]*)\](?: \[pid (?<pid>[^\]]*)\])? \[client (?<client>\d+\.\d+\.\d+\.\d+):\d+\].(?<message>.*)$/

    1. From the Log Filter list, select Grep.
    2. From the Directive field, select Regex (to include logs) or Exclude.
    3. In the Key field, enter the key from the log expression.
      Get the keys from the log expression. For example, in the Apache expression, host, user, time, method, path, code, size, refer, and agent are keys.
    4. In the Pattern field, enter the value to be included or excluded, enclosed within forward slashes (//).
    5. Click + to add another grep expression.
      Here is an example:

      Sample logs:
      The value of the message field contains cool.
      The value of the hostname field matches web<INTEGER>.example.com.
      The value of the message field does NOT contain uncool.
      The following logs are collected:
      {"message":"It's cool outside today", "hostname":"web001.example.com"}
      {"message":"That's not cool", "hostname":"web1337.example.com"}
      The following logs are excluded:
      {"message":"I am cool but you are uncool", "hostname":"db001.example.com"}
      {"hostname":"web001.example.com"}
      {"message":"It's cool outside today"}

    Java multiline

    Date format and firstline format expressions are displayed in the Format Firstline and Format 1 fields.

    To parse the following sample logs:

    2021-09-07 14:19:17 INFO [main] Generating some log messages 0
    2021-09-07 14:19:17 INFO [main] Sleeping for 1 second.
    2021-09-07 14:19:17 INFO [main] Generating some log messages 1

    Modify the default expression for multiline. Here is how you can modify the out-of-the-box expression (note the square brackets location in the expressions):
    Default: /^(?<time>\d{4}-\d{1,2}-\d{1,2} \d{1,2}:\d{1,2}:\d{1,2}) \[(?<thread>.*)\] (?<level>[^\s]+)(?<message>.*)/

    Updated: /^(?<time>\d{4}-\d{1,2}-\d{1,2} \d{1,2}:\d{1,2}:\d{1,2}) (?<thread>.*) \[(?<level>[^\s]+)\](?<message>.*)/

    To verify the expression, visit  rubular Open link or  fluentular Open link .

    1. From the Log Filter list, select Grep.
    2. From the Directive field, select Regex (to include logs) or Exclude.
    3. In the Key field, enter the key from the log expression.
      You can get the keys from the log expression. For example, in the Java multiline expression, time, thread, level, and message are keys.
    4. In the Pattern field, enter the value to be included or excluded, enclosesd within forward slashes (//).
    5. Click + to add another grep expression.
      Here is an example:

      Sample logs:
      The value of the message field contains cool.
      The value of the message field does NOT contain uncool.
      The following logs are collected:
      {"message":"It's cool outside today"}
      The following logs are excluded:
      {"message":"I am cool but you are uncool"}

    Json


    1. From the Log Filter list, select Grep.
    2. From the Directive field, select Regex (to include logs) or Exclude.
    3. In the Key field, enter the key from the log expression.
      Get the keys from logs. For example, you have the following log entry: {"time":1362020400,"host":"111.111.0.1","size":777,"method":"PUT"}. Here, you have the following keys: time, host, size, and method.
    4. In the Pattern field, enter the value to be included or excluded, enclosesd within forward slashes (//).
    5. Click + to add another grep expression.
      Here is an example:

      Sample log:
      {"time":1362020400,"host":"111.111.0.1","size":777,"method":"PUT"}
      {"time":1362020400,"host":"111.111.0.1","size":777,"method":"POST"}
      {"time":1362020400,"host":"111.111.0.1","size":777,"method":"GET"}

      The following logs are collected:
      {"time":1362020400,"host":"111.111.0.1","size":777,"method":"GET"}
      The following logs are excluded:
      {"time":1362020400,"host":"111.111.0.1","size":777,"method":"PUT"}
      {"time":1362020400,"host":"111.111.0.1","size":777,"method":"POST"}

    CSV

    Enter the field names (separated by comma) that you want to provide to the values in the CSV file in the order they appear in the file.

    For example, a CSV contains the following values:

    2013/02/28 12:00:00,192.168.0.1,111,user1

    2013/02/28 12:00:00,192.168.0.1,112,user2

    2013/02/28 12:00:00,192.168.0.1,113,user3

    For this example, enter time,host,req_ID,user.

    The CSV is parsed as:

    1. From the Log Filter list, select Grep.
    2. From the Directive field, select Regex (to include logs) or Exclude.
    3. In the Key field, enter the key from the log expression.
      Keys are the field names that you entered for the columns in the CSV file.
    4. In the Pattern field, enter the value to be included or excluded, enclosesd within forward slashes (//).
    5. Click + to add another grep expression.
      Here is an example:

      Sample CSV format:

      2013/02/28 12:00:00,111.111.0.1,111,user1
      2013/02/28 12:00:00,111.111.0.1,111,user2
      2013/02/28 12:00:00,111.111.0.1,111,user3


      The following logs are collected:
      2013/02/28 12:00:00,111.111.0.1,111,user2
      2013/02/28 12:00:00,111.111.0.1,111,user3
      The following logs are excluded:
      2013/02/28 12:00:00,111.111.0.1,111,user1
    Custom

    Use the Custom option in the following scenarios:

    • The out-of-the-box expression is different from the logs expression in your files.
    • The expression in your log files contains multiple key-value pairs.
    • The required log format is not listed in the field.

    To use the Custom option, enter the format in the Type field and configure the expression in the form of parameter name and parameter value. 
    The following expressions are supported:

    • regexp
    • apache2
    • apache_error
    • nginx
    • csv
    • json
    • multiline

    Example:
    For multiline format type, enter the following values:

    Parameter Name: format_firstline; Parameter Value: /\d{4}-\d{1,2}-\d{1,2}/

    Parameter Name: format1; Parameter Value: /^(?<time>\d{4}-\d{1,2}-\d{1,2} \d{1,2}:\d{1,2}:\d{1,2}) \[(?<thread>.*)\] (?<level>[^\s]+)(?<message>.*)/

    1. From the Log Filter list, select Grep.
    2. From the Directive list, select Regexp or Exclude.
    3. Enter the key and pattern.
      Get the keys from the log expression.
    4. Click + to add another grep expression. 

      To verify the expression, visit  rubular Open link or  fluentular Open link .

  9. In the Tags field, enter the tags to identify logs of the specified files.

  10. Save the entity configuration and then save the integration.
    You can view the added configuration by clicking Integrations and then Configured Integrations. 
    You can also see the logs being received, the connector status, and other details on the integration tile:

Troubleshooting tip

Scenario: The configured integration is showing a disconnected state

Solution: The server on which the connector is downloaded is down or it is not sending the heartbeats. Go to your virtual machine and ensure that the docker container is up and running.


To verify log collection configuration

To verify whether log collection has started, click Log Explorer > Discover. If the log format expression that you have configured is incorrect, in the /opt/bmc/connectors/<connector_name>/logs/fluent.log file, you get Response: 201.

To verify whether the parameters are correctly populated in the fluentd pipeline, go to /opt/bmc/connectors/<connector_name> /<integration_name>/pipeline. Open the file_log_pipeline.conf file by running the cat file_log_pipeline.conf command.

Related topic

Troubleshooting log collection by using Fluentd

Was this page helpful? Yes No Submitting... Thank you
Last modified by Swati Malhotra on Feb 07, 2022

Comments

Log in or register to comment.

Collecting logs
Collecting logs by using Logstash and Filebeat
© Copyright 2021-2022 BMC Software, Inc.
Legal notices

Powered by Atlassian Confluence and Scroll Viewport

© Copyright 2005-2023 BMC Software, Inc. Use of this site signifies your acceptance of BMC’s Terms of Use . BMC, the BMC logo, and other BMC marks are assets of BMC Software, Inc. These trademarks are registered and may be registered in the U.S. and in other countries.