Troubleshooting filtering and parsing logs

The multiline parser cannot be configured during the edit of a collection policy

Issue symptom

While editing a collection policy, you cannot configure a multiline parser from the Create Parsing Rule page.

Issue scope

This issue occurs when you edit a collection policy that you created in BMC Helix Log Analytics version 24.1.02 or earlier.

Resolution

Configure the multiline parser while configuring collection logs. Starting with release 24.2, you can use the Customize Log Data page to configure a multiline parser.

To configure the multiline parser while editing a collection policy that is created in version 24.1.02 or earlier :

  1. In the Parsing Rule list, select None to remove the existing multiline parser.
  2. In the Configuration section, click Configure against the existing log collection details.
  3. Use the Customize Log Data page to reconfigure the multiline parser.
  4. Save the policy.

Errors in the connector log file

Issue scope

The connector log file displays the following error message:

Got incomplete line before first line from <log file path>

Resolution

Make sure that the regular expression for the Pattern to Recognize First Line field in the collection policy is correct and matches the first line of multiline message. Perform the following steps to resolve this issue:

  1. In BMC Helix Log Analytics, go to Collection > Collection Policies.
  2. From the action menu action_menu.png of the affected collection policy, click Edit.
  3. In the Configuration section, click Configure.
  4. In the Customize Log Data page, make sure that the value in the Pattern to Recognize First Line field is correct.
  5. Make the required changes and save the policy.

Issue scope

The connector log file displays the following error message:

:ConfigError error="Invalid regexp '<regular expression>': No named captures

Resolution

Make sure that the regular expression for the Pattern to Recognize Subsequent Lines field in the collection policy is correct and must have one name capturing expression. Perform the following steps to resolve this issue:

  1. In BMC Helix Log Analytics, go to Collection > Collection Policies.
  2. From the action menu action_menu.pngof the affected collection policy, click Edit.
  3. In the Configuration section, click Configure.
  4. In the Customize Log Data page, make sure that the value in the Pattern to Recognize Subsequent Lines  field is correct.
  5. Make the required changes and save the policy.

Issue scope

The connector log file displays the following error message:

dump an error event: error_class=ArgumentError error="<field name> does not exist

Resolution

Make sure that the parsing rule is configured correctly. Perform the following steps to resolve this issue:

  1. In BMC Helix Log Analytics, go to Collection > Parsing rules.
  2. From the action menu action_menu.pngof the parsing rule that you have used in the affected collection policy, click Edit.
  3. In the Rule Configuration section, make sure that the value in the Record Field to be Parsed field is correct.
  4. Make the required changes and save the parsing rule.

Issue scope

The connector log file displays the following error message:

dump an error event: error_class=Fluent::Plugin::Parser::ParserError error=pattern not matched with data

Resolution

Make sure that the regular expression in the parsing rule matches with the message in the log file. Perform the following steps to resolve this issue:

  1. In BMC Helix Log Analytics, go to Collection > Parsing rules.
  2. From the action menu action_menu.pngof the parsing rule that you have used in the affected collection policy, click Edit.
  3. In the Rule Configuration section, make sure that the value in the Expression field is correct.
  4. Make the required changes and save the parsing rule.

Multiline logs appear as individual records in the Explorer

Issue scope

In the Explorer, multiline logs appear as individual log records.

Resolution

Make sure that the multiline parser configuration is correct. Perform the following steps to resolve this issue:

  1. In BMC Helix Log Analytics, go to Collection > Collection Policies.
  2. From the action menu action_menu.pngof the affected collection policy, click Edit.
  3. In the Configuration section, click Configure.
  4. In the Customize Log Data page, make sure that the values in the following fields are correct:
    • Parser
    • Pattern to Recognize First Line
    • Pattern to Recognize Subsequent Lines
  5. Make the required changes and save the policy.

Even after configuring filters, unwanted log messages are visible in the Explorer

Issue scope

Even if you have configured prefilters and/or filters in the collection policy, you can see unwanted log messages in the Explorer.

Resolution

Make sure that the prefilter or filter rules are configured correctly in the collection policy. Perform the following steps to resolve this issue:

  1. In BMC Helix Log Analytics, go to Collection > Filtering rules.
  2. From the action menu action_menu.pngof the filtering rule that you have used in the affected collection policy, click Edit.
  3. In the Rule Configuration section, make sure that the values in the Key and Pattern fields are correct.
  4. Make the required changes and save the filter rule.

Fields are not extracted from log messages

Issue scope

Fields are not extracted from the log messages, and they are visible as a single record in the Explorer.

Resolution

Make sure that you have configured a parsing rule and selected it in the collection policy. 

To learn how to create a parsing rule, see Creating a parsing rule.

To learn how to select a parsing rule in a collection policy, see Creating collection policies.

Error in connector log file

Issue scope

There is an error "dump an error event: error_class=Fluent::Plugin::Parser::ParserError error=pattern not matched with data" in connector log file.

Resolution

Make sure that regular expression provided in the parsing rule matches with the message present in the log file.

No logs are ingested

Issue scope

This issue might occur because incorrect key or pattern are configured in the parsing or filtering rule. This issue might also occur if the format expression that you configured in the parsing rule does not match the the logs that you are collecting.

Resolution

  • Correct the key and pattern configured in the parsing and filtering rules.
  • Make sure that the format expression configured in the parsing rule matches the logs that you are collecting.

Logs are not being filtered while collection

Resolution

Check if key and pattern are configured correctly for Regexp and Exclude for the grep filter.

 

Tip: For faster searching, add an asterisk to the end of your partial query. Example: cert*