Troubleshooting log collection from files

Log events are not generated in BMC Helix Operations Management

Issue symptom

This issue might occur because of a mismatch between the timezones of application logs and the connector host.

Issue scope

Log events are not generated in BMC Helix Operations Management even though the log count exceeded the defined threshold.

Example

The application logs are generated in the GMT +02:00 timezone, and the host where the connector is installed is in the GMT +03:30 timezone.

The 1 hour 30 minutes difference causes the connector to parse logs with the incorrect time stamp, treating the logs as older data based on epoch time. The logs could not meet the alert conditions because they appeared to fall outside the monitoring time.

Resolution

To resolve this issue, add the timezone parameter in the file_log_pipeline.conf file at the following location:

  • Windows
    <Install_Dir>\opt\td-agent\etc\data\<Policy_ID_Random_Number>\pipeline\file_log_pipeline.conf
  • Linux
    /opt/td-agent/etc/data/<Policy_ID_Random_Number>/pipeline/file_log_pipeline.conf

Perform the following steps to add the timezone parameter:

  1. In the file_log_pipeline.conf file, locate the first occurrence of the parse parameter.
  2. After the time_format parameter, add the timezone parameter. 
  3. Specify the application timezone in the timezone parameter of the file_log_pipeline.conf file.
  4. Stop and restart the Fluentd connector service to apply the changes.

Important

If you update the collection policy after changing the timezone in the file_log_pipeline.conf file, it will overwrite your timezone changes. After updating the collection policy, you must reapply the timezone parameter to retain the changes.

Example: The timezone parameter in multiline and regex configurations

  • Multiline

    <parse>
        @type multiline
    format_firstline /\d{4}\d{2} \d{2}:\d{2}:\d{2}/
    format1 /(?<time>\d{4}\d{2} \d{2}:\d{2}:\d{2})\s+(?<msg1>.*)/
    time_format %F %T
    timezone +0200
    </parse>

  • Regex

    <parse>
      @type regexp
      expression /(?<time>\d{4}\d{2}\d{2} \d{2}:\d{2}:\d{2})\s+(?<message123>.*)/
       time_format %F %T
       timezone +0200
    </parse>


Invalid format expression

Resolution

Validate the expression at Rubular or Fluentular.


Configured integration is in a disconnected state

Resolution

Ensure that the docker container is up and running.


Verify parameters are correctly populated in the fluentd pipeline

Resolution

Verify that the configurations entered on the UI are saved successfully in the configuration file.

  • Docker connector: Go to /opt/bmc/connectors/<connector_name>/<integration_name>/pipeline. Open the file_log_pipeline.conf file by running the cat file_log_pipeline.conf command.
  • Windows connector: Go to <connector download folder>\opt\td-agent\file_log_pipeline.conf. Verify the configurations in this file.


config error .... error="Duplicate key: <key_Name>" in connector logs

Issue symptom

Integration is shown in the Configured state, but this error is shown in the connector td-agent logs.

Issue scope

Issue might occur with all connectors and integrations.

Resolution

In the integration configuration for log collection, when you configure the grep filter, ensure that for the same directive, keys are unique. For example, do not repeat the msgType key for multiple Regexp directives.

Then, verify that the configurations entered on the UI are saved successfully in the configuration file:

  • Docker connector: Go to /opt/bmc/connectors/<connector_name>/<integration_name>/pipeline. Open the file_log_pipeline.conf file by running the cat file_log_pipeline.conf command.
  • Windows connector: Go to <connector download folder>\opt\td-agent\file_log_pipeline.conf. Verify the configurations in this file.
  • Linux connector: Open the file_log_pipeline.conf file located in the opt/td-agent/etc/data/<Integration_ID>/pipeline/ folder and modify the the file by removing the duplicate entry. Then, restart the td-agent service.


Unable to run the batch script to install the Windows connector

Resolution

Confirm that you have the permission to execute the script. Ensure that you launch the command prompt with the Run as Administrator option.


Unable to run the shell script to install the Linux connector

Resolution

Ensure that you have the root permissions to run a script file.


Batch script to install the Windows connector failed

Resolution

Check the logs for error. The installation script execution logs are displayed on the command prompt and are also added to the installation log file, which is located at <folder where you extracted the downloaded zip file>\BMC-DevTools\opt\<connector_Name> with name <connector_name>_install.log.


Path too long error while extracting the Windows connector .zip file

Resolution

You get this error when you use Windows out-of-the-box utility to extract a .zip file. Resolve the error by following either of the following:

  • Ensure that the destination folder name is short.
  • Use 7-zip or any other utility to extract the file.



System cannot find the specified path error while installing the Windows connector

Resolution

You get the BMC-DevTools folder after extracting the tdc-connector-windows-<build_number>.zip file. The BMC-DevTools folder and the .bat file to install the connector must be at the same level in the folder hierarchy. For example, if you extract the downloaded connector file in the C:\Connector folder, ensure that the downloaded .bat file is also present inside the Connector folder.

Windows_connector_location.png


Not able to execute install script error while installing the Linux connector

Resolution

Ensure that you have extracted the .tar.gz file in the /opt directory.


/opt/td-agent already exist error while running the Linux connector install script

Resolution

On the Linux server, either the connector is already installed or a td-agent is currently running. Remove the existing installation and run install script again.


/opt/bmc-devtools/td-agent does not exist error while running the Linux connector install script

Resolution

Extract the installation .tar.gz file in the /opt directory only.


Permission denied error

When you are installing the Windows connector, you get the permission denied error even though you are installing as an administrator.

Issue symptom

Windows connector installation fails.

Issue scope

Issue might occur with Windows connector installation.

Resolution

Make sure that there are no spaces in the name of the folder where you are installing the connector.


Connector state is suspended

Issue symptom

The following error is shown in the fluentd.log/td-agent.log file (located at /opt/td-agent for Linux and <connector download folder>\BMC-DevTools\opt\td-agent for Windows):

config error file="/fluentd/etc/fluent.conf" error_class=Fluent::ConfigError error="format should be Regexp //

Issue scope

This issue might occur in file-based log collection if you update and enter incorrect regular expression in Apache, Apache Error, Nginx, Regexp, or Java Multiline formats in the parsing rule. 

Resolution

  1. Open the log file:
    • Windows connector: <installation-directory>\BMC-DevTools\opt\td-agent\etc\data\<integration_id>\pipeline\file_log_pipeline.conf
    • Linux connector: /opt/td-agent/etc/data/<integration_id>/pipeline/file_log_pipeline.conf
  2. Search for the following line in file_log_pipeline.conf file:
    @type <format>
    For example, @type regexp
  3. Make sure that the regular expression is enclosed in back slashes (/) and the syntax of the expression is correct:
    expression /<regex_expression>/
  4. Restart the connector service.






 

Tip: For faster searching, add an asterisk to the end of your partial query. Example: cert*