Collecting logs from files

Collect logs by using docker-based or Windows-based Fluentd connector that you can configure by using the Collect logs from file option added to BMC Helix Developer Tools. Set up parsing based on the log formats present in your log files and also set up filtering rules to include or exclude logs.

All log file extensions are supported. Collect log files of any application, provided the files are text based.

The following connectors are available to collect logs from files:

• Docker connector: collects logs from the Linux container and cloud environments. For more information, see Downloading and installing Docker connectors.
• Windows connector: collects logs from the Windows-based applications. It is a standalone Fluentd connector. For more information, see Downloading and installing Windows connectors.

The following image shows how logs are collected by using Docker connector:

The following image shows how logs are collected by using the Windows connector:

Before you begin

Perform the following actions before collecting logs:

• Download and install the connector. Download dockerized connector to collect logs from linux docker environment and Windows connector to collect logs from Windows-based applications.
• In Linux environment, if the logs are located outside the container where the connector is downloaded and installed, create mount points for such directories.

To collect logs from files

1. In BMC Helix Developer Tools, open the Available integrations tab if it is not open by default.

2. On the Collect Logs from File tile, click Configure.
   
   
   On this tile, the build number of the product and the name of the entity that made the product available for configuration are displayed. In this example, the product is made available by BMC.
3. Enter the integration name.
   
4. Select the connector that you downloaded and saved.
   If no connector is available, click Add Connector to download and install a connector. 
5. In the Customize Entity Configuration section, click Configure.

6. In the Log Collection File Path field, enter the file locations to collect logs.
   For best results:

   • Enter only directory paths and an absolute file name with the path.

   • Separate multiple entries with a comma.

   • Ensure that all log files have the same format. Supported formats are available in the Format field.

   • To collect logs from Windows-based applications, ensure that you enter the path of the computer where you have installed the connector.

   • In Windows file path, replace back slashes (\) with forward slashes (/). For example, if your file path is C:\app1\logs\app.log, change it to C:/app1/logs/app.log.  
     
     

      File location examples in Linux

     Example 1: Log file is present in the logs folder of your connector (/opt/bmc/connectors/<connector_name>/logs/<log_file_name>), enter /fluentd/log/<log_file_name>. Replace /opt/bmc/connectors/<connector_name>/logs with fluentd/log.

     
     Example 2: Log file is present in a folder (other than the connector logs folder) on the same computer (or virtual machine) where the connector is installed. Mount the folder with log files to the connector log folder (such as, /opt/bmc/connectors/<connector_name>/logs). Then, enter /fluentd/log/<log_file_name>.

     
     Example 3: Log file location and connector are on different computers (or virtual machines). Mount the remote folder with log files to the connector log folder (such as, /opt/bmc/connectors/<connector_name>/logs). Then, enter /fluentd/log/<log_file_name>. 

7. (Optional) If you have entered a path with multiple folders and you want to exclude some folders from collection, in the Exclude Paths field, remove those folders.
   For example, you have entered the log collection path as /opt/bmc/connectors/<connector_name>/logs/applicationLogs and this folder contains the following folders: app1, app2, app3. The app1, app2, and app3 folders are shown in the Exclude Paths field. To prevent log collection from the app3 folder, remove the app3 folder from the field.  

8. From the Format field, select the format present in your logs and perform the appropriate steps to filter the logs.
   For more information, see Logs parsing and filtering.

   To verify the expression, visit rubular or  fluentular .

9. In the Tags field, enter the tags to identify logs of the specified files.

10. Save the entity configuration and then save the integration.
    You can view the added configuration by clicking Integrations and then Configured Integrations. 
    You can also see the status of logs being received, the connector status, and other details on the integration tile:
    

To verify log collection configuration

To verify whether log collection has started, in BMC Helix Log Analytics, select Log Explorer > Discover. 

In the /opt/bmc/connectors/<connector_name>/logs/fluent.log (Docker connector) or td-agent.log (Windows connector) file, you get the following entries:

"StatusCode":200
"StatusMessage":"QUEUED"

The log collection configurations are correct and logs are being collected.

The td-agent.log file is available in the folder where you extracted the connector; for example, C:\connector_installation\BMC-DevTools\opt\td-agent\td-agent.log, where connector_installation is the folder where you have extracted the connector ZIP file.

Related topic

Troubleshooting log collection from files