Collecting Linux syslog records

As an administrator, You want to collect syslog records and analyze them to troubleshoot issues.

Use the open source Fluentd for Linux to collect the syslog messages. Here are the high-level steps involved:

Before you begin

1. Get the API key of your tenant by performing the following steps:
   1. Go to BMC Helix Operations Management.
   2. Click Administration > Repository.
   3. Click Copy API Key and paste it in a notepad.
2. Copy the URL of your BMC Helix Log Analytics tenant and paste it in the notepad.
   For example, https://tenanttrial.bmc.com.

Step 1: To download and install Fluentd for Linux

Download and install the open-source Fluentd for Linux. For more information, see Fluentd installation documentation.

When the Fluentd is installed, it runs as the td-agent.service.

Step 2: To add port to the rsyslog.conf file

We will send the syslog messages to a port from where Fluentd will collect them by using the syslog plugin. We add the port to the rsyslog.conf file.

1. Open the rsyslog.conf file.
   Default location in Red Hat Enterprise Linux (RHEL): /etc/rsyslog.conf.

2. Add the default port (5140):

   # Send log messages to Fluentd
   *.* @127.0.0.1:5140
3. Restart the rsyslog service:
   systemctl restart rsyslog

Step 3: To add the syslog plugin

The syslog plugin collects the syslog records.

1. Open the td-agent.conf (in RHEL) or fluent.conf (in other Linux versions) file.
   In RHEL, default location is /etc/td-agent/td-agent.conf.

2. Add the source plugin of type syslog.

   Sample source plugin of type syslog

   <source>

   @type syslog
     tag rhelSystem_syslog_LA_ingest
     
     # other parameters

   </source>

3. (Optional) Add the filter plugin.
   Use this plugin to enrich logs with meaningful information. For example, add the hostname to logs that will help you in filtering logs in the Explorer tab.

   Sample filter plugin

   <filter rhelSystem_syslog_LA_ingest*.*.*>
     @type record_transformer
     <record>
       hostname ${hostname}
       Apps SSHD
     </record>
   </filter>

4. To send the collected logs to BMC Helix Log Analytics, add the match plugin.
   You will need the API key and URL of your BMC Helix Log Analytics tenant.

   Sample match plugin

   <match rhelSystem_syslog_LA_ingest*.*.*>
       @type http
       @id OOB_TD-agent-RHELSysLog_logingest
       endpoint https://<url_for_the_tenant>/log-service/api/v1.0/logs
       headers {"Authorization":"apiKey <apiKeyoftenant>","Content-Type":"application/json"}
       json_array true
       <format>
         @type json
       </format>
   </match>

   Notes

   • The tag specified in source plugin is generated by the tag parameter, facility code, and priority level as tag = "#{@tag}.#{facility}.#{priority}". Set the facility code and priority levels in the facility_key and severity_key parameters. For more information, see Fluentd documentation.  

   • In this example, the match plugin is set with <tag_name>*.*.* to get default facility code and priority level. You can set it as per your requirement.
5. Save the file.
   To troubleshoot the issues with the td-agent configuration, refer to the td-agent.log file in the /var/log/td-agent/ directory.
6. Restart the td-agent service:
   systemctl restart td-agent.service

Step 4: To view the collected logs

1. In BMC Helix Log Analytics, click the Explorer tab.
2. Search the syslog records logs by using a key:value pair present in the records.
   For example, search the logs by using the hostname that you added to the filter plugin.