Collecting logs by using the command line interface

The following video (3:32) illustrates the configurations required to collect logs.

https://youtu.be/5dXTTZcvth0

Before you begin

To prepare to configure log collection, perform the following steps:

• Generate the API key from BMC Helix Portal or BMC Helix Operations Management by performing one of the following steps:
  • Open BMC Helix Portal and do the following:
    1. Go to User access > Users and keys.
    2. In the Access keys tab, click the Actions menu of a key and select Key details.
    3. On the Key details page, click Copy key details.
    4. Paste the key in a text file.
  • Open BMC Helix Operations Management and do the following:
    1. Go to Administration > Repository.
    2. On the Repository page, click Copy API key.
    3. Paste the API key in a text file.

• Download and install Beats on the computers from where you want to collect logs.

• Download and install Logstash​​​​​

To configure Logstash

For detailed information about the files used in the configurations, see Logstash documentation.

1. Configure Logstash to accept data from Beats
   1. From the Logstash installation folder, open the config\logstash-sample.conf file.
      If you are configuring Logstash by using RPM on Linux operating systems, copy the /etc/logstash/logstash-sample.conf file to the /etc/logstash/conf.d folder and then open it.

   2. In the input plugin, enter the port number using which Beats send data to Logstash.

      input{
         beats{
              port=><port number (for example 5044)>
                  }
               }

      Important

      Ensure that the port is open on the computer where Logstash is installed.

2. Configure Logstash to send the collected logs to the REST endpoint by entering the following details to the output plugin in the config\logstash-sample.conf file.
   In Linux environments, after updating the logstash-sample.conf file, move it to the /etc/logstash/conf.d folder.

   Important
   Use the following format for the API key that is generated from Helix Portal or BMC Helix Operations Management for successful authentication.
   Tenant ID::Access Key::Secret key
   1654181562::36X49Z3E0XPVWXKYUIMJ94HKN7XSKW::8uhbxcEScNKYz1eYhlrjZgcudlwSLJSYFdqZLwjglu9oVj6cQM

   output{
   http
   {
   url=>"https://<Tenant URL provided by BMC>/log-service/api/v1.0/logs"
   http_method=>"post"
   content_type=>"application/json"
   format=>"json_batch"
   retry_failed=>false
   http_compression=>true
   headers => {
   "Content-Type" => "application/json"
   "Authorization" => "apiKey <API key of tenant>"}
   }
   }

3. (Optional) Add a structure to the logs - field:value pattern by using the grok plugin in the config\logstash-sample.conf file.
    

   Example...

   input {

       file {

           type => "apachelog"

           path => ["C:/logs/apachelogs.log"]

           start_position => "beginning"

            }

            }

    filter {

        if [type] == "apachelog" {

           grok { match => {"message" => '%{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-)'}

            }

         }

       }

4. (Optional) If you want convert the time zone of the collected logs, use the date filter.

   Example

   date {

              match => ["tmpstamp", "MMM dd, yyyy hh:mm:ss a"]

              target => "@timestamp"

              locale => "en_US"

              timezone => "UTC"

         }

5. Start Logstash by running the following command:

   - bin/logstash

   For example for Windows, the command would be:

   - bin/logstash -f config/logstash-sample.conf

   Important

   If you have enabled a firewall in your environment, open the outbound HTTPS port 443.

To configure Beats

1. Configure Beats to communicate with Logstash by updating the filebeat.yml and winlogbeat.yml files, available in the installed Beats installation folder. Mark the output.elasticsearch plugin as a comment and uncomment the output.logstash plugin.
2. To send data to Logstash as, add the Logstash communication port:
   output.logstash:
   hosts: <logstash computer>:<port to communicate with logstash>
3. In the type plugin, change the value of enabled to true.
4. Configure log sources by adding the path to the filebeat.yml and winlogbeat.yml files and start Beats.
   type: log
   enabled: true
   paths:
       - <path of log source. For example, C:\Program Files\Apache\Logs or /var/log/message>

5. To ensure that you collect meaningful logs only, use include.

   Note

   The following settings in the .yml files will be ineffective:

   • Elasticsearch template setting
   • Dashboards (these settings are for Kibana Dashboards)
   • Kibana (these settings are for dashboards loaded via the Kibana API)

To reduce the size of the collected logs

While collecting logs, Logstash and Filebeat add metadata to the logs that are saved with the collected logs. Thus, the size of the collected logs increases. Here is an example of an application logs collected by using Logstash and Filebeat.

In this example, metadata added by Logstash is:"event.original": "Aug 10, 2022 11:37:14 AM com.bmc.ola.collection.collector.BaseCollector execute():410 \nINFO: Collector=ITDA Collection Metrics_Collection Metrics_<hostname>.bmc.com, CollectionPollId=48,  2 events read and sent for indexing.",

The remaining metadata is added by Filebeat. This metadata increases the size of the collected logs and your storage space is utilized more than necessary.

Perform the following steps to reduce the size of the collected logs:

1. In the filebeat.yml file, add # before the metadata fields that you want to prevent from being added to the logs.

2. To remove the metadata from Logstash, in the pipeline.xxx file, enter the following:

   mutate
        { remove_field => ["[event][original]"] } 
   

To verify the log collection

Navigate to  > Discover.

Troubleshooting log collection

The following table lists the possible scenarios that you might run into while collecting logs and the steps that you can perform to troubleshoot the issue: