Deriving insight by analyzing logs

• Copy the API key of your BMC Helix Operations Management tenant and paste it in a text file. In BMC Helix Operations Management, go to Administration > Repository and click Copy API key.
• Download and install Beats on the computers from where you want to collect logs.
• Download and install Logstash

For detailed information about the files used in the configurations, see Logstash documentation .

1. Configure Logstash to accept data from Beats
   1. From the Logstash installation folder, open the config\logstash-sample.conf file.
      If you are configuring Logstash by using RPM on Linux operating systems, copy the /etc/logstash/logstash-sample.conf file to the /etc/logstash/conf.d folder and then open it.

   2. In the input plugin, enter the port number using which Beats send data to Logstash.

      input{
       beats{
          port=><port number (for example 5044)>
            }
           }

      
      Note: Ensure that the port is open on the computer where Logstash is installed.

2. Configure Logstash to send the collected logs to the REST endpoint by entering the following details to the output plugin in the config\logstash-sample.conf file.
   In Linux environments, after updating the logstash-sample.conf file, move it to the /etc/logstash/conf.d folder.

   output{
      http
         {
         url=>"https://<DNS Name that is provided by BMC>/log-service/api/v1.0/logs"
         http_method=>"post"
         content_type=>"application/json"
         format=>"json_batch"
         retry_failed=>false
         http_compression=>true
         headers => {
         "Content-Type" => "application/json"
         "Authorization" => "apiKey <API key of  tenant>"}
         }
         }

3. (Optional) Add a structure to the logs - field:value pattern by using the grok plugin in the config\logstash-sample.conf file.

    Example...

   input {
     file {
       type => "apachelog"
       path => ["C:/logs/apachelogs.log"]
       start_position => "beginning"
     }
   }
   filter {
   if [type] == "apachelog" {
           grok {
               match => {"message" => '%{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-)'}
           }
       }
   }

4. Start Logstash by running the following command - bin/logstash
   For example for Windows - bin/logstash -f config/logstash-sample.conf.
   Note: If you have enabled firewall in your environment, open the outbound https port 443.

1. Configure Beats to communicate with Logstash by updating the filebeat.yml and winlogbeat.yml files, available in the installed Beats installation folder. Mark the output.elasticsearch plugin as a comment and uncomment the output.logstash plugin.

2. To send data to Logstash as, add the Logstash communication port:

   output.logstash:
   hosts: <logstash computer>:<port to communicate with logstash>

3. In the type plugin, change the value of enabled to true.

4. Configure log sources by adding the path to the filebeat.yml and winlogbeat.yml files and start Beats.

   type: log
   enabled: true
   paths:
     - <path of log source. For example, C:\Program Files\Apache\Logs or /var/log/message>

   
   

5. To ensure that you collect meaningful logs only, use the include

Navigate to BMC Helix Portal > BMC Helix Log Analytics > Discover.

Note: A default index pattern is created. Do not delete it. If you create a new index pattern, ensure that the name begins with - log-xx_r14_v1. The xx value is available in the matches suggested to you.

1. To view the collected logs, go to BMC Helix Portal > BMC Helix Log Analytics > Discover.
   All the collected logs are displayed here.
2. To filter data, select the required fields from the Available fields list.
   In place of the data type icon of a field, if you see the '?' sign, refresh the field list on the index pattern page (Management > Index pattern > index pattern name).
3. Select a date range to filter data.
4. (Optional) You can also use Kibana queries to filter data. For more information, see  Kibana documentation .
5. To save the filter criteria for later use, save it as saved search.
   You can use the saved search to create a visualization.
   For information about log retention and storage limit, see BMC Helix Subscriber Information .

If you have a search criteria that you think might occur repeatedly, you can save it and add it to a dashboard as a visualization.

1. Click Visualize > Create new visualization.
2. Select the type of visualization that you want to use.
   For example, an area chart.
3. Select the search that you have saved.
4. Apply additional filters to the data and save the visualization.
5. To add the visualization to a dashboard:
   1. Click Dashboard.
   2. You can create a new dashboard or edit an existing one.
   3. Click Add and select the visualization.

When you see an error in the collected logs and want to be notified if it occurs again, configure an alert. When the error occurs again, an event is generated in BMC Helix Operations Management. You can configure a maximum of 20 alerts.

On the Alerts > Create an alert page, click Create new alert and do the following:

1. Specify a unique name.
2. Configure the query based on which alert will be generated.

3. Configure the conditions that define when the event is triggered.
   
   

   Best practice

   Run the query for last 5 minutes as the frequency to create alerts is 5 minutes in BMC Helix Log Analytics.

4. Select the severity level of the alert that is generated.
5. Enter the message of the event and its additional details.
6. Save the alert.