Collecting application logs

To start collecting logs, you add all the collection-related configurations and other details to a collection policy. You save time by reusing these configurations in multiple collection policies.

The following image shows how logs are collected by using the Linux (RHEL/CentOS) and Windows connectors:

Before you begin

• Make sure that you have downloaded and installed a connector. For more information, see Installing-and-managing-connectors.
• Create a parsing rule. For more information, see Creating-a-parsing-rule.
• Create a filtering rule. For more information, see Creating-a-filtering-rule.
  If you plan to filter logs before they are parsed, make sure that you add a separate rule for the pre-filtering process in addition to the filtering process. By prefiltering logs, you limit the number of logs that are parsed and thereby enhance system performance.

To collect application logs

In BMC Helix Log Analytics, use the Collection > Collection Policies > Create button and perform the following steps:

1. Add the policy information by performing the following steps:
   1. Enter a unique name and description.
   2. From the Connector type list, select the connector type that you have installed.
2. Configure the connector by performing the following steps:
   1. In the Connector Type field, select the connector type that you want to configure.

   2. In Connector Selection Criteria, create the connector selection criteria to identify connectors for collection.
      When you click in the box, you are prompted to make a selection. Each time you make a selection, you are progressively prompted to make another selection. 
      The selection criteria consist of an opening parenthesis, followed by the slot name, the operator, the slot value (which can be a string based on the type of slot selected), and the closing parenthesis. You can optionally select the logical operator AND or OR to add additional conditions. Specifying the opening and closing parentheses is optional.
      The connector fields available to create the selection criteria are status, name, version, host_name, ip, and tags. 

      Important

      The values that you enter for a field in the selection criteria are case-sensitive. For example, if the host name is WebServer.example.com, add the selection criteria as ( host_name Equals WebServer.example.com ). If you enter ( host_name Equals webserver.example.com ), the connector is not selected. To add case-insensitive values, use the Equals ignore case operator.

3. Configure log collection by performing the following steps:
   1. In the Configuration step, click Configure.

   2. In the Log Collection File Path field, enter the path of the log files that you want to collect.
      For example: /opt/tomcat/apache.log (Linux) or C:/app1/logs/app.log (Windows).
      When you enter folder locations, sub-folders and files present in the folder are shown. 

      Best practices

      • Enter only directory paths and an absolute file name with the path.
      • Separate multiple entries with a comma.
      • Make sure that all log files have the same format so that a single parsing rule parses all the logs.
      • To collect logs from Windows-based applications, ensure that you enter the path of the computer where you have installed the connector.
      • In Windows file path, replace back slashes (\) with forward slashes (/). For example, if your file path is C:\app1\logs\app.log, change it to C:/app1/logs/app.log.  
      • If your log files are created on the basis of size, enter the name of the file where the latest logs are written and do not enter * in the file path.
   3. (Optional) If you have entered a path with multiple folders and you want to exclude some folders from collection, in the Exclude Paths field, remove those folders.
      For example, you have entered the log collection path as /opt/bmc/connectors/<connector_name>/logs/applicationLogs and this folder contains the following folders: app1, app2, app3. The app1, app2, and app3 folders are shown in the Exclude Paths field. To prevent log collection from the app3 folder, remove the app3 folder from the field.
   4. (Optional) If you want to start collecting logs from the beginning of the file, select the Read Files from Beginning check box. 
      Else, log collection starts when you enable the policy.
      By default, all logs present in a log file are collected. They include the logs that are read for the first time and logs from the last read line. If your log files are created (and rotated) on the basis of time, you have given wildcard (*) in the file path, and the Read Files from Beginning check box is cleared, all logs created after the policy is enabled are collected.
   5. (Optional) Select the Open File on Every Update check box to enable the Open File on Every Update parameter.
      When some applications stop updating their log files, the monitoring agents add a permanent read lock to the application log files. To avoid this issue, enable the Open File on Every Update parameter so that the connector opens the log files when they are updated. By default, this parameter is disabled.

1. 1. (Optional) If you want to start collecting and storing multiple lines of log data in a single log entry, select the Multiline Parser check box and perform the following steps:
      1. From the Parser list, select Java multiline to parse the multiline logs generated by the standard Java application. The default parser configurations have standard values that can be modified.
      2. From the Parser list, select Custom to parse the multiline logs as per your customized expressions for multiline parsing.
   2. Click Save.
   3. In the Tags field, enter the tags to identify the policy with the collected logs.
      The values that you enter in this field are added to the bmc_tags field that is present in the collected logs. You can use the field or tags to search and analyze logs in Explorer.
   4. In Fields, enter the custom information that you want to add to collected logs in the form of key-value pairs.
      Use these fields to search and analyze the logs in Explorer. For example, Key: applicationContext; Value: Apache. Use applicationContext:Apache as a search string to search and analyze the collected logs.

2. (Optional) From the Pre-filtering Rule list, select a filtering rule to help you remove the unwanted log messages before parsing the logs.
   If you don't filter the log data before parsing, all the log data will be processed for parsing.

   Click here to see a scenario that explains the benefit of prefiltering logs

   Example

   Sarah is a tenant administrator at Apex Global, which uses BMC Helix Log Analytics for collecting and analyzing logs. Apex Global parses all messages generated for file type logs. The system performance is affected by the amount of logs generated. Apex Global wants to enhance system performance for file type logs. How can they achieve this?

   They can prefilter file type logs before the logs are parsed. This makes sure that only the required messages are parsed and filtered further, resulting in enhanced system performance.

3. (Optional) From the Parsing Rule list, select a parsing rule to convert raw log data into key-value pairs, making it easier to search, query, and analyze. 
   You cannot configure a multiline parser while creating or using a parsing rule. To configure a multiline parser, see step 3e.
   If you have not created a parsing rule, see Creating-a-parsing-rule for instructions.

1. (Optional) From the Filtering Rule list, select a filtering rule to filter parsed messages.
   If you don't filter the logs, all the logs are processed further. If you have not created a filtering rule, see Creating-a-filtering-rule for instructions.
2. (Optional) From the User group list, select one or more user groups to assign to the collection policy.
   Users associated with this user group can see the data collected by this collection policy. If you don't select the user group, log data will be visible to all the users.
3. Enable and save the policy by performing the following steps:
   1. To start collecting logs, select the Enable Collection Policy check box.
      You can choose to enable the collection policy later.
   2. Click Save.
      The created policy is shown on the Collection Policies page. Use the Actions menu to edit, enable (or disable), and delete the policy.

To verify log collection

1. Click the Explorer tab.
2. Search the logs based on a unique field value.
   For example, you can use the tags that you have added to the collection policy. Let's say you added the tag apache_logs. Search for logs by using bmc_tags:apache_logs or applicationContext:Apache.

Where to go from here

Configuring-logs

Generating-alerts-from-logs

Deriving-insights-from-logs