Collecting application logs

To start collecting logs, you add all the collection-related configurations and other details to a collection policy. You save time by reusing these configurations in multiple collection policies.

The following image shows how logs are collected by using the Linux (RHEL/CentOS) and Windows connectors:

Apps_log_collection.jpg


Before you begin

  • Make sure that you have downloaded and installed a connector. For more information, see Installing-and-managing-connectors.
  • Create a parsing rule. For more information, see Creating-a-parsing-rule .
  • Create a filtering rule. For more information, see Creating-a-filtering-rule.
    If you plan to filter logs before they are parsed, make sure that you add a separate rule for the pre-filtering process in addition to the filtering process. By prefiltering logs, you limit the number of logs that are parsed and thereby enhance system performance.


To collect application logs

In BMC Helix Log Analytics, use the Collection > Collection Policies > Create button and perform the following steps:

  1. Add the policy information by performing the following steps:
    1. Enter a unique name and description.
    2. From the Connector type list, select the connector type that you have installed.
  2. Configure the connector by performing the following steps:
    1. In the Connector Type field, select the connector type that you want to configure.

    2. In Connector Selection Criteria, create the connector selection criteria to identify connectors for collection .
       When you click in the box , you are prompted to make a selection. Each time you make a selection, you are progressively prompted to make another selection. 
       The selection criteria consist of an opening parenthesis, followed by the slot name, the operator, the slot value (which can be a string based on the type of slot selected), and the closing parenthesis. You can optionally select the logical operator AND or OR to add additional conditions. Specifying the opening and closing parentheses is optional.
       The connector fields available to create the selection criteria are status, name, version, host_name, ip, and tags.  

      Important

      The values that you enter for a field in the selection criteria are case-sensitive. For example, if the host name is WebServer.example.com, add the selection criteria as ( host_name Equals WebServer.example.com ). If you enter ( host_name Equals webserver.example.com ), the connector is not selected. To add case-insensitive values, use the Equals ignore case operator.

  3. Configure log collection by performing the following steps:
    1. In the Configuration step, click Configure.

    2. In the Log Collection File Path field, enter the path of the log files that you want to collect.
      For example: /opt/tomcat/apache.log (Linux) or C:/app1/logs/app.log (Windows).
      When you enter folder locations, sub-folders and files present in the folder are shown. 

      Best practices

      • Enter only directory paths and an absolute file name with the path.
      • Separate multiple entries with a comma.
      • Make sure that all log files have the same format so that a single parsing rule parses all the logs.
      • To collect logs from Windows-based applications, ensure that you enter the path of the computer where you have installed the connector.
      • In Windows file path, replace back slashes (\) with forward slashes (/). For example, if your file path is C:\app1\logs\app.log, change it to C:/app1/logs/app.log.  
      • If your log files are created on the basis of size, enter the name of the file where the latest logs are written and do not enter * in the file path.
    3. (Optional) If you have entered a path with multiple folders and you want to exclude some folders from collection, in the  Exclude Paths  field, remove those folders.
      For example, you have entered the log collection path as  /opt/bmc/connectors/<connector_name>/logs/applicationLogs  and this folder contains the following folders:  app1, app2, app3. The app1, app2, and app3 folders are shown in the  Exclude Paths  field. To prevent log collection from the  app3 folder, remove the app3 folder from the field.
    4. (Optional) If you want to start collecting logs from the beginning of the file, select the Read Files from Beginning check box. 
      Else, log collection starts when you enable the policy.
             By default, all logs present in a log file are collected. They include the logs that are read for the first time and logs from the last read line. If your log files are created (and rotated) on the basis of time, you have given wildcard (*) in the file path, and the Read Files from Beginning check box is cleared, all logs created after the policy is enabled are collected.
    5. (Optional) Select the Open File on Every Update check box to enable the Open File on Every Update parameter.
      When some applications stop updating their log files, the monitoring agents add a permanent read lock to the application log files. To avoid this issue, enable the Open File on Every Update parameter so that the connector opens the log files when they are updated. By default, this parameter is disabled.
    6. Click Save.
    7. In the Tags field, enter the tags to identify the policy with the collected logs.
      The values that you enter in this field are added to the bmc_tags field that is present in the collected logs. You can use the field or tags to search and analyze logs in Explorer.
    8. In Fields, enter the custom information that you want to add to collected logs in the form of key-value pairs.
      Use these fields to search and analyze the logs in Explorer. For example,  Key: applicationContext;  Value: Apache. Use applicationContext:Apache as a search string to search and analyze the collected logs.

  4. (Optional) From the Pre-filtering Rule list, select a filtering rule to help you remove the unwanted log messages  before parsing the logs.
    If you don't   filter the log data  before parsing , all the log data  will   be   processed for parsing .

    Click here to see a scenario that explains the benefit of prefiltering logs

    Example

    Sarah is a tenant administrator at Apex Global, which uses BMC Helix Log Analytics for collecting and analyzing logs. Apex Global parses all messages  generated   for   file type logs . The   system   performance   is   affected   by   the  amount   of   logs   generated . Apex Global wants to enhance system performance for file type logs. How can they achieve this?

    They can prefilter file type logs before the logs are parsed. This makes sure that only the required messages are parsed and filtered further, resulting in enhanced system performance.

  5. (Optional) From the Parsing Rule 1 list, select a single-line or multiline parsing rule.
    If you select a multiline parsing rule, the     Parsing Rule 2 list is displayed, where you can select a parsing rule other than multiline parsing rule.
    A parsing rule converts raw log data into key-value pairs, making it easy for BMC Helix Log Analytics     to search, query, and analyze logs. With multiline parsing rules, you can collect and store multiple lines of log data in a single log entry.
     If you have not created a parsing rule, see Creating-a-parsing-rule for instructions.
  6. (Optional) From the Filtering Rule list, select a filtering rule to filter parsed messages.
     If you don't filter the logs, all the logs are processed further. If you have not created a filtering rule, see Creating-a-filtering-rule for instructions.
  7. (Optional) From the User group list, select one or more user groups to assign to the collection policy.
     Users associated with this user group can see the data collected by this collection policy. If you don't select the user group, log data will be visible to all the users.
  8. Enable and save the policy by performing the following steps:
    1. To start collecting logs, select the Enable Collection Policy check box.
      You can choose to enable the collection policy later.
    2. Click Save .
      The created policy is shown on the Collection Policies page. Use the Actions menu to edit, enable (or disable), and delete the policy.

Important

The multiline parsing option is removed from the Configuration section on the Collection Policies page. You can create a multiline parsing rule from Collection > Collection Policies > Create Collection Policy > Log Parsing > Parsing Rule 1.


To verify log collection

  1. Go to BMC Helix Log Analytics and click the Explorer tab.
  2. Search the logs based on a unique field value.
    For example, you can use the tags that you have added to the collection policy. Let's say you added the tag apache_logs. Search for logs by using bmc_tags:apache_logs or applicationContext:Apache.


Where to go from here

Configuring-logs

Generating-alerts-from-logs

Deriving-insights-from-logs



 

Tip: For faster searching, add an asterisk to the end of your partial query. Example: cert*