Collecting application logs

To start collecting logs, you add all the collection-related configurations and other details to a collection policy. You save time by reusing these configurations in multiple collection policies.

The following image shows how logs are collected by using the Linux (RHEL/CentOS) and Windows connectors:

Apps_log_collection.jpg


Before you begin

  • Make sure that you have downloaded and installed a connector. For more information, see Installing-and-managing-connectors.
  • Create a parsing rule. For more information, see Creating-a-parsing-rule .
  • Create a filtering rule. For more information, see Creating-a-filtering-rule.
    If you plan to filter logs before they are parsed, make sure that you add a separate rule for the pre-filtering process in addition to the filtering process. By prefiltering logs, you limit the number of logs that are parsed and thereby enhance system performance.


To collect application logs

In BMC Helix Log Analytics, use the Collection > Collection Policies > Create button and perform the following steps:

  1. In the Policy Information section, perform the following steps:
    1. Enter a unique name and description.
    2. From the Connector type list, select the connector type that you have installed.
  2. In the Connector configurations section, configure the connector by performing the following steps:
    1. In the Connector Type field, select the connector type that you want to configure.

    2. In Connector Selection Criteria, create the connector selection criteria to identify connectors for collection .
       When you click in the box , you are prompted to make a selection. Each time you make a selection, you are progressively prompted to make another selection. 
       The selection criteria consist of an opening parenthesis, followed by the slot name, the operator, the slot value (which can be a string based on the type of slot selected), and the closing parenthesis. You can optionally select the logical operator AND or OR to add additional conditions. Specifying the opening and closing parentheses is optional.
       The connector fields available to create the selection criteria are status, name, version, host_name, ip, and tags.  

      Important

      The values that you enter for a field in the selection criteria are case-sensitive. For example, if the host name is WebServer.example.com, add the selection criteria as ( host_name Equals WebServer.example.com ). If you enter ( host_name Equals webserver.example.com ), the connector is not selected. To add case-insensitive values, use the Equals ignore case operator.

  3. In the Configuration section, configure log collection by performing the following steps:
    1. Click Configure.
      configure_button.png

    2. In the Log Collection File Path field, enter the path of the log files that you want to collect.
      For example: /opt/tomcat/apache.log (Linux) or C:/app1/logs/app.log (Windows).
      When you enter folder locations, sub-folders and files present in the folder are shown. 

      Best practices

      • Enter only directory paths and an absolute file name with the path.
      • Separate multiple entries with a comma.
      • Make sure that all log files have the same format so that a single parsing rule parses all the logs.
      • To collect logs from Windows-based applications, ensure that you enter the path of the computer where you have installed the connector.
      • In Windows file path, replace back slashes (\) with forward slashes (/). For example, if your file path is C:\app1\logs\app.log, change it to C:/app1/logs/app.log.  
      • If your log files are created on the basis of size, enter the name of the file where the latest logs are written and do not enter * in the file path.
    3. (Optional) If you have entered a path with multiple folders and you want to exclude some folders from collection, in the  Exclude Paths  field, remove those folders.
      For example, you have entered the log collection path as  /opt/bmc/connectors/<connector_name>/logs/applicationLogs  and this folder contains the following folders:  app1, app2, app3. The app1, app2, and app3 folders are shown in the  Exclude Paths  field. To prevent log collection from the  app3 folder, remove the app3 folder from the field.
    4. (Optional) If you want to start collecting logs from the beginning of the file, select the Read Files from Beginning check box. 
      Else, log collection starts when you enable the policy.
             By default, all logs present in a log file are collected. They include the logs that are read for the first time and logs from the last read line. If your log files are created (and rotated) on the basis of time, you have given wildcard (*) in the file path, and the Read Files from Beginning check box is cleared, all logs created after the policy is enabled are collected.
    5. (Optional) Select the Open File on Every Update check box to enable the Open File on Every Update parameter.
      When some applications stop updating their log files, the monitoring agents add a permanent read lock to the application log files. To avoid this issue, enable the Open File on Every Update parameter so that the connector opens the log files when they are updated. By default, this parameter is disabled.
    6. Click Save.
    7. In the Tags field, enter the tags to identify the policy with the collected logs.
      The values that you enter in this field are added to the bmc_tags field that is present in the collected logs. You can use the field or tags to search and analyze logs in Explorer.
    8. In Fields, enter the custom information that you want to add to collected logs in the form of key-value pairs.
      Use these fields to search and analyze the logs in Explorer. For example,  Key: applicationContext;  Value: Apache. Use applicationContext:Apache as a search string to search and analyze the collected logs.

  4. (Optional) In the Pre-filtering Rule section, select a filtering rule to help you remove unwanted log messages before parsing logs.
    If you don't filter the log data before parsing , all log data is processed for parsing .
     If you have not created a filtering rule, perform one of the following actions:

    • Go to Collection > Filtering Rules and click Create. For more information, see Creating-a-filtering-rule.

    • In the Pre-filtering Rule  section, click Create New and complete the configurations. For information about the configurations, see Creating-a-filtering-rule.

      image-2024-9-2_11-6-11.png

    Click here to view an example that explains the benefits of a pre-filtering rule

    Sarah is a tenant administrator at Apex Global, which uses BMC Helix Log Analytics for collecting and analyzing logs. Apex Global parses all messages  generated for file type logs. The system performance is affected by the amount of logs generated. Apex Global wants to enhance system performance for file type logs. How can they achieve this?

    They can prefilter file type logs before the logs are parsed. This makes sure that only the required messages are parsed and filtered further, resulting in enhanced system performance.

  5. (Optional) In the Log Parsing section, select parsing rules so that BMC Helix Log Analytics connector can convert raw log data into key-value pairs for an efficient log search and analysis.
    Raw log data is not parsed if you don't select a parsing rule.
    • From the  Parsing Rule 1  list, select a parsing rule.
    • Parsing is supported at two levels for all log data, including multiline logs. Use the  Parsing Rule 2 list to select a rule to parse the logs at the second level.
      If you have not created a parsing rule, perform one of the following actions:
      • Go to  Collection > Parsing Rules  and click Create. For more information, see  Creating-a-parsing-rule .

      • In the  Log Parsing  section, click  Create New  against a parsing rule and complete the configurations. For information about the configuration, see  Creating-a-parsing-rule .

        image-2024-9-5_17-20-48.png

  6. (Optional) In the Filtering Rule section, select a filtering rule to filter parsed messages.
    If you don't filter the logs, all the logs are processed further. If you have not created a filtering rule, see Creating-a-filtering-rule for instructions.
    If you have not created a filtering rule, perform one of the following actions:
    • Go to Collection > Filtering Rules  and click Create. For more information, see  Creating-a-filtering-rule .
    • In the Filtering Rule section, click Create New and complete the configurations. For information about the configurations, see Creating-a-filtering-rule.

      image-2024-9-2_11-13-40.png
  7. (Optional) In the User group section, select one or more user groups to assign to the collection policy.
     Users associated with this user group can see the data collected by this collection policy. If you don't select the user group, log data will be visible to all the users.
  8. Enable and save the policy by performing the following steps:
    1. To start collecting logs, select the Enable Collection Policy check box.
      You can choose to enable the collection policy later.
    2. Click Save .
      The created policy is shown on the Collection Policies page. Use the Actions menu to edit, enable (or disable), and delete the policy.

Important

The multiline parsing option is removed from the Configuration section on the Collection Policies page. You can create a multiline parsing rule from Collection > Collection Policies > Create Collection Policy > Log Parsing > Parsing Rule 1.


To verify log collection

  1. Go to BMC Helix Log Analytics and click the Explorer tab.
  2. Search the logs based on a unique field value.
    For example, you can use the tags that you have added to the collection policy. Let's say you added the tag apache_logs. Search for logs by using bmc_tags:apache_logs or applicationContext:Apache.


Where to go from here

Configuring-logs

Generating-alerts-from-logs

Deriving-insights-from-logs





 

Tip: For faster searching, add an asterisk to the end of your partial query. Example: cert*