Detecting anomalies from logs

The following procedure is used by BMC Helix Log Analytics to detect anomalous logs:

1. Processing the ingested logs.
   An administrator configures alert policies to identify anomalies in the logs. These logs are processed to remove dates, special characters, and unnecessary keywords. The cleaning process uses regular expressions to remove dates and filter characters.
2. Detecting anomalies.
   BMC Helix Log Analytics uses hierarchical clustering and cross-similarity techniques to detect anomalies in logs. Anomalies are detected by using the following method:

   1. The hierarchical clustering identifies anomalies from the first batch of logs.
      BMC Helix Log Analytics starts detecting anomalies after a default initial learning time of 15 minutes. The learning time is used to determine false positives.
      To change the learning time, contact BMC Support.

      Important

      Logs are ingested as a single file for any policy. If there are multiple log files matching the selection criteria defined in the policy, the logs from all files are combined in a single file.

   2. The identified anomalies are stored in a tags table in the binary format for further analysis.
   3. For every subsequent batch, the system performs a cross-similarity check between newly identified anomalies and the data stored in the tags binary.

1. 1. Any newly identified anomalies that differ from the previous data are considered new anomalies.
      An interval between the detection of anomalies is two hours. That means, if an anomaly is detected, and the anomaly reappears within two hours, the behavior is not considered anomalous. However, if an anomaly is detected, and the anomaly reappears after two hours, the behavior is flagged as anomalous.
      To change the interval duration, contact BMC Support.
      The complete list of anomalies includes both new and existing anomalies in tags.
   2. When the tag count reaches the maximum limit, 20% of the oldest tags are removed. 5000 is the tag count limit.
      To change the percentage of the oldest tags to be removed and the tag count limit, contact BMC Support.

   3. The anomalies use the context extraction technique for quick insights, where the top three keywords are extracted from the anomalies.

      Important

      Logs that contain the INFO, DEBUG, and TRACE keywords are not considered for anomaly detection. The keywords are not case-sensitive.

2. Analyzing anomalous logs.
   Use the Explorer tab to analyze log anomalies.
   Because of the anomaly detection alert policies, log anomaly events are generated in BMC Helix Operations Management. Use the Events page in BMC Helix Operations Management to analyze the log anomaly events.
   For more information, see Analyzing anomalous logs and anomaly events.

Anamolous log record examples

Configuring anomaly detection

As an administrator, create alert policies in BMC Helix Log Analytics to receive anomaly notifications. After you enable an alert policy, you can use the Explorer tab to analyze anomalous logs.

For instructions about creating an alert policy, see Generating-alerts-from-logs.

Analyzing anomalous logs and anomaly events

Use BMC Helix Log Analytics to analyze  anomalous logs. Use BMC Helix Operations Management to analyze the related anomaly events.

To analyze anomalous logs

1. In BMC Helix Log Analytics, on the Explorer tab, select the logml-* index pattern to view all the anomalous log messages.
2. Analyze the anomaly score of the logs to understand the anomaly strength.
   Each anomalous record contains the Anomaly and Anomaly_Score fields. The value of the Anomaly field is set to 1.0. The Anomaly_Score field represents the anomaly strength and has a value between 0 and 1. If the score is higher, the anomaly strength of the record is high.
   

BMC Helix Log Analytics automatically assigns severity to anomalous log messages depending on the keywords that the message contains.For example, if a message contains the words error or critical, the anomaly is assigned the High severity.The following table displays the severity level and its associated keywords:

To analyze log anomaly events

In BMC Helix Operations Management, use the Events page to analyze log anomaly events. Click the event to view the event details and analyze it.

The following procedure explains how you can go to BMC Helix Log Analytics from BMC Helix Operations Management.

1. On the Events page in BMC Helix Operations Management, click an anomaly event to view the event details.
   Log anomaly events are generated with the Log Event class. You can hover over Class  to see the class of the event.
   

   Tip: Filter log anomaly events to quickly analyze them

   To filter all events of the Log Event class, use advanced filter in BMC Helix Operations Management. For more information about advanced filters, see Filtering events.

2. Click the Others tab.
3. In the Search Parameters field, click the Review Logs link to open the Explorer tab in BMC Helix Log Analytics.

The Explorer tab opens in BMC Helix Log Analytics and the logs that generated the event are displayed. The anomalous logs are shown in the index pattern that begins with logml.
The anomalous log events are further processed for creating situations. For more information about situations, see Monitoring and investigating situations in the BMC Helix AIOps documentation.

Visualizing log anomaly events in BMC Helix Dashboards

Use the Self Monitoring dashboard in BMC Helix Dashboards to visualize anomalous log data.

The following image displays the Self Monitoring dashboard:

Use the Search parameter column to open the Explorer tab in BMC Helix Log Analytics.

Use the Event Details column to open the Event Details page in BMC Helix Operations Management.

For more information about the Self Monitoring dashboard, see Self-monitoring dashboard in the BMC Helix Dashboards documentation.