Detecting anomalies from logs
Example
Sarah is an administrator in Apex Global, which uses BMC Helix Log Analytics for log collection and analysis. Sarah wants to be alerted if there are any deviations from the normal behavior in the system. She knows that small deviations often lead to bigger failures, and she wants to debug these deviations before they become problems. How can Sarah receive such alerts?
Sarah configures alert policies to receive anomaly notifications. When an anomaly is detected, notifications are generated in the form of events. These events are generated in BMC Helix Operations Management. Sarah can also view these events in BMC Helix AIOps and BMC Helix Dashboards.
The following video (2:10) provides a high-level summary of the anomaly detection feature in BMC Helix Log Analytics.
The following procedure is used by BMC Helix Log Analytics to detect anomalous logs:
- Processing the ingested logs.
An administrator configures alert policies to identify anomalies in the logs. These logs are processed to remove dates, special characters, and unnecessary keywords. The cleaning process uses regular expressions to remove dates and filter characters. - Detecting anomalies.
BMC Helix Log Analytics uses hierarchical clustering and cross-similarity techniques to detect anomalies in logs. Anomalies are detected by using the following method:The hierarchical clustering identifies anomalies from the first batch of logs.
BMC Helix Log Analytics starts detecting anomalies after a default initial training period of 15 minutes. The training period is used to determine false positives.- The identified anomalies are stored in a tags table in the binary format for further analysis.
- For every subsequent batch, the system performs a cross-similarity check between newly identified anomalies and the data stored in the tags binary.
-
- Any newly identified anomalies that differ from the previous data are considered new anomalies.
The default aging period of an anomaly is two hours. That means, if an anomaly is detected, and the anomaly reappears within two hours, the behavior is not considered anomalous. However, if an anomaly is detected, and the anomaly reappears after two hours, the behavior is flagged as anomalous.
The complete list of anomalies includes both new and existing anomalies in tags. - The default anomaly tag limit is 2000. That means when the tag count reaches the maximum limit, 20% of the oldest tags are removed.
The recommended range for the anomaly tag limit is between 1000 and 2000. If the limit exceeds this range, it can lead to issues in generating log anomalies.
- Any newly identified anomalies that differ from the previous data are considered new anomalies.
-
The anomalies use the context extraction technique for quick insights, where the top three keywords are extracted from the anomalies.
- Analyzing anomalous logs.
Use the Explorer tab to analyze log anomalies.
Because of the anomaly detection alert policies, log anomaly events are generated in BMC Helix Operations Management. Use the Events page in BMC Helix Operations Management to analyze the log anomaly events.
For more information, see Analyzing anomalous logs and anomaly events.
Anamolous log record examples
Configuring anomaly detection
As an administrator, create alert policies in BMC Helix Log Analytics to receive anomaly notifications. After you enable an alert policy, you can use the Explorer tab to analyze anomalous logs.
For instructions about creating an alert policy, see Generating-alerts-from-logs.
BMC Helix Log Analytics provides anomaly detection settings to help you accurately identify unusual patterns and potential issues in your log data. Use these settings to configure the following parameters that determine how anomalies are detected.
With these settings, you can configure the parameters that are described in the following table:
To configure anomaly detection settings
As an administrator:
- From the Configurations menu, select Anomaly Detection Settings.
- On the Anomaly Detection Settings page, modify Training Period, Aging Period, and Anomaly Tag Limit according to your requirements.
- Click Save.
To set these values to the default, click Reset to Default.
Analyzing anomalous logs and anomaly events
Use BMC Helix Log Analytics to analyze anomalous logs. Use BMC Helix Operations Management to analyze the related anomaly events.
To analyze anomalous logs
- In BMC Helix Log Analytics, on the Explorer tab, select the logml-* index pattern to view all the anomalous log messages.
- Analyze the anomaly score of the logs to understand the anomaly strength.
Each anomalous record contains the Anomaly and Anomaly_Score fields. The value of the Anomaly field is set to 1.0. The Anomaly_Score field represents the anomaly strength and has a value between 0 and 1. If the score is higher, the anomaly strength of the record is high.
BMC Helix Log Analytics automatically assigns severity to anomalous log messages depending on the keywords that the message contains.For example, if a message contains the words error or critical, the anomaly is assigned the High severity.The following table displays the severity level and its associated keywords:
Severity | Associated keywords |
|---|---|
High | error, critical, crash, failure, fatal, exception, outage, deadlock, segfault, corrupt, unreachable |
Medium | warn, warning, timeout, degraded, inconsistency, stale, inefficiency, bottleneck, throttle |
Low | All anomalous log messages that do not contain the keywords listed for High and Medium severities are granted Low severity. |
Severity | Associated keywords |
|---|---|
High | error, critical, crash, failure, fatal, exception, outage, deadlock, segfault, corrupt, unreachable |
Medium | warn, warning, timeout, degraded, inconsistency, stale, inefficiency, bottleneck, throttle |
Low | All anomalous log messages that do not contain the keywords listed for High and Medium severities are granted Low severity. |
Severity | Associated keywords |
|---|---|
High | error, critical, crash, failure, fatal, exception, outage, deadlock, segfault, corrupt, unreachable |
Medium | warn, warning, timeout, degraded, inconsistency, stale, inefficiency, bottleneck, throttle |
Low | All anomalous log messages that do not contain the keywords listed for High and Medium severities are granted Low severity. |
Severity | Associated keywords |
|---|---|
High | error, critical, crash, failure, fatal, exception, outage, deadlock, segfault, corrupt, unreachable |
Medium | warn, warning, timeout, degraded, inconsistency, stale, inefficiency, bottleneck, throttle |
Low | All anomalous log messages that do not contain the keywords listed for High and Medium severities are granted Low severity. |
To analyze log anomaly events
In BMC Helix Operations Management, use the Events page to analyze log anomaly events. Click the event to view the event details and analyze it.
The following procedure explains how you can go to BMC Helix Log Analytics from BMC Helix Operations Management.
On the Events page in BMC Helix Operations Management, click an anomaly event to view the event details.
Log anomaly events are generated with the Log Event class. You can hover over Class
to see the class of the event.- Click the Others tab.
- In the Search Parameters field, click the Review Logs link to open the Explorer tab in BMC Helix Log Analytics.
The Explorer tab opens in BMC Helix Log Analytics and the logs that generated the event are displayed. The anomalous logs are shown in the index pattern that begins with logml.
The anomalous log events are further processed for creating situations. For more information about situations, see Monitoring and investigating situations in the BMC Helix AIOps documentation.
Visualizing log anomaly events in BMC Helix Dashboards
Use the Self Monitoring dashboard in BMC Helix Dashboards to visualize anomalous log data.
The following image displays the Self Monitoring dashboard:
Use the Search parameter column to open the Explorer tab in BMC Helix Log Analytics.
Use the Event Details column to open the Event Details page in BMC Helix Operations Management.
For more information about the Self Monitoring dashboard, see Self-monitoring dashboard in the BMC Helix Dashboards documentation.