Detecting anomalies from logs

How are anomalies detected?

The following procedure is used by BMC Helix Log Analytics to detect anomalies:

1. Processing the ingested logs.
   An administrator configures alert policies to identify anomalies in the logs. These logs are processed to remove dates, special characters, and unnecessary keywords. The cleaning process uses regular expressions to remove dates and filter characters.
2. Detecting anomalies.
   BMC Helix Log Analytics uses hierarchical clustering and cross-similarity techniques to identify anomalies in logs. Anomalies are detected by using the following method:
   1. The hierarchical clustering identifies anomalies from the first batch of logs.
   2. The identified anomalies are stored in a tags table in the binary format for further analysis.
   3. For every subsequent batch, the system performs a cross-similarity check between the newly identified anomalies and the data stored in the tags binary.
      The complete list of anomalies includes both new and existing anomalies in tags.

1. 1. The newly identified anomalies that are different from the previous data are considered new anomalies.
   2. The anomalies use the context extraction technique for quick insights, where the top three keywords are extracted from the anomalies.
2. Analyzing anomalous logs.
   Use the Explorer tab to analyze log anomalies.
   Because of the anomaly detection alert policies, log anomaly events are generated in BMC Helix Operations Management. Use the Events page in BMC Helix Operations Management to analyze the log anomaly events.
   For more information, see Analyzing anomalous logs and anomaly events.

Anamolous log record examples

To configure anomaly detection

As an administrator, create alert policies in BMC Helix Log Analytics to receive anomaly notifications. After you enable an alert policy, you can use the Explorer tab to analyze anomalous logs.

For instructions about creating an alert policy, see Generating-alerts-from-logs.

Analyzing anomalous logs and anomaly events

Use BMC Helix Log Analytics to analyze  anomalous logs. Use BMC Helix Operations Management to analyze the related anomaly events.

To analyze anomalous logs

1. In BMC Helix Log Analytics, on the Explorer tab, select the logml-* index pattern to view all the anomalous log messages.
2. Analyze the anomaly score of the logs to understand the anomaly strength.
   Each anomalous record contains the Anomaly and Anomaly_Score fields. The value of the Anomaly field is set to 1.0. The Anomaly_Score field represents the anomaly strength and has a value between 0 and 1. If the score is higher, the anomaly strength of the record is high.
   

To analyze log anomaly events

In BMC Helix Operations Management, use the Events page to analyze log anomaly events. Click the event to view the event details and analyze it.

The following procedure explains how you can go to BMC Helix Log Analytics from BMC Helix Operations Management.

1. On the Events page in BMC Helix Operations Management, click an anomaly event to view the event details.
   Log anomaly events are generated with the Log Event class. You can hover over Class  to see the class of the event.
   

   Tip: Filter log anomaly events to quickly analyze them

   To filter all events of the Log Event class, use advanced filter in BMC Helix Operations Management. For more information about advanced filters, see Filtering events.

2. Click the Others tab.

3. In the Search Parameters field, click the Review Logs link to open the Explorer tab in BMC Helix Log Analytics.
   More colours

   The Explorer tab opens in BMC Helix Log Analytics and the logs that generated the event are displayed. The anomalous logs are shown in the index pattern that begins with logml.
   The anomalous log events are further processed for creating situations. For more information about situations, see Monitoring and investigating situations in the BMC Helix AIOps documentation.

Visualizing log anomaly events in BMC Helix Dashboards

Use the Self Monitoring dashboard in BMC Helix Dashboards to visualize anomalous log data.

The following image displays the Self Monitoring dashboard:

Use the Search parameter column to open the Explorer tab in BMC Helix Log Analytics.

Use the Event Details column to open the Event Details page in BMC Helix Operations Management.

For more information about the Self Monitoring dashboard, see Self-monitoring dashboard in the BMC Helix Dashboards documentation.