Analyzing anomalous logs
How anomalous logs are identified
When the logs are ingested, the first step is to process and clean them. It involves parsing the logs, extracting relevant information, and performing necessary preprocessing steps, such as removing irrelevant data, normalizing text, or handling missing values. When the logs are cleaned, an ML model is generated by training it on the initial set of logs. The training requires around 50,000 or more log records and around 5 to 10 minutes. When a model is generated, it sets an anomaly threshold value.
The model analyzes the incoming logs semantically and assigns a score to each log record. The value of the score lies between 0 and 1. If the score of the log record is higher than the threshold, it is considered anomalous. If the score is higher, the anomaly strength of the log record is high.
The model is updated and trained every 10 minutes by using the latest logs (up to 10,000).
Here are a few examples of anomalous log records:
The following video (1:50) provides you an overview of how BMC Helix Log Analytics detects anomalies.
Analyzing anomalous logs
The anomalous logs are available for analysis in the index pattern that starts with logml-*.
Each anomalous record contains the Anomaly and Anomaly_Score fields. The value of the Anomaly field is set as 1.0. The Anomaly_Score field represents the anomaly strength and has the value between 0 and 1. If the score is higher, the anomaly strength of the record is high.
When anomaly is found in the logs, an anomalous log event is generated as a result of an alert policy that you configure for anomaly detection. The event is generated in BMC Helix Operations Management where an operator can take appropriate actions. To view anomalous logs, in the Search Parameters field of the event under the Others tab, there is the link to launch BMC Helix Log Analytics. When you click this link, the Explorer opens in BMC Helix Log Analytics and the anomalous logs are displayed. To configure an alert policy to detect anomalies or rare patterns, see Generating-alerts-from-logs.
The following image shows the anomalous logs:
Learn more
Read the product blog to learn more about automated log analysis with ML-based anomaly detection Predictive Log Alerting with ML Anomaly Detection.