Analyzing anomalous logs


BMC Helix Log Analytics provides automated analysis with machine learning (ML)-based anomaly detection of abnormal or rare log patterns (or anomalies) that indicate any deviation from the normal behavior. This analysis helps you find concerns proactively before they become a problem and also helps troubleshoot errors when they arise.

How anomalous logs are identified

When the logs are ingested, the first step is to process and clean them. It involves parsing the logs, extracting relevant information, and performing necessary preprocessing steps, such as removing irrelevant data, normalizing text, or handling missing values. When the logs are cleaned, an ML model is generated by training it on the initial set of logs. The training requires around 50,000 or more log records  and around 5 to 10 minutes. When a model is generated, it sets an anomaly threshold value.

The model analyzes the incoming logs semantically and assigns a score to each log record. The value of the score lies between 0 and 1. If the score of the log record is higher than the threshold, it is considered anomalous. If the score is higher, the anomaly strength of the log record is high. 

The model is updated and trained every 10 minutes by using the latest logs (up to 10,000).

Here are a few examples of anomalous log records:

Example of anomalous Apache logs

Non-anomalous logs

[Fri Mar 24 04:47:44 2023] [notice] workerEnv.init() ok /etc/httpd/conf/workers2.properties
[Fri Mar 24 04:51:08 2023] [notice] jk2_init() Found child 6725 in scoreboard slot 10

Anomalous logs

[Fri Mar 24 01:04:31 2023] [error] [client 218.62.18.218] Directory index forbidden by rule: /var/www/html/
[Fri Mar 24 20:47:17 2023] [error] jk2_init() Can't find child 2087 in scoreboard
Example of anomalous Linux logs

Non-anomalous logs

Mar 24 06:06:21 combo kernel: usbcore: registered new driver hub
Mar 24 06:06:23 combo kernel: ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
Mar 24 06:06:28 combo kernel: PCI: Found IRQ 11 for device 0000:00:1f.2
Mar 24 06:06:29 combo kernel: uhci_hcd 0000:00:1f.2: new USB bus registered, assigned bus number 1

Anomalous Records

Mar 24 06:06:29 combo kernel: audit(1138278101.749:164014): avc:  denied  { ioctl } for  pid=594 exe=/usr/lib/vte/gnome-pty-helper path=/dev/pts/0 dev= ino=2 scontext=system_u:system_r:kernel_t tcontext=system_u:object_r:devpts_t tclass=chr_file
Mar 24 06:06:29 combo kernel: audit(1138278101.766:164029): avc:  denied  { setattr } for  pid=594 exe=/usr/lib/vte/gnome-pty-helper name=0 dev= ino=2 scontext=system_u:system_r:kernel_t tcontext=system_u:object_r:devpts_t tclass=chr_file
Example of anomalous Windows logs

Non-anomalous logs

2022-07-28 04:30:31, Info  CBS  SQM: Initializing online with Windows opt-in: False
2022-07-28 04:30:31, Info  CBS  SQM: Cleaning up report files older than 10 days.
2022-07-28 04:30:31, Info  CBS  SQM: Requesting upload of all unsent reports.
2022-07-29 00:00:46, Info  CBS  Startup processing thread terminated normally

Anomalous logs

2022-07-28 04:30:31, Warn CBS No startup processing required, TrustedInstaller service was not set as autostart, or else a reboot is still pending.
2022-07-29 00:00:47, Error CBS Failed to create backup log cab. [HRESULT = 0x80070001 - ERROR_INVALID_FUNCTION]

The following video (1:50) provides you an overview of how BMC Helix Log Analytics detects anomalies.


icon-play@2x.pngWatch the YouTube video to get an overview of the anomaly detection process in BMC Helix Log Analytics.

Analyzing anomalous logs

The anomalous logs are available for analysis in the index pattern that starts with logml-*.

Each anomalous record contains the Anomaly and Anomaly_Score fields. The value of the Anomaly field is set as 1.0. The Anomaly_Score field represents the anomaly strength and has the value between 0 and 1. If the score is higher, the anomaly strength of the record is high.

When anomaly is found in the logs, an anomalous log event is generated as a result of an alert policy that you configure for anomaly detection. The event is generated in BMC Helix Operations Management where an operator can take appropriate actions. To view anomalous logs, in the Search Parameters field of the event under the Others tab, there is the link to launch BMC Helix Log Analytics. When you click this link, the Explorer opens in BMC Helix Log Analytics and the anomalous logs are displayed. To configure an alert policy to detect anomalies or rare patterns, see Generating-alerts-from-logs

The following image shows the anomalous logs:

AnomalyScore.jpg 

Important

The log records of type Info and Debug are considered non-anomalous and are not reported.

Learn more

Read the product blog to learn more about automated log analysis with ML-based anomaly detection Predictive Log Alerting with ML Anomaly Detection.

 

Tip: For faster searching, add an asterisk to the end of your partial query. Example: cert*