Event deduplication and suppression for reducing event noise

Incoming events from third-party sources through BMC Helix Intelligent Integrations are processed in BMC Helix Operations Management through a set of deduplication rules to determine whether the incoming event is a duplicate event or a new event. If it is a new event, it is ingested and displayed on the Events page. However, if the event is a duplicate, it is deduplicated and dropped.

Event suppression and event deduplication are performed together in the same stage of event processing. Event deduplication and suppression can help you eliminate event noise.

Out-of-the-box event policies and templates Open link

Event deduplication

During deduplication, events are consolidated or folded into a single event representation based on the first occurred event.

As new or incoming events are received, they are checked against existing events based on the deduplication slot values. If the incoming event has the same dedup values as an existing event, the incoming event is identified as a duplicate. When an event is identified as a duplicate, new information from that event is used to update the existing event and the new event is dropped. Dropped events are not ingested and therefore not available on the Events page. Deduplication occurs for the IIMonitorEvent, AWSCloudWatchAlert, and NetcoolEvent event classes.

The following table lists the event policies created in BMC Helix Operations Management:

Event policy	Description
Update Old Events - BMC Helix Intelligent Integrations (Disabled by default in BMC Helix Operations Management) This policy is created when you use BMC Helix Intelligent Integrations for the first time to fetch events from any third-party source supported by BMC Helix Intelligent Integrations. You can edit the policy as required.	Creates a new event if the old event is CLOSED. Updates the old event severity with the new event severity if the old event is of the same type as the new event, based on the following deduplication slot values: Status Source Identifier Message Increments the repeat count of the old event by 1. Drops the new event. Updates the old event with the notes containing the event ID of the dropped event.
Update IBM Netcool Events - BMC Helix Intelligent Integrations (Enabled by default in BMC Helix Operations Management) This policy is created when you use BMC Helix Intelligent Integrations for the first time to fetch events from IBM Netcool. You can edit the policy as required.	Updates the old event slot values with the new event if a new event for the same metric and entity is received based on the following deduplication slot values: manager Source Identifier sourceAlertGroup sourceAlertKey Status Drops the new event. Updates the old event with the notes containing the event ID of the dropped event.
Update AWS CloudWatch Events - BMC Helix Intelligent Integrations (Enabled by default in BMC Helix Operations Management) This policy is created when you use BMC Helix Intelligent Integrations for the first time to fetch events from AWS CloudWatch. You can edit the policy as required.	Updates the old event slot values with the new event if a new event for the same metric and entity is received based on the following deduplication slot values: alarmArn Metric Name Source Identifier Status Drops the new event. Updates the old event with the notes containing the event ID of the dropped event.
Update Azure Events - BMC Helix Intelligent Integrations (Enabled by default in BMC Helix Operations Management) This policy is created when you use BMC Helix Intelligent Integrations for the first time to fetch events from Azure Monitor. You can edit the policy as required.	Updates the old event slot values with the new event if a new event for the same metric and entity is received based on the following deduplication slot values: alarmArn Metric Name Source Identifier Status Drops the new event. Updates the old event with the notes containing the event ID of the dropped event.

A small set of dedup slots are defined for particular event classes. However, you can define custom dedup slots in existing classes by using the events/classes API endpoint. For more information, see Managing events with REST APIs Open link .

Event suppression

In a suppression policy, the event selection criteria determines which events are selected for suppression. The selected events are permanently dropped. Dropped events are not ingested and therefore not available on the Events page. Deduplication occurs for the IIMonitorEvent event class.

Unlike event deduplication, the first occurred event is not updated with the new details of a duplicate event.

For example, suppose you create a suppression policy with the event selection criteria: Class="EVENT" AND Message=page not found

All the events with the message, "page not found", are directly dropped. Note that suppression is applied on incoming events only. Existing events that are already ingested cannot be suppressed by using a suppression policy.

Event suppression can be very useful for known scenarios of event noise.

For example, based on your past experience, suppose you know that events containing a certain message are of no value. You create a suppression policy and provide an event selection criteria to select events containing that specific message. In this scenario, say a new event matching the desired criteria occurs. After three minutes another event occurs; then another after 30 minutes, and again another event occurs the next day, and so on. The suppression policy will keep dropping these events until you disable or delete the policy.

The following table lists the event policies created in BMC Helix Operations Management:

Event policy

Description

Drop Duplicate Events - BMC Helix Intelligent Integrations (Enabled by default in BMC Helix Operations Management)

This policy is created when you use BMC Helix Intelligent Integrations for the first time to fetch events from any third-party source supported by BMC Helix Intelligent Integrations. You can edit the policy as required.

Drops the new event if the old event is of the same type as the new event, based on the following deduplication slot values:

- Creation Time
- Message
- Occurred
- Severity
- Source Identifier
- Status

Policy evaluation order for processing events

Event deduplication and suppression polices are automatically run in the following order:

Drop Duplicate Events - BMC Helix Intelligent Integrations
Update Old Events - BMC Helix Intelligent Integrations
Update AWS CloudWatch Events - BMC Helix Intelligent Integrations or Update Azure Events - BMC Helix Intelligent Integrations or Update IBM Netcool Events - BMC Helix Intelligent Integrations

Event deduplication and suppression for reducing event noise