Event deduplication, suppression, and closure for reducing event noise

Third-party events that come in through BMC Helix Intelligent Integrations are processed in BMC Helix Operations Management with the out-of-the-box BMC Helix Intelligent Integrations policies. The policies determine whether an incoming event is new or is a duplicate event. Depending on whether an event is new or duplicate, one of the following actions are taken:

The new event is dropped and the old event is updated (event deduplication)
The new event is dropped and the old event is not updated (event suppression)
The new event is closed and the old event is updated (event closure)

If these policies are disabled, events are received by BMC Helix Operations Management; however, they are not evaluated for duplication.

Event deduplication, suppression, and closure are performed together in the same stage of event processing and help you eliminate event noise.

Event deduplication

During deduplication, events are consolidated into a single event based on the event that arrived first.

New events are checked against existing events based on the deduplication slot values. If the incoming event has the same deduplication values as an existing event, the incoming event is identified as a duplicate. Any new information from that event is used to update the existing event and the new event is dropped. Dropped events are not ingested and therefore not available on the Events page.

Event suppression

In a suppression policy, the event selection criteria determines which events are selected for suppression. The selected events are permanently dropped. Dropped events are not ingested and therefore not available on the Events page. The event that arrived first is not updated with the details of a duplicate event.

Event closure

In an closure policy, the event selection criteria determines which events are selected for closure. When an event is identified for closure, new information from that event is used to update the existing event and the new event is closed.

Out-of-the-box policies

Incoming events from third-party sources are processed in BMC Helix Operations Management through a set of out-of-the-box deduplication rules defined in the following polices:

Update Old Events - BMC Helix Intelligent Integrations (Disabled by default in BMC Helix Operations Management)
Drop Duplicate Events - BMC Helix Intelligent Integrations (Enabled by default in BMC Helix Operations Management)

These policies are created in BMC Helix Operations Management when you use BMC Helix Intelligent Integrations for the first time to fetch events from any third-party source supported by BMC Helix Intelligent Integrations. You can edit the policy as required.

Some sources such as Datadog and IBM Netcool have a specific format in which they provide information about events to BMC Helix Intelligent Integrations. For such sources, the following policies are defined in BMC Helix Operations Management:

Update AWS CloudWatch Events - BMC Helix Intelligent Integrations
Update Azure Events - BMC Helix Intelligent Integrations
Update Datadog Events - BMC Helix Intelligent Integrations
Update IBM Netcool Events - BMC Helix Intelligent Integrations
Update Icinga Events - BMC Helix Intelligent Integrations
Update Netreo Events - BMC Helix Intelligent Integrations
Close Old AWS Prometheus Alerts - BMC Helix Intelligent Integrations
Close Old Catchpoint Events - BMC Helix Intelligent Integrations
Close Old Dynatrace Alerts - BMC Helix Intelligent Integrations
Close Old MS SCOM Events - BMC Helix Intelligent Integrations

The policy for a specific source is created in BMC Helix Operations Management when you use BMC Helix Intelligent Integrations for the first time to fetch events from that source. You can edit the policy as required. The policy is enabled by default in BMC Helix Operations Management.

Policies applicable for all sources

The following table describes the policies that are applicable for all sources:

Event policy	Description
Update Old Events - BMC Helix Intelligent Integrations (Disabled by default in BMC Helix Operations Management) This policy is created when you use BMC Helix Intelligent Integrations for the first time to fetch events from any third-party source supported by BMC Helix Intelligent Integrations. You can edit the policy as required.	Creates a new event if the old event is CLOSED. Updates the old event severity with the new event severity if the old event is of the same type as the new event, based on the following deduplication slot values: Source Identifier Message Increments the repeat count of the old event by 1. Drops the new event. Updates the old event with the notes containing the event ID of the dropped event.
Drop Duplicate Events - BMC Helix Intelligent Integrations (Enabled by default in BMC Helix Operations Management) This policy is created when you use BMC Helix Intelligent Integrations for the first time to fetch events from any third-party source supported by BMC Helix Intelligent Integrations. You can edit the policy as required.	Drops the new event if the old event is of the same type as the new event, based on the following deduplication slot values: source_unique_event_Id Id Occurred Source Identifier

Policies applicable for specific sources

The following table describes the policies that are applicable for specific sources:

Event policy	Description
Update AWS CloudWatch Events - BMC Helix Intelligent Integrations (Enabled by default in BMC Helix Operations Management) This policy is created when you use BMC Helix Intelligent Integrations for the first time to fetch events from AWS CloudWatch. You can edit the policy as required.	Updates the old event slot values with the new event if a new event for the same metric and entity is received based on the following deduplication slot values: alarmArn Metric Name Source Identifier Drops the new event. Updates the old event with the notes containing the event ID of the dropped event.
Update Azure Events - BMC Helix Intelligent Integrations (Enabled by default in BMC Helix Operations Management) This policy is created when you use BMC Helix Intelligent Integrations for the first time to fetch events from Azure Monitor. You can edit the policy as required.	Updates the old event slot values with the new event if a new event for the same entity is received based on the following deduplication slot values: Message Source Identifier Drops the new event if the new event is closed. Updates the old event with the notes containing the source_unique_event_id of the dropped event.
Update Datadog Events - BMC Helix Intelligent Integrations (Enabled by default in BMC Helix Operations Management) This policy is created when you use BMC Helix Intelligent Integrations for the first time to fetch events from Datadog. You can edit the policy as required.	Updates the old event slot values with the new event if a new event for the same monitor and entity is received based on the following deduplication slot values: Source Identifier Source Monitor Identifier Drops the new event. Updates the old event with the notes containing the event ID of the dropped event.
Update IBM Netcool Events - BMC Helix Intelligent Integrations (Enabled by default in BMC Helix Operations Management) This policy is created when you use BMC Helix Intelligent Integrations for the first time to fetch events from IBM Netcool. You can edit the policy as required.	Updates the old event slot values with the new event if a new event for the same metric and entity is received based on the following deduplication slot values: manager Source Identifier sourceAlertGroup sourceAlertKey Drops the new event. Updates the old event with the notes containing the event ID of the dropped event.
Update Icinga Events - BMC Helix Intelligent Integrations (Enabled by default in BMC Helix Operations Management) This policy is created when you use BMC Helix Intelligent Integrations for the first time to fetch events from Icinga. You can edit the policy as required.	Updates the old event slot values with the new event if a new event for the same service and the same entity is received based on the following slot values: Source Event Identifier Status is not Closed or Blackout Closes the old event and updates the old event notes containing the event ID of the new event.
Update Netreo Events - BMC Helix Intelligent Integrations (Enabled by default in BMC Helix Operations Management)	Updates the old event status with the new event status if the old event is of the same type as the new event, based on the following deduplication slot values: Source Identifier source_unique_event_id Drops the new event. Updates the old event with the notes containing the source_unique_event_id of the dropped event.
Close Old AWS Prometheus Alerts - BMC Helix Intelligent Integrations (Enabled by default in BMC Helix Operations Management ) This policy is created when you use BMC Helix Intelligent Integrations for the first time to fetch alerts from AWS Prometheus. You can edit the policy as required.	Creates a new event and closes the old event if an incoming event contains has the same fingerprint as an existing event. Updates the old event with the notes containing the source_unique_event_id of the new event based on the following slot values: Source Identifier source_fingerprint
Close Old Catchpoint Events - BMC Helix Intelligent Integrations (Enabled by default in BMC Helix Operations Management ) This policy is created when you use BMC Helix Intelligent Integrations for the first time to fetch alerts from Catchpoint. You can edit the policy as required.	(Applicable for Catchpoint versions earlier than Cheetah and BMC Helix Intelligent Integrations earlier than 24.1.02) Creates a new event and closes the old event if an incoming event contains the results of same test on the same node as an existing event. Updates the old event with the notes containing the event ID of the new event based on the following slot values: Source Identifier Source Event Identifier (Applicable for Catchpoint versions Cheetah and later, and BMC Helix Intelligent Integrations versions 24.1.02 and later) The policy creates a new event for each node on which a test was performed and the test triggered the Critical level alert in Catchpoint. After the new event is created, the policy closes and updates the old event with notes containing the event ID of the new event, node ID, and test ID based on the following slot values: Source Identifier Source Event Identifier If an incoming event contains the Improved level alert triggered by a test on any of the nodes, the policy closes all the old events generated for all the nodes on which the test was performed.
Close Old Dynatrace Alerts - BMC Helix Intelligent Integrations (Enabled by default in BMC Helix Operations Management)	Creates a new event and closes the old event if an incoming event contains has the same event ID as an existing event. Updates the old event with the notes containing the source_unique_event_id of the new event based on the following deduplication slot values: Source Identifier source_unique_event_id
Close Old MS SCOM Events - BMC Helix Intelligent Integrations (Enabled by default in BMC Helix Operations Management)	Creates a new event and closes the old event if an incoming event contains has the same event ID as an existing event. Updates the old event with the notes containing the source_unique_event_id of the new event based on the following deduplication slot values: source_unique_event_id source_eventId

Policy evaluation order for processing events

Event deduplication and suppression policies are automatically run in the following order:

Drop Duplicate Events - BMC Helix Intelligent Integrations
Update Old Events - BMC Helix Intelligent Integrations
Ensure that you have enabled this policy if you want the events to be processed against this policy.
Update AWS CloudWatch Events - BMC Helix Intelligent Integrations, Update Azure Events - BMC Helix Intelligent Integrations, Update Datadog Events - BMC Helix Intelligent Integrations, Update IBM Netcool Events - BMC Helix Intelligent Integrations, Update Icinga Events - BMC Helix Intelligent Integrations, Update Netreo Events - BMC Helix Intelligent Integrations, Close Old AWS Prometheus Alerts - BMC Helix Intelligent Integrations, Close Old Catchpoint Events - BMC Helix Intelligent Integrations, Close Old Dynatrace Alerts - BMC Helix Intelligent Integrations, Close Old MS SCOM Events - BMC Helix Intelligent Integrations