Event deduplication and suppression for filtering unwanted events

Incoming events are processed through a set of out-of-the-box deduplication rules to determine whether the incoming event is a duplicate event or a new event. If it is a new event, it is ingested and displayed on the Events page. However, if the event is a duplicate, it is deduplicated and dropped. 

As an administrator, you can suppress duplicate events based on custom event selection criteria by configuring a suppression policy as part of an advanced enrichment policy.

Event suppression and event deduplication are performed together in the same stage of event processing. Event deduplication and suppression can help you filter out unwanted and unnecessary.

Out-of-the-box event deduplication

During deduplication, events are consolidated or folded into a single event representation based on the first-occurred event. 

As new or incoming events are received, they are checked against existing events based on deduplication slot values (for example, source address, alarm ID, and so on). If the incoming event has the same dedup values as an existing event, the incoming event is identified as a duplicate. When an event is identified as a duplicate, new information from that event is used to update the existing event and the new event is dropped. Dropped events are not ingested and therefore not available on the Events page. The product comes with a set of out-of-the-box dedup slots that form the default criteria for event deduplication. For more information about dedup slots, see Slot facets.

The following classes are supported for the out-of-the-box deduplication:

• ALARM
• HELIX_SM_EV
• Prediction
• ANOMALY
• PATROL_EV

The system deduplicates events that arrive with or without delay from the same event source. For the deduplication of events that arrive from different event sources to work seamlessly, we recommend that you specify a delay (for example, 2 seconds or more) between events that you send.

For example, suppose an alarm event occurs at 9:00 A.M. Another alarm event occurs at 9:10 A.M. with the same alarm ID (deduplication slot) as the first alarm event, but a different severity.

The following steps describe the deduplication process:

1. The first alarm event (9:00 A.M. event) is automatically updated with these details:
   • New severity: The first occurred event is updated with the severity of the subsequent event (9:10 A.M. event). 
   • Updated repeat count: The Repeat count in the first-occurred event is updated with +1. The repeat count refers to the number of occurrences of similar events. 
   • Updated modified time: The first occurred event is updated with the time when it was updated.
2. The subsequent event (9:10 A.M. event) is permanently dropped. 

Deduplication occurs only for out-of-the-box event classes. If you want to perform event deduplication for custom classes, you can configure an advanced enrichment policy by:

1. Performing a Lookup on the deduplicate events criteria
2. Using the DropNewEvent function.
   For more information, see details of the Use case: Drop duplicate events and update the existing event with new severity at Examples: Event policies for enrichment, correlation, notification, and suppression.

For sample deduplication policies configured for events received from BMC Helix Intelligent Integrations, see Out-of-the-box event policies and templates.

A small set of dedup slots are defined for particular event classes. However, you can define custom dedup slots in existing classes by using the events/classes API endpoint. For more information, see Event management endpoints in the REST API. 

Deduplicating events based on selection criteria

You can use the event and deduplication selection criteria in event policies to eliminate duplicate events.

---

Example

Sarah is a system administrator at Apex Global, which uses BMC Helix Operations Management to monitor its systems. Sarah notices frequent event storms during network outages. The system is flooded by duplicate events. As a result, the operators are overwhelmed, and the root-cause analysis of issues is delayed. To handle this problem, Sarah configures a deduplication policy. She configures event criteria and deduplication criteria to eliminate duplicate events coming into the system. Now, when multiple identical events are sent in quick succession, only the first event is processed, and its repeat count is incremented. This has reduced event noise and has enabled accurate incident tracking. The operators at Apex Global can now focus on resolving the underlying issue rather than sifting through duplicate events.

---

Note the following points for a deduplication policy:

• You can create only a single deduplication policy for one class.
• You cannot add multiple deduplication criteria for a deduplication policy. 
• You can add only a single deduplication configuration while creating an event policy.
• Policy precedence is not considered in deduplication.
• Only the ANDand EQUAL operators are supported in the deduplication criteria.
• The deduplication criteria fetch only the last event. If multiple events are searched, the latest event is retrieved.
• The deduplication policy does not work on the event update operation and other event operations.
• If an event policy does exist for the child class, the system checks for a policy on the parent class.
• If a time frame is not enabled for an event policy, the duplication policy is not executed. For the deduplication policy to run, the timeframe must be enabled or not configured at all.

You can get the best out of a deduplication policy with the following recommendations:

• Use the same source identifier for deduplication.
• For the deduplication selection criteria, use only those slots that will not change during the lifecycle of an event.

Deduplication on incoming events

The DropNewEvent function is supported for deduplication on new events.

The deduplication phase follows the following process for incoming or new events:

1. An incoming event is matched for the event selection criteria. In this case, the following actions are taken:
   1. If an event policy is not configured for the child event class or if the policy is disabled, the incoming event is matched against the criteria configured on the parent class.
   2. If an event policy is not configured on the parent class or if the policy is disabled, the incoming event is not deduplicated. 
2. After the incoming event matches with the event selection criteria, it is matched with the deduplication criteria that are configured in the deduplication policy.
3. If the event matches the deduplication criteria, it is determined to be a duplicate event for an existing event.
4. The duplicate event is dropped.

Deduplication on old events

The AddNote function is supported for deduplication on old events.

The deduplication phase follows the following process for old or existing events:

1. An existing or old event is matched against the criterion configured in the deduplication policy.
2. If the criterion matches, based on the deduplication policy configuration, the old event slot is enriched with the new event slot or with the AddNote function.
3. The old event is updated with an incremental count of duplicate events.

To configure the deduplication phase in event policies

1. While creating an event policy, make sure that you add information in the Policy Information, Event Selection Criteria, and Time Frame areas.
2. In the Policy Configuration area, click Add Configuration and then click Deduplication.
   
3. Click the Dedup node and add the deduplication criteria for existing events.
   Only the ANDand EQUAL operators are supported.
4. Click Update Incoming Events and select the DropNewEvent function to build your policy.
5. Click Update Old Events and select one of the following actions to build your policy.
   For more information about the following actions, see Actions for advanced and time-based enrichment.
   • Variable
   • If
   • Enrich
   • Function
6. Add the policy configurations and click Save. 

Suppressing events based on the event selection criteria

If you want to drop events based on a custom event selection criterion, you need to configure a suppression policy. In a suppression policy, the event selection criteria determine which events are selected for suppression. The selected events are permanently dropped. Dropped events are not ingested and therefore not available on the Events page. Unlike event deduplication, the first-occurring event is not updated with the new details of a duplicate event.

For example, suppose you create a suppression policy with the event selection criteria: Class="EVENT" AND Message=page not found

All the events with the message "page not found" are directly dropped. Note that suppression is applied on incoming events only. Existing events that are already ingested cannot be suppressed by using a suppression policy.

Event suppression can be very useful for known scenarios of unnecessary event occurrence. For example, based on your past experience, suppose you know that events containing a certain message are unnecessary and of no value. You create a suppression policy and provide an event selection criteria to select events containing that specific message. In this scenario, say a new event matching the desired criteria occurs. After three minutes another event occurs; then another after 30 minutes, and again another event occurs the next day, and so on. The suppression policy will keep dropping these events until you disable or delete the policy. 

For sample suppression policy configured for events received from BMC Helix Intelligent Integrations, see Out-of-the-box event policies and templates.

Suppressing events by using an advanced enrichment policy

While creating an advanced enrichment policy, you can build a policy workflow based on complex conditions and add a void function to drop new events. For more information, see Functions for advanced and time-based enrichment.

Similar to event deduplication and event suppression, dropped events are not ingested and therefore not available on the Events page. 

An advanced enrichment policy works based on the following criteria:

1. Event selection criteria that determines which events to select for processing (same as suppression policy)
2. Complex conditions defined in the policy workflow

This capability allows you to add a more granular condition to process events even after defining an event selection criteria. Also, this capability can be very useful for isolated use cases where you want to quickly drop specific events. For more information, see Example: Drop duplicate events and update the existing event with new severity.

The void function processing happens in the phase of advanced enrichment processing. Event processing happens in phases based on built-in rules. Based on these rules, the policies are evaluated in a certain order. For more information, see Event policy types and evaluation order.