How log enrichment works

When a log entry is received, the enrichment algorithm filters the enrichment policies where the selection criteria match the logs. Next, the algorithm sorts these policies based on the precedence value that defines the priority for executing the policy. Note that a policy with a lower precedence number is executed first. Enrichments from all policies with the highest precedence are applied first (for example, precedence value 1). Then, the algorithm moves on to the policies with higher precedence numbers (for example, precedence value 2). The algorithm moves in an ascending order of precedence value. 

In addition to the precedence value, enrichment is applied in the order enrichment sources are selected in a policy. For example, in an enrichment policy, you configured enrichment sources in the following order:

• GeoIP
• LDAP
• DNS

First, the GeoIP enrichment is applied, then LDAP, and then DNS. 

While configuring the enrichment sources, ensure that you select the sources in the correct order. For example, from an enrichment source, you get the country value. By using the country value, you want to get the city value from another enrichment source. In such cases, in an enrichment policy, you first select the enrichment source that provides the country value and then select the enrichment source that provides the city value. Based on your requirements, you can select these sources in different enrichment policies and configure their precedence accordingly.

The following video (2:34) illustrates how logs are enriched in BMC Helix Log Analytics.

Watch the YouTube video about how logs are enriched in BMC Helix Log Analytics.

Consider the following scenario where the sample collected log entry is:

“method”: “POST”,
 “IPAddress”:  “11.11.11.11”,
 “status”: “500”,
 “message”: “Internal Server Error”,

Enrichment policies: 

Policies selected by the algorithm: Enrichment policy 1, 3, and 5

where the policy selection criteria matched the logs: method Equals POST

The enrichments configured in these policies are applied based on the precedence value of these policies. Therefore, first the enrichment configured in Enrichment policy 1 is applied.

“method”: “POST”,
 “IPAddress”:  “11.11.11.11”,
 “status”: “500”,
 “message”: “Internal Server Error”,
 "country": "USA"

Now, the algorithm moves to Enrichment policy 3 with precedence 2.

“method”: “POST”,
“IPAddress”:  “11.11.11.11”,
“status”: “500”,
“message”: “Internal Server Error”,
"country": "USA",
 "city": "New York",
"office_location": "Main Street",

Next, the algorithm applies enrichment configured in Enrichment policy 5 with precedence value 3.

“method”: “POST”,
“IPAddress”:  “11.11.11.11”,
“status”: “500”,
“message”: “Internal Server Error”,
 "country": "United States of America",
"city": "New York",
"office_location": "Main Street",
"seat_location": "8W 5R 15",
"floor_number": "8",

Note that the country information changes after applying Enrichment policy 5—from USA to United States of America. It is because the country specified in the enrichment source configured in Enrichment policy 5 is United States of America. If the enrichment source for the same field is different in different enrichment policies (example country), the value from the last enrichment source is retained in the logs.