How log enrichment works
When a log entry is received, the algorithm to apply enrichment filters all the enrichment policies where the selection criteria matches the logs. Next, the algorithm sorts these policies based on the precedence value. Note that lower the number, higher the precedence. Enrichments from all policies with highest precedence are applied first (for example, precedence value 1). Then, the algorithm moves on to the policies with higher precedence number (for example, precedence value 2). The algorithm moves in an ascending order of precedence value.
In addition to the precedence value, enrichment is applied in the order enrichment sources are selected in a policy. For example, in an enrichment policy, you configured enrichment sources in the following order:
- GeoIP
- LDAP
- DNS
First the GeoIP enrichment is applied, then LDAP, and then DNS.
While configuring the enrichment sources, ensure that you select the sources in correct order. For example, from an enrichment source, you get the country value. By using the country value, you want to get the city value from another enrichment source. In such cases, in an enrichment policy, first you select the enrichment source that provides the country value and then select the enrichment source that provides the city value. Based on your requirements, you can select these sources in different enrichment policies and configure their precedence accordingly.
The following video (2:34) illustrates how logs are enriched in BMC Helix Log Analytics.
Consider the following scenario where the sample collected log entry is:
“method”: “POST”,“IPAddress”: “11.11.11.11”,“status”: “500”,“message”: “Internal Server Error”,
Enrichment policies:
Enrichment policy 1 | Enrichment policy 2 | Enrichment policy 3 | Enrichment policy 4 | Enrichment policy 5 | |
Precedence | 1 | 1 | 2 | 2 | 3 |
Condition | method Equals POST | method Equals GET | method Equals POST | method Equals GET | method Equals POST |
Enrichment applied with the help of configured sources | country | country code | city and office location | city code and zip code | country, seat location, and floor number |
Policies selected by the algorithm: Enrichment policy 1, 3, and 5
where the policy selection criteria matched the logs: method Equals POST
The enrichments configured in these policies are applied based on the precedence value of these policies. Therefore, first enrichment configured in Enrichment policy 1 is applied.
“method”: “POST”,“IPAddress”: “11.11.11.11”,“status”: “500”,“message”: “Internal Server Error”,"country": "USA"
Now, the algorithm moves to Enrichment policy 3 with precedence 2.
“method”: “POST”,
“IPAddress”: “11.11.11.11”,
“status”: “500”,
“message”: “Internal Server Error”,
"country": "USA","city": "New York",
"office_location": "Main Street",
Next, the algorithm applies enrichment configured in Enrichment policy 5 with prcedence value 3.
“method”: “POST”,
“IPAddress”: “11.11.11.11”,
“status”: “500”,
“message”: “Internal Server Error”,"country": "United States of America",
"city": "New York",
"office_location": "Main Street",
"seat_location": "8W 5R 15",
"floor_number": "8",
Note that the country information changes after applying Enrichment policy 5—from USA to United States of America. It is because the country specified in the enrichment source configured in Enrichment policy 5 is United States of America. If the enrichment source for same field is different in different enrichment policies (example country), the value from the last enrichment source is retained in the logs.