IX plan for log enrichment



Project information

Product and release

BMC Helix Log Analytics

Features

Log enrichment

Content developer

Swati Malhotra

Epic / Use cases

Enrich logs by using external sources like DNS servers, etc.

Design documents

Personas

Administrators

Master space

Whatfix self-help plan

NA

Product style sheet

Style-sheet-for-BMC-Helix-Operations-Management (Shall we follow BHOM stylesheet?)

SR: If it is a separate product and space, create a separate style sheet please. Thank you. Sure.


Use cases

Use case

Persona

Situation

Customer's information need

Delivery medium

Search keywords

Real-world example

Test case 

Testing needs R&D help?

Review comments

Enable Steve to enrich log entries by using external sources like DNS Servers, CSV files etc. so he can enrich logs with meaningful information that helps in faster troubleshooting.

Administrator







Happy Path

Can I add custom fields to the plain log entries to make them more meaningful?

In a Wiki topic (Enriching logs), we begin by telling users about the capability, various options available to enrich logs and then introduce the child topics. We also explain the whole process with an image.

We will create the following child topics:

  • We will have one landing topic for all enrichment sources (Adding enrichment sources).
  • A separate topic to explain how to get source and target enrichment fields (JSON format of logs) - a Pre-requisite topic.
  • One child topic for each supported enrichment source like DNS, GeoIP, IP Whitelist, IP Blacklist. On these child topics, if possible, we will include an example at the beginning and show the user the steps to achieve it.
  • One topic - Creating enrichment policy (task topic) - where we explain how to create an enrichment policy where you configure conditions when and which enrichment is applied.

log enhancement, custom tags to logs, log enrichment, adding fields to logs, DNS enrichment, LDAP enrichment, GeoIP enrichment, CSV enrichment, IP Whitelist enrichment, IP Blacklist enrichment

To get from RnD for each external source. (conversation started with dev)


SR:

logs shall get enriched with the inputs from external sources or CSV file.

Yes

 SR: In the use case, try to identify why Steve wants to to this.

SM: Done


What sources can I use to add such values to the log entries?

Intro on the Enriching logs topic







How do I use these sources?

Child topic for each enrichment source







Can I give a condition when these values be added to logs?

Creating an enrichment policy page







Can I continue to see these enhanced logs in Log Analytics dashboard?

Intro on the Enriching logs topic







Can I use these values in Helix Dashboard?

Intro on the Enriching logs topic







Can I add alerts using these values?

Intro on the Enriching logs topic







Can I create dashboards in Kibana by using these values?

Intro on the Enriching logs topic







What if I have some custom tags* like information that I want to add to logs?

*Tags are not added by using any source. It will be a custom requirement of a user.

Child topic for tags as a enrichment source (I will wait for the feature to come out to decide exactly how we will cover it in docs.)








Corner case










Troubleshooting










Wiki structure

In the outline, list the sets of tasks, concepts, and reference information that forms a complete workflow for the use case. Depending on the complexity of the feature, you might have multiple workflows or parent/child workflows. If possible, try to keep topics only three levels deep (L2 - L4).  If a topic contains a help context ID, review guidelines on IDD Central before renaming the topic.

Role

L1 - Branch

L2

L3

L4

L5

Subheadings

Topic type

Rich media

Writer notes

Review comments

All

Release notes and notices









SR: Please include the plans for the enhancement blurb.

SM: Sure. I had forgotten to include it here.



2022 enhancements















Enrich logs with external data




SR: "Enriching log messages with external data"

Are they enriching the logs, log messages, log records, or log events?

Enriching logs makes it sound like they are changing the log source. It that is the case, then okay.

Admin/Operator

Enriching logs for enhanced diagnostics






Process overview

Video


SR: "Enriching log messages [for monitoring]"

SM: I will confirm it with the PM. It is to make logs more meaningful.

SR: "...for enhanced diagnostics" is good, but are they enriching the logs, log messages, or log events?











SR: Please add the subheadings for the entire plan.

You don't need a subheading if you have a single task title, but do specify them if you have multiple sections. Follow the model for tasks.

SM: Done



Obtaining the JSON format of a log message





Task


I have added this topic at this location because the steps would be applicable for both Adding enrichment sources and Configuring enrichment policy topics

SR: If it is a task, you need a gerund title.

The current title does not identify what this is about. Is there any way you can bring out the context? Is this about gathering information for log enrichment?

SM: Done

The location is fine.

Thanks!












Admin/Operator


Adding enrichment sources




None

Process overview

None


SR: "Defining log enrichment sources"

Any time you have a parent topic, it serves as navigation for child topics. See the model for process overviews on choices for formatting this page.

SM: Please let me bring up a point. Users can add multiple enrichment sources. While defining a policy, they can select multiple enrichment sources. So, I was thinking that Adding is suited. Please suggest.

SR: Okay. I guess I thought this branch would include defining the policy too.




Adding a CSV file as an enrichment source



Before you begin

To add CSV enrichment source

Task



SR: For all of these topics, you need to be more specific about the source in the title. Also, I'm not clear if they can have one source for each type or multiple sources of each type.  Use singular or plural appropriately.

"Adding a CSV file | CSV files as an enrichment source"

SM: I have changed all of them to singular.




Adding a DNS server as an enrichment source



Before you begin

To add DNS enrichment source

Task

None


SR: "Adding|Connecting to a DNS server as an enrichment source"

SM: It is not only connecting to the server. It is more about the information that we get from the server and add it to logs.















Adding an LDAP server as an enrichment source



Before you begin

To add LDAP enrichment source

Task

None

















Adding a GeoIP server as an enrichment source



Before you begin

To add GeoIP enrichment source

Task



SR:  Is it GeoIP2? Google results show GeoIP as legacy. 

This term should go in a terminology sheet.




Adding IP Whitelist enrichment source



Before you begin

To add IP Whitelist enrichment source

Task



SR: Whitelist and blacklist are terms that the industry is moving away from. Reach out the editors to help with other choices you can share with your team.

SM: Sure. These are the terms used in the UI. However, the team is not taking it up in 22.1. When we take up these sources, we will work on having better terms.




Adding IP Blacklist enrichment source



Before you begin

To add IP Blacklist enrichment source

Task







Adding custom tags to logs



Before you begin

To add custom tags to logs

Task


As far as I understand, RnD might add this functionality to the Enrichment policy. I will be able to decide better about the use case when the policy UI is prepared.




Configuring policies to enrich logs




Before you begin

To add or modify a policy to enrich logs

To delete an enrichment policy

To enable or disable an enrichment policy (I am not sure about this topic yet)

Task



SR: "Configuring policies to enrich log events"

or

"Configuring log enrichment policies"

Because we have enrichment policies topics in other spaces, we want to have the word "log" in the title. I'd also do related topic links with current enrichment policies topic.

SM: I have changed the title. However, there is no existing enrichment policies topic. Which one are you mentioning? In other spaces as you mentioned? Please suggest.

SR: If this one is about defining the policy, then use that verb here.

Here are others:

Defining-event-policies-for-enrichment-correlation-notification-and-suppression

Not sure what the latest functionality is for TSOM

https://docs.bmc.com/docs/tsim105/creating-a-new-enrichment-policy-616456258.html


Estimates

Deliverable

Effort in person hours

Notes

UI, tooltip, or error message text review

4


Whatfix guided assistance (flow), self-help links, task list, or pop-ups



Video

30


Tutorial based on OOTB data



Tutorial or video



Wiki topic with graphics or interactive content

20 * 11 = 220 


Troubleshooting guide in collaboration with Support or link to KB article (for corner cases, written by Support)



Total

254


 

Tip: For faster searching, add an asterisk to the end of your partial query. Example: cert*

BMC Helix Log Analytics 23.3