Extracting fields

A lot of useful information is available inside the log message. You can extract this information as fields. 

The extracted fields are available on the Explorer > Discover page in the Available fields section.

Use the fields that you extract from a log message for the following purposes:

• To analyze logs for a particular field value.
• To create visualizations in the Explorer tab and BMC Helix Dashboards.
• To use these fields in other capabilities such as enrichment and alerts.

The following video (1:19) provides a brief overview of the field extraction feature.

 https://youtu.be/F2ZPfXyW0f4

To extract fields

1. Click the Configurations menu and select Field Extraction.
2. On the Field Extraction Policies page, click Create.
3. Enter a unique name such as ApplicationLogsFieldExtraction, and an optional description.

4. In the Precedence field, set a precedence number for the policy. This precedence number defines the priority for executing the policy.
   Note that a policy with a lower precedence number is executed first.
   If the incoming logs satisfy the selection criteria of multiple field extraction policies, the value in this field determines the execution order of the policies. The changes defined in the last field extraction policy that is applied to the incoming logs are saved.
   

   Example

   Field Extraction Policy 1

   Precedence: 2

   Sample log:

   127.0.0.1 xyz.bmc.com [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326 Critical

   Regular expression:

   (?<ip>\S+) (?<Hostname>\S+) (?<time>\[[^]]+]) (?<method>\"[^\"]+\") (?<status>\S+) (?<bytes>\S+) (?<loglevel>\S+)

   Example extracted field: Hostname: xyz.bmc.com

   Field Extraction Policy 2

   Precedence: 3

   Sample log:

   127.0.0.1 abc.bmc.com [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326 Alarm

   Regular expression:

   (?<ip>\S+) (?<Hostname>\S+) (?<time>\[[^]]+]) (?<method>\"[^\"]+\") (?<status>\S+) (?<bytes>\S+) (?<loglevel>\S+)

   Example extracted field: Hostname: abc.bmc.com

   Result:

   The precedence number of Field Extraction Policy 1 is lower than that of Field Extraction Policy 2. Therefore, the Field Extraction Policy 1 is executed first. As Field Extraction Policy 2 is executed last, the Hostname: abc.bmc.com field is extracted from the log message.

5. In the Policy Selection Criteria field, configure the condition to identify the logs from which the fields should be extracted.
   For example, kubernetes.container_name Equals log-processing-service.
6. In the Field Extraction Configuration section, from the Log Field list, select the field from which fields should be extracted.
   For example, message.
7. Copy the value of the field that you have selected in Log Field and paste it in the Log Field Value field.

8. In the Regular Expression field, enter the name group regular expression for Java to read and interpret the log entry that you have pasted in the Log Field Value field.

   Examples

   Example 1

   Regular expression

   (?<date>[0-9]{4}-[0-9]{2}-[0-9]{2}\s[0-9]{2}:[0-9]{2}:[0-9]{2}.[0-9]{3}\s\+[0-9]{4}\s)\[[^\]]*\]\s\[[^\]]*\]\s\-\s[^\"]+\s\d+\s[a-zA-Z]+\s\[[^\]]*\]\s\-\s[a-zA-Z\s]+:\s+[a-zA-Z]+,\s+[a-zA-Z]+:\s+[a-zA-Z]+=(?<Class>[a-zA-Z]+),\s+[a-zA-Z\s]+=(?<TenantId>[0-9]+),\s+[a-zA-Z\s]+=(?<EventId>[-.a-zA-Z0-9]*),\s+[a-zA-Z\s+]+=(?<EventSrcHostName>[-.a-z0-9]*)

   Sample input

   2022-09-21 10:59:57.607 +0000 [EventsEPSnull-0] [INFO ] - dfdc0b45-0fd3-4400-992b-115b56723d4d 1762135121 EventProcessorServiceLogger [com.bmc.truesight.saas.eps.eventprocessor.EPSEventsMessageListener:normalize:437] - Completed phase: Normalize, event: Class=EVENT, Tenant ID=1762135121, Event ID=eps.1762135121.16113217853307640.190465e0-c854-4009-92dd-008997ec01cb, Event source hostname=evt-ind-<random_number>.bmc.com, Is new event=false, Status=CLOSED, Source ID=EVENT_psr_event06_xuAZJzWJFA_Eny9jjRr6u.evt-ind-739164.bmc.com@3181.1655253786566.1824925970, time spent: 0ms

   Extracted fields

   Field name	Field value
   date	2022-09-21 10:59:57.607 +0000
   Class	EVENT
   TenantId	1762135121
   EventId	eps.1762135121.16113217853307640.190465e0-c854-4009-92dd-008997ec01cb
   EventSrcHostName	evt-ind-<random_number>.bmc.com

   Example 2

   Regular expression

   (?<dateTim10.42.68.174e>\[[^]]+\]) (?<ipAddress>\S+) (?<logLevel>\S+) (?<user>\S+) (?<httpMethod>\S+) (?<status>\S+)

   Sample input

   [22-09-2022 13:46:04.372:1] 11.11.11.111 ERROR root GET 501 Service not available. Please contact administrator.

   Extracted fields

   Field name	Field value
   dateTime	[22-09-2022 13:46:04.372:1]
   ipAddress	11.11.11.111
   logLevel	ERROR
   user	root
   httpMethod	GET
   status	501

9. Click Extract. 
   The fields that can be extracted are displayed in the Extracted Fields table and the Select Fields to Extract field.
10. (Optional) To remove a field from extraction, in the Select Fields to Extract field, remove the field.
11. Select Enable Policy.
12. Save the policy.
    View all your policies on the Field Extraction Policies page. To edit, enable, disable, or delete a policy, use the Actions menu.
    
    

Related topic

Field-extraction-policy-creation-and-management-endpoints-in-the-REST-API