Extracting fields

A lot of useful information is available inside the log message. You can extract this information as fields. 

The extracted fields are available on the Explorer > Discover page in the Available fields section.


Use the fields that you extract from a log message for the following purposes:

  • To analyze logs for a particular field value.
  • To create visualizations in the Explorer tab and BMC Helix Dashboards.
  • To use these fields in other capabilities such as enrichment and alerts.

The following video (1:19) provides a brief overview of the field extraction feature.


 Watch the YouTube video about overview of the field extraction feature in BMC Helix Log Analytics.

To extract fields

  1. Click the Configurations menu and select Field Extraction.
  2. On the Field Extraction Policies page, click Create.
  3. Enter a unique name such as ApplicationLogsFieldExtraction, and an optional description.
  4. In the Precedence field, set a precedence number for the policy. This precedence number defines the priority for executing the policy.
    Note that a policy with a lower precedence number is executed first.
    If the incoming logs satisfy the selection criteria of multiple field extraction policies, the value in this field determines the execution order of the policies. The changes defined in the last field extraction policy that is applied to the incoming logs are saved.

    Field Extraction Policy 1

    Precedence: 2

    Sample log:

    127.0.0.1 xyz.bmc.com [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326 Critical

    Regular expression:

    (?<ip>\S+) (?<Hostname>\S+) (?<time>\[[^]]+]) (?<method>\"[^\"]+\") (?<status>\S+) (?<bytes>\S+) (?<loglevel>\S+)

    Example extracted field: Hostname: xyz.bmc.com

    Field Extraction Policy 2

    Precedence: 3

    Sample log:

    127.0.0.1 abc.bmc.com [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326 Alarm

    Regular expression:

    (?<ip>\S+) (?<Hostname>\S+) (?<time>\[[^]]+]) (?<method>\"[^\"]+\") (?<status>\S+) (?<bytes>\S+) (?<loglevel>\S+)

    Example extracted field: Hostname: abc.bmc.com

    Result:

    The precedence number of Field Extraction Policy 1 is lower than that of Field Extraction Policy 2. Therefore, the Field Extraction Policy 1 is executed first. As Field Extraction Policy 2 is executed last, the Hostname: abc.bmc.com field is extracted from the log message.

  5. In the Policy Selection Criteria field, configure the condition to identify the logs from which the fields should be extracted.
    For example, kubernetes.container_name Equals log-processing-service.

    Important

    The values that you enter for a field in the selection criteria are case-sensitive. For example, if the host name is WebServer.example.com, add the selection criteria as ( host_name Equals WebServer.example.com ). If you enter, ( host_name Equals webserver.example.com ), fields are not extracted.


  6. In the Field Extraction Configuration section, from the Log Field list, select the field from which fields should be extracted.
    For example, message.
  7. Copy the value of the field that you have selected in Log Field and paste it in the Log Field Value field.
  8. In the Regular Expression field, enter the name group regular expression for Java to read and interpret the log entry that you have pasted in the Log Field Value field.

    Example 1

    Regular expression

    (?<date>[0-9]{4}-[0-9]{2}-[0-9]{2}\s[0-9]{2}:[0-9]{2}:[0-9]{2}.[0-9]{3}\s\+[0-9]{4}\s)\[[^\]]*\]\s\[[^\]]*\]\s\-\s[^\"]+\s\d+\s[a-zA-Z]+\s\[[^\]]*\]\s\-\s[a-zA-Z\s]+:\s+[a-zA-Z]+,\s+[a-zA-Z]+:\s+[a-zA-Z]+=(?<Class>[a-zA-Z]+),\s+[a-zA-Z\s]+=(?<TenantId>[0-9]+),\s+[a-zA-Z\s]+=(?<EventId>[-.a-zA-Z0-9]*),\s+[a-zA-Z\s+]+=(?<EventSrcHostName>[-.a-z0-9]*)

    Sample input

    2022-09-21 10:59:57.607 +0000 [EventsEPSnull-0] [INFO ] - dfdc0b45-0fd3-4400-992b-115b56723d4d 1762135121 EventProcessorServiceLogger [com.bmc.truesight.saas.eps.eventprocessor.EPSEventsMessageListener:normalize:437] - Completed phase: Normalize, event: Class=EVENT, Tenant ID=1762135121, Event ID=eps.1762135121.16113217853307640.190465e0-c854-4009-92dd-008997ec01cb, Event source hostname=evt-ind-<random_number>.bmc.com, Is new event=false, Status=CLOSED, Source ID=EVENT_psr_event06_xuAZJzWJFA_Eny9jjRr6u.evt-ind-739164.bmc.com@3181.1655253786566.1824925970, time spent: 0ms

    Extracted fields

    Field nameField value
    date2022-09-21 10:59:57.607 +0000
    ClassEVENT
    TenantId1762135121
    EventIdeps.1762135121.16113217853307640.190465e0-c854-4009-92dd-008997ec01cb
    EventSrcHostNameevt-ind-<random_number>.bmc.com

    Example 2

    Regular expression

    (?<dateTim10.42.68.174e>\[[^]]+\]) (?<ipAddress>\S+) (?<logLevel>\S+) (?<user>\S+) (?<httpMethod>\S+) (?<status>\S+)

    Sample input

    [22-09-2022 13:46:04.372:1] 11.11.11.111 ERROR root GET 501 Service not available. Please contact administrator.

    Extracted fields

    Field nameField value
    dateTime[22-09-2022 13:46:04.372:1]
    ipAddress11.11.11.111
    logLevelERROR
    userroot
    httpMethodGET
    status501
  9. Click Extract
    The fields that can be extracted are displayed in the Extracted Fields table and the Select Fields to Extract field.
  10. (Optional) To remove a field from extraction, in the Select Fields to Extract field, remove the field.
  11. Select Enable Policy.
  12. Save the policy.
    View all your policies on the Field Extraction Policies page. To edit, enable, disable, or delete a policy, use the Actions menu.

Related topic

Field extraction policy creation and management endpoints in the REST API

Was this page helpful? Yes No Submitting... Thank you

Comments