Creating a data pattern and extracting fields of interest

This topic provides an example of creating a data pattern and extracting fields of interest.

This example helps you answer questions such as:

• How do I select an appropriate date format that matches the sample timestamp?
• How do I identify fields of interest?
• How do I edit a primary pattern to extract fields of interest?
• How do I assign the correct log type for each field?

Use the following step-by-step instructions for creating this data pattern by navigating to Administration > Data Patterns >  Add Data Pattern.

• • Sample text
  • Step 1: Enter sample timestamp and click Auto-detect
  • Step 2: Select the most relevant date format
  • Step 3: Verify if the timestamp is extracted correctly
  • Step 4: Edit the primary pattern to extract other fields
  • Step 5: Save the data pattern

Sample text

Upon analyzing the sample text, you can see that the data follows a particular pattern that can be used for extracting various fields. For more information about identifying fields in the data file, see Managing-data-patterns.

The following table provides a list of name=value pairs that you can extract by creating the data pattern:

Step 1: Enter sample timestamp and click Auto-detect

Copy the timestamp from the sample text (or your log file) into the Sample Timestamp field, and click Auto-detect to display a filtered list of date formats that are most relevant to the sample timestamp.

Entering the sample timestamp

(Click the image to expand it)

Step 2: Select the most relevant date format

As a result of performing step 1, in the Date Format list, you can see the date formats that match the sample timestamp:

• yyyy/MM/dd HH:mm:ss.SSS

• yyyy/MM/dd HH:mm:ss

The Auto-detect function does not show you exact matches; therefore, you can see the second option appearing in the list of date formats.

From the filtered list of date formats, select the first option, because that is most appropriate to the sample timestamp. This action updates the primary pattern, as shown in the following figure.

Selecting the date format that matches the sample timestamp

(Click the image to expand it)

Edit the primary pattern to remove “%{Data:_ignore}\s*”. This is necessary, because in the sample text, no data appears before the timestamp.

Add an opening square bracket ([) before the timestamp and a closing square bracket (]) after the timestamp preceded by a backward slash (\). Because the square brackets are special characters, you must precede them with the backward slash to escape them. This step is necessary to capture the timestamp string only and ignore the square brackets that appear around the timestamp in the sample text.

On doing this, the primary pattern is looks as shown in the following figure.

Editing the primary pattern

(Click the image to expand it)

Step 3: Verify if the timestamp is extracted correctly

Click Preview to verify if the timestamp is extracted as expected, as shown in the following figure:

Verifying the timestamp extraction

(Click the image to expand it)

Step 4: Edit the primary pattern to extract other fields

Edit the primary pattern to extract the "level" field, as shown in the following figure.

Editing the primary pattern to extract the "level" field

(Click the image to expand it)

Repeat this step to extract all the other fields of interest, and verify each time by clicking Preview. Also, change the field type for the "level" field from INTEGER to STRING. The "level" field type shows as INTEGER because an existing data pattern with the same field is marked with the field type INTEGER. In this case, the field type STRING is more relevant. For more information about field types, see Adding-a-new-data-pattern.

The following figure displays the "tid" (transaction ID) field extraction.

Editing the primary pattern to extract the "tid" field

(Click the image to expand it)

The following figure displays the "transactionGroup" field extraction as well as all field types.

Editing the primary pattern to extract the "transactionGroup" field

(Click the image to expand it)

Step 5: Save the data pattern

Provide a logical name to identify the data pattern, select an appropriate category under which to add this data pattern, and click Create to save the data pattern.