Creating a data pattern with custom date format and new subpattern

This topic provides an example of creating a data pattern with a custom date format and new subpattern.

This example helps you answer questions such as:

• How do I create a custom date format?
• How do I create a new subpattern?
• What is the usage of the "details" field?
• How do I edit the primary pattern to accommodate the custom date format and new subpattern changes?

Use the following step-by-step instructions for creating this data pattern by navigating to Administration > Data Patterns >  Add Data Pattern.

• • Sample text
  • Step 1: Enter sample timestamp and click Auto-detect
  • Step 2: Create a custom date format
    • Part 1 Begin by creating a subpattern that can extract the date and time string from the data
    • Part 2 Create the custom date format
  • Step 3: Edit the primary pattern
  • Step 4: Enter sample text and verify the timestamp extraction
  • Step 5: Extract fields of interest
  • Step 6: Save the data pattern

Sample text

Step 1: Enter sample timestamp and click Auto-detect

Copy the timestamp from the sample text (or your log file) into the Sample Timestamp field and click Auto-detect. Use a timestamp that the product can use for determining the date format more precisely. In this case, you can use "01-SEP-2011 13:14:40" as the sample timestamp. This sample indicates a 24-hour date format. When you perform this step, you get an error message indicating that the date format is not detected (as shown in the following figure).

Error message asking to create a custom subpattern to capture the date format

(Click the image to expand it)

Step 2: Create a custom date format

The product did not recognize the date format from the sample timestamp. So you must create a custom date format and for that you need a corresponding subpattern.

Part 1 Begin by creating a subpattern that can extract the date and time string from the data

Before you begin creating a new subpattern, you can check if there is any existing subpattern that you can use for creating the new subpattern and save time.

Enter the sample timestamp "01-SEP-2011 13:14:40" into the search bar under the Manage Subpatterns section to search for closely matching subpatterns, as shown in the following figure.

Searching for a relevant subpattern

(Click the image to expand it)

In the preceding figure, you can see that the search results displayed for the subpattern are generic. Using a generic subpattern such as ".*" (catch-all) is not recommended, as this can result in errors.

As a closely matching subpattern is not found, you must create a new subpattern to match the date and time string.

You can also see if there is an existing date format that might be a closer match to the custom date format that you want to create. This can act as a starting point for creating the custom date format.

Based on the sample timestamp, you can look for date formats that start with "dd," followed by "MMM," followed by "yyyy." Suppose you select "dd MMM yyyy HH:mm:ss" as the date format. The primary pattern is updated as follows:

The following figure describes using a date format that closely matches the custom date format that you want to create, and correlating the primary pattern for that date format.

Studying a closely matching date format

(Click the image to expand it)

By now, you know that the subpattern name that matches the selected date format is "ITDADatePattern200."

Search for the subpattern with the same name, as shown in the following figure.

Searching for the subpattern that matches the relevant date format

(Click the image to expand it)

Copy the expression available for the subpattern that you searched to create a new subpattern.

Under the Add New Subpattern section, copy the subpattern expression in the Subpattern Expression field, as shown in the following figure. After doing this, edit the subpattern expression to add a dash character (-) in the syntax so that it matches the custom date format that you want to create.

Also, provide a subpattern name to identify the new subpattern and then enter the sample timestamp from the sample text.

Click Test to validate if the subpattern works as expected.

Creating a new subpattern and testing it

(Click the image to expand it)

After you are satisfied that the subpattern expression works correctly, add it to the list of existing subpatterns by clicking Add  next to the Test tab.

Part 2 Create the custom date format

From the Date Format list, select Custom, and then enter dd-MMM-yyyy HH:mm:ss as the new date format in the Custom box.

Adding a custom date format

(Click the image to expand it)

Step 3: Edit the primary pattern

After adding a custom date format, you must edit the primary pattern to use the new subpattern that corresponds with the custom date format. In the primary pattern, replace the string "<Custom Date subpattern name>" with the new subpattern name ("OracleListenerDateFormat") that you created in Step 2, Part 1, as shown in the following figure.

Also, delete the string "%{Data:_ignore}\s*" from the primary pattern, because you do not expect any text before the timestamp.

Editing the primary pattern

(Click the image to expand it)

Step 4: Enter sample text and verify the timestamp extraction

Copy the sample text from your log file in the Sample Text field, and click Preview to verify that the timestamp extraction is working correctly. The following figure shows the extracted timestamp entries.

Verifying the timestamp extraction

(Click the image to expand it)

Step 5: Extract fields of interest

The next step is to extract fields of interest to enable you to search more effectively.

In this case, no consistent pattern of fields is seen in the log file, so we can skip this step.

However, before you save the data pattern, you can experiment with the "details" field to understand its usage.

To do so, rename the "details" field to "details1" as shown in the following figure.

Renaming the "details" field

(Click the image to expand it)

In the preceding figure, you can see that the data in the second row (sample text) is not getting extracted as part of the "details" field. This is because the sample text has multiple lines in the data. For this, you must select the Multiline Entry check box.

Now if you preview again, you can see the fields extracted from the second line as well, as shown in the following figure.

Field extraction for the second row of sample text

(Click the image to expand it)

Because the second row in the sample text contains name=value pairs as part of the "details" field, these are automatically extracted as fields and do not need to be explicitly added to the primary pattern expression.

Step 6: Save the data pattern

Provide a logical name to identify the data pattern, select an appropriate category under which you want to add this data pattern, and click Create to save the data pattern, as shown in the following figure.

Specifying a name and category for the data pattern

(Click the image to expand it)