Examples of creating a data pattern

Sample text

01-SEP-2011 12:26:48 * (CONNECT_DATA=(SID=xe)(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))) * (ADDRESS=(PROTOCOL=tcp)(HOST=127.0.0.1)(PORT=64741)) * establish * xe * 0
01-SEP-2011 12:26:48 * (CONNECT_DATA=(SID=xe)(CID=(PROGRAM=)(HOST=__jdbc__)(USER=))) * (ADDRESS=(PROTOCOL=tcp)(HOST=127.0.0.1)(PORT=64742)) * establish * xe * 0
01-SEP-2011 13:14:40 * service_update * xe * 0
01-SEP-2011 13:24:43 * service_update * xe * 0
No longer listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=127.0.0.1)(PORT=8080))(Presentation=HTTP)(Session=RAW))
01-SEP-2011 21:42:21 * service_died * xe * 12547
TNS-12547: TNS:lost contact
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=127.0.0.1)(PORT=8080))(Presentation=HTTP)(Session=RAW))

Step 1: Enter sample timestamp and click Auto-detect

Copy the timestamp from the sample text (or your log file) into the Sample Timestamp field and click Auto-detect. Use a timestamp that the product can use for determining the date format more precisely. In this case, you can use "01-SEP-2011 13:14:40" as the sample timestamp. This sample indicates a 24-hour date format. When you perform this step, you get an error message indicating that the date format is not detected (as shown in the following figure).

Error message asking to create a custom subpattern to capture the date format

(Click the image to expand it)

Step 2: Create a custom date format

The product did not recognize the date format from the sample timestamp. So you must create a custom date format and for that you need a corresponding subpattern.

Part 1 Begin by creating a subpattern that can extract the date and time string from the data

Before you begin creating a new subpattern, you can check if there is any existing subpattern that you can use for creating the new subpattern and save time.

Enter the sample timestamp "01-SEP-2011 13:14:40" into the search bar under the Manage Sub-Patterns section to search for closely matching subpatterns, as shown in the following figure.

Searching for a relevant subpattern

(Click the image to expand it)

In the preceding figure, you can see that the search results displayed for the subpattern are generic. Using a generic subpattern such as ".*" (catch-all) is not recommended, as this can result in errors.

As a closely matching subpattern is not found, you must create a new subpattern to match the date and time string.

You can also see if there is an existing date format that might be a closer match to the custom date format that you want to create. This can act as a starting point for creating the custom date format.

Based on the sample timestamp, you can look for date formats that start with "dd," followed by "MMM," followed by "yyyy." Suppose you select "dd MMM yyyy HH:mm:ss" as the date format. The primary pattern is updated as follows:

%{Data:_ignore}\s*%{ddMMMyyyyHHmmssDATETIME:timestamp}
\s*%{MultilineEntry:details}

The following figure describes using a date format that closely matches the custom date format that you want to create, and correlating the primary pattern for that date format.

Studying a closely matching date format

(Click the image to expand it)

By now, you know that the subpattern name that matches the selected date format is "ddMMMyyyyHHmmssDATETIME."

Search for the subpattern with the same name, as shown in the following figure.

Searching for the subpattern that matches the relevant date format

(Click the image to expand it)

Copy the expression available for the subpattern that you searched to create a new subpattern.

Under the Add New Sub-Pattern section, copy the subpattern expression in the Sub-Pattern Expression field, as shown in the following figure. After doing this, edit the subpattern expression to add a dash character (-) in the syntax so that it matches the custom date format that you want to create.

Also, provide a subpattern name to identify the new subpattern and then enter the sample timestamp from the sample text.

Click Test to validate if the subpattern works as expected.

Creating a new subpattern and testing it

(Click the image to expand it)

After you are satisfied that the subpattern expression works correctly, add it to the list of existing subpatterns by clicking Add  next to the Test tab.

Part 2 Create the custom date format

From the Date Format list, select Custom, and then enter dd-MMM-yyyy HH:mm:ss as the new date format in the Custom box.

Adding a custom date format

(Click the image to expand it)

Step 3: Edit the primary pattern

After adding a custom date format, you must edit the primary pattern to use the new subpattern that corresponds with the custom date format. In the primary pattern, replace the string "<Custom Date subpattern name>" with the new subpattern name ("OracleListenerDateFormat") that you created in Step 2, Part 1, as shown in the following figure.

Also, delete the string "%{Data:_ignore}\s*" from the primary pattern, because you do not expect any text before the timestamp.

Editing the primary pattern

(Click the image to expand it)

Step 4: Enter sample text and verify the timestamp extraction

Copy the sample text from your log file in the Sample Text field, and click Preview to verify that the timestamp extraction is working correctly. The following figure shows the extracted timestamp entries.

Verifying the timestamp extraction

(Click the image to expand it)

Step 5: Extract fields of interest

The next step is to extract fields of interest to enable you to search more effectively.

In this case, no consistent pattern of fields is seen in the log file, so we can skip this step.

However, before you save the data pattern, you can experiment with the "details" field to understand its usage.

To do so, rename the "details" field to "details1" as shown in the following figure.

Renaming the "details" field

(Click the image to expand it)

In the preceding figure, you can see that the data in the second row (sample text) is not getting extracted as part of the "details" field. This is because the sample text has multiple lines in the data. For this, you must select the Multiline Entry check box.

Now if you preview again, you can see the fields extracted from the second line as well, as shown in the following figure.

Field extraction for the second row of sample text

(Click the image to expand it)

Because the second row in the sample text contains name=value pairs as part of the "details" field, these are automatically extracted as fields and do not need to be explicitly added to the primary pattern expression.

Step 6: Save the data pattern

Provide a logical name to identify the data pattern, select an appropriate category under which you want to add this data pattern, and click Create to save the data pattern, as shown in the following figure.

Specifying a name and category for the data pattern

(Click the image to expand it)

Sample text

[2012/05/18 18:17:23.5310] [ INFO ] [TID: 0000003056] : CMDBEngInitialization -- Finished initializing Atrium CMDB.
[2012/05/24 10:24:19.8490] [WARNING] [TID: 0000003056] : CommitCacheForModifyWithoutLongWait -- Attempt to obtain EXCLUSIVE READ lock on cache for modification FAILED. m_cacheImpl: a5a45040 m_readLock: 2c4c7330. Error: 2

Upon analyzing the sample text, you can see that the data follows a particular pattern that can be used for extracting various fields. For more information, see Identifying fields in the data file.

The following table provides a list of name=value pairs that you can extract by creating the data pattern:

Timestamp = 2012/05/18 18:17:23.5310

Level = INFO or WARNING

TransactionId = 0000003056

TransactionName/Group = CMDBEngInitialization or CommitCacheForModifyWithoutLongWait

Step 1: Enter sample timestamp and click Auto-detect

Copy the timestamp from the sample text (or your log file) into the Sample Timestamp field, and click Auto-detect to display a filtered list of date formats that are most relevant to the sample timestamp.

Entering the sample timestamp

(Click the image to expand it)

Step 2: Select the most relevant date format

As a result of performing step 1, in the Date Format list, you can see the date formats that match the sample timestamp:

• yyyy/MM/dd HH:mm:ss.SSS

• yyyy/MM/dd HH:mm:ss

The Auto-detect function does not show you exact matches; therefore, you can see the second option appearing in the list of date formats.

From the filtered list of date formats, select the first option, because that is most appropriate to the sample timestamp. This action updates the primary pattern, as shown in the following figure.

Selecting the date format that matches the sample timestamp

(Click the image to expand it)

Edit the primary pattern to remove “%{Data:_ignore}\s*”. This is necessary, because in the sample text, no data appears before the timestamp.

Add an opening square bracket ([) before the timestamp and a closing square bracket (]) after the timestamp preceded by a backward slash (\). Because the square brackets are special characters, you must precede them with the backward slash to escape them. This step is necessary to capture the timestamp string only and ignore the square brackets that appear around the timestamp in the sample text.

On doing this, the primary pattern is looks as shown in the following figure.

Editing the primary pattern

(Click the image to expand it)

Step 3: Verify if the timestamp is extracted correctly

Click Preview to verify if the timestamp is extracted as expected, as shown in the following figure:

Verifying the timestamp extraction

(Click the image to expand it)

Step 4: Edit the primary pattern to extract other fields

Edit the primary pattern to extract the "level" field, as shown in the following figure.

Editing the primary pattern to extract the "level" field

(Click the image to expand it)

Repeat this step to extract all the other fields of interest, and verify each time by clicking Preview. Also, change the field type for the "level" field from INTEGER to STRING. The "level" field type shows as INTEGER because an existing data pattern with the same field is marked with the field type INTEGER. In this case, the field type STRING is more relevant. For more information about field types, see Adding a new data pattern.

The following figure displays the "tid" (transaction ID) field extraction.

Editing the primary pattern to extract the "tid" field

(Click the image to expand it)

The following figure displays the "transactionGroup" field extraction as well as all field types.

Editing the primary pattern to extract the "transactionGroup" field

(Click the image to expand it)

Step 5: Save the data pattern

Provide a logical name to identify the data pattern, select an appropriate category under which to add this data pattern, and click Create to save the data pattern.

Sample text

[5/15/12 16:14:07:113 PDT] 00000025 SystemErr     R com.ibm.ws.exception.RuntimeError:
java.lang.RuntimeException: java.lang.NoClassDefFoundError:
com.ibm.lang.management.MemoryMXBeanImpl (initialization failure)

[5/15/12 16:14:07:113 PDT] 00000025 SystemErr     R  at
com.ibm.ws.runtime.component.ApplicationMgrImpl.startApplication(ApplicationMgrImpl.java:789)

In the sample text, you can see sample data with two different formats. If you look at the second row in the sample text, you can see that it contains concrete information that you can easily use for extracting fields such as class name and line number. However, the first row does not follow the same pattern of information.

Step 1: Enter sample timestamp and click Auto-detect

Copy the timestamp from the sample text into the Sample Timestamp field, and click Auto-detect. Because the data pattern for this log file already exists, you can see the automatically detected date format "MM/dd/yy HH:mm:ss:SSS Z" that exactly matches the sample timestamp. Leave this date format unchanged and proceed further.

Copying the sample timestamp to automatically detect the appropriate date format

(Click the image to expand it)

Step 2: Editing the primary pattern to verify timestamp extraction

Edit the primary pattern to remove "%{Data:_ignore}\s*,” because the sample text contains no data before the timestamp to ignore.

Also, surround the timestamp with square brackets ([ ]) so that the primary pattern looks as follows:

\[%{IbmWebsphereTimestamp:timestamp}\]\s*%{MultilineEntry:details}

Surrounding the timestamp with square brackets is necessary to ignore the brackets and capture only the string (timestamp) that appears within the square brackets in the sample text.

Click Preview to verify that the timestamp was extracted correctly, as shown in the following figure.

Verifying the timestamp extraction

(Click the image to expand it)

Step 3: Editing the primary pattern to extract fields of interest

Before you edit the primary pattern for extracting fields of interest, you must understand the format of the data that appears in the two rows of sample text.

The following table shows a comparison of the data formats, to reveal similarities and differences. The sequence of the format displayed in the sample text (from left to right) is displayed as a numbered list (from top to bottom). The raw data and a corresponding description (added under the raw data) are displayed to help you understand the similarities and differences in the two formats.

Table of comparison for the two data formats

[5/15/12 16:14:07:113 PDT]

[5/15/12 16:14:07:113 PDT]

00000025

00000025

SystemErr

SystemErr

R

R

com.ibm.ws.exception.RuntimeError:
java.lang.RuntimeException: java.lang.
NoClassDefFoundError:
com.ibm.lang.management.MemoryMXBeanImpl 
(initialization failure)

com.ibm.ws.runtime.component.
ApplicationMgrImpl

startApplication

789

From the preceding table, you know that the primary pattern must be edited in such a way that the differences in the sample text formats are accommodated. To do so, proceed as follows.

Part 1 Edit the primary pattern to extract common fields

In the sample text, the timestamp is followed by the group ID that is common across the two data formats. To extract the group ID, edit the primary pattern as follows:

\[%{IbmWebsphereTimestamp:timestamp}\]\s%{Data:groupid}\s+%{MultilineEntry:details}

Click Preview to verify the "groupid" field extraction as shown in the following figure.

Extracting the group ID field

(Click the image to expand it)

The group ID is followed by a static text (SystemErr) that indicates the type of log file. This is not valuable information as you are already indexing the SystemErr.log file. Therefore, you do not need to extract this information as a separate field. You can add it as expected text after the group ID by editing the primary pattern as follows:

\[%{IbmWebsphereTimestamp:timestamp}\]\s%{Data:groupid}
\sSystemErr\s+%{MultilineEntry:details}

The static text is followed by the log level represented by "R" in the sample text.

Edit the primary pattern to extract this information (field "level") as follows:

\[%{IbmWebsphereTimestamp:timestamp}\]\s%{Data:groupid}
\sSystemErr\s+%{Data:level}\s+%{MultilineEntry:details}

Click Preview to verify the "level" field extraction as shown in the following figure.

Extracting the level field

(Click the image to expand it)

Part 2 Edit the primary pattern to extract different fields together

The data formats in the sample text differ after the log level information. Therefore, you must edit the primary pattern in a way that works for both data formats:

• For row 1: You can extract all the information that appears after the log level information as the "details" field.
• For row 2: You can extract multiple fields for this information by breaking it down into the fully qualified class name, followed by the function name, and finally the line number, as described in the preceding table of comparison.

Start by editing the primary pattern to extract fields for the second row of data, and then use the OR operator (|) to add the "details" field required for the first row of data, as follows:

\[%{IbmWebsphereTimestamp:timestamp}\]\s%{Data:groupid}\sSystemErr\s
+%{Data:level}\s+(?:at\s+%{GreedyData:class}\.%{Data:function}\
((?:.*:%{Data:linenum}|.*)\)|%{MultilineEntry:details})

Click Preview to verify the field extraction, as shown in the following figure.

Extracting additional fields added to accommodate differences in the data formats

(Click the image to expand it)

The "level" field type shows as INTEGER because an existing data pattern with the same field is marked with the field type INTEGER. In this case, the field type STRING is more relevant.

Click Cancel to exit without saving any changes.