Adding a new data pattern

Select one of the categories in which you want to include this data pattern.

This is useful for searching the data patterns by category.

Data Pattern Name

Name to identify the data pattern.

Copy and paste the timestamp from one of the lines in your data file into this field.

Copy and paste a few lines from your data file as sample text. Click Auto-detect to automatically detect the primary pattern and the date format.

If the primary pattern or date format cannot be automatically detected, then you must customize the primary pattern and specify a custom date format.

Provide the primary pattern (or the base pattern) to be used for the data pattern. The primary pattern is a collection of subpatterns.

Construct the primary pattern in the following format:

%{subTypeName1:tokenName1}%{subTypeName2:tokenName2}

You can add subpatterns to the primary pattern by clicking Add to Primary Pattern next to the subpatterns in the Manage Subpatterns section.

In the preceding syntax, tokenName refers to the field name that will be displayed in the search results on the Search tab. You cannot use underscore (_) in the field name.

Date Format

Select from a list of date/time formats that match the date format in the data file.

The list provides various formats that include days (d), months (M), years (y), hours (H), minutes (m), seconds (s), milliseconds (S), time zone (Z), day of the week (E), and so on.

Examples:

• HH:mm:ss
• MMM dd HH:mm:ss
• EEE MMM dd HH:mm:ss Z yyyy

You can either select from this list or create a customized date format. To do this, select Custom from the list and provide the date/time format in the Custom box. To use a custom date format, you must also create a custom subpattern corresponding to the date format and then use it in the primary pattern.

For more information, see Sample date formats.

Multi-line entry

Select this check box to capture raw data entries that continue on many lines.

This will enable you to see the entire data entry (all the lines) by expanding the entry in the search results area on the Search tab.

After specifying the primary pattern and date format, click Preview to validate the sample data entries and specify the field type for each of the fields that you specified in the primary pattern. When you click Preview, the Sample Log Validation Results box is displayed with the following information:

• The Sample log entries provide a list of sample data entries extracted.
• In the Choose Field Type area on the right, a list of extracted field names is displayed. Raw data entries in the name=value format are automatically extracted as fields. You need to assign the way in which these fields must be stored in the data store. You can select one of the following options:
  • INTEGER: Whole numbers from 0 to 2,147,483,647 (2,147,483,647 = 2³¹-1)
  • LONG: Long integer values from 0 to 9,223,372,036,854,775,807 (9,223,372,036,854,775,807 = 2⁶³-1)
  • STRING: Text or decimal value

If you do not use sample data entries and define the field type, then by default, all of the fields are saved as a string in the data store.

Storing fields with a particular field type enables you to use particular search commands to search fields effectively. For example, suppose that you are trying to find all requests that failed with various error response codes, such as:

• 404 Not Found
• 500 Internal Server Error
• 503 Service Unavailable

Suppose that you run the following search query on the response field:

clientip=* && (response >=400)

For this query to provide the correct results, response must be of the field type INTEGER or LONG.

Search for a subpattern from the list of default subpatterns by entering a name or value (or both) in the respective search bar and then clicking Search .

Suppose you want to see if a subpattern exists for a particular type of information in your data file; you can copy that entry as the value and see the subpatterns that match that value.

In the Actions column, click Add to Primary Pattern to add that subpattern to the primary pattern.

To remove an already added subpattern, click Remove next to the subpattern.

Add New Subpattern

You can add a new subpattern in the following way:

1. Subpattern Name: Enter a logical name to identify the subpattern.
2. Subpattern Expression: Enter the regular expression for the subpattern.
3. Actions: Enter sample text for validating the regular expression.
4. Click Test to verify if the sample text matches the subpattern regular expression.
5. Click Add .

For more information, see Sample subpatterns.

If you create a custom date format, then you must create a corresponding subpattern and use it in the primary pattern that you are constructing.

Impact: Without this, you cannot collect data using the particular data pattern.

The following fields are internal fields and might not be available for previewing to validate the sample data entries.

• timestamp

• details

• SEQUENCE_ID

• _ignore

Impact: These fields are not searchable.

While constructing a primary pattern, you cannot assign more than one subpattern for extracting the timestamp (field).

Instead of using more than one subpattern in the primary pattern, you can create a more complex subpattern that provides the unified value that you were trying to achieve with multiple subpatterns.

Impact: A data pattern containing such a primary pattern is invalid and is not usable for data-collection purposes.

%{Data:_ignore}\s*
%{%MonthDay}/%{Month}/%{Year} %{Time}:timestamp}
\s*%{MultilineEntry:details}

Primary pattern:

%{Mytimestamp:timestamp} \[%{Data:debuglevel}\] 
%{Data:component} - \[Thread=%{Data:threadid}\] 
%{Ip:clientip} - %{MultilineEntry:details}

Supporting subpattern:

Mytimestamp: %{DigitDay:day}\s+%{Month:month}\s+
%{FullYear}\s+%{Hour}:%{Minute}:%{Second}

You can assign the details field for miscellaneous information that you do not want to categorize with a specific field. All name=value pairs in the section to which this field is applied are extracted as fields.

Impact: At the time of indexing, the details field is ignored. If you do not specify the details field in your primary pattern, then the product looks for name=value pairs in the entire raw data record and extracts them as fields.

You can assign the _ignore field to the the portion of your data that you want to ignore and not categorize with a specific field. For example, if you want to ignore the extra digits (the milliseconds) in the custom date and timestamp 2014 Thu May 14 05:25:14.12321, you can assign this field to the extra digits. In this case, you can use the following subpattern to ignore the last two digits:

%{extraDigits:_ignore}

where,

extraDigits = \d{2}

Impact: The portion of data to which this field is applied is not categorized with a field.

For a custom date format, the letter X that indicates the ISO 8601 time zone is not supported. To enable you to capture the time zone, when you create a data collector, select an option in the Time Zone field.

Impact: You cannot collect data.