Page tree
Skip to end of metadata
Go to start of metadata

This topic provides information about editing or cloning a data pattern.

Notes

  • The information required while editing and cloning a data pattern is the same except that when you edit a data pattern, you cannot change the name.
  • You cannot edit a data pattern that was imported via a content pack. However, you can clone the data pattern, customize it, and then save it.

Editing a data pattern can be useful in the following scenarios:

  • For making minor modifications like changing the category, date locale, multiline entry setting, or the date format.
  • For advanced field extraction – this requires modifying the primary pattern and for this you need knowledge of Java regular expressions.

This topic contains the following information:

Before you begin

Editing or cloning a data pattern

Depending on whether you want to edit or clone a data pattern, navigate to the Administration > Data Patterns tab, and proceed as follows:

  • To edit a data pattern, click Edit Data Pattern, modify the information described in the following table, and click Update.
  • To clone a data pattern, click Clone Data Pattern , modify the information described in the following table, and click Create.

The following table provides information about the list of fields available while editing or cloning a data pattern. These fields are segregated into logical sections to aid reading.

Editing or cloning a data pattern

Field

Description

Data Pattern
Data pattern definition; includes the primary pattern and date format.
Category

Select one of the categories in which you want to include this data pattern.

This is useful for searching the data patterns by category.

Data Pattern Name

Name to identify the data pattern.

This field cannot be changed.

Sample Timestamp

Copy and paste the timestamp from one of the lines in your data file into this field.

Sample Text

Copy and paste a few lines from your data file as sample text. Click Auto-detect to automatically detect the primary pattern and the date format.

If the primary pattern or date format cannot be automatically detected, then you must customize the primary pattern and specify a custom date format.

Primary Pattern

Provide the primary pattern (or the base pattern) to be used for the data pattern. The primary pattern is a collection of subpatterns.

Construct the primary pattern in the following format:

%{subTypeName1:tokenName1}%{subTypeName2:tokenName2}

You can add subpatterns to the primary pattern by clicking Add to Primary Patternnext to the subpatterns in the Manage Subpatterns section.

In the preceding syntax, tokenName refers to the field name that will be displayed in the search results on the Search tab. You cannot use underscore (_) in the field name.

Date Format

Select from a list of date/time formats that match the date format in the data file.

The list provides various formats that include days (d), months (M), years (y), hours (H), minutes (m), seconds (s), milliseconds (S), time zone (Z), day of the week (E), and so on.

Examples:

  • HH:mm:ss
  • MMM dd HH:mm:ss
  • EEE MMM dd HH:mm:ss Z yyyy

You can either select from this list or create a customized date format. To do this, select Custom from the list and provide the date/time format in the Custom box. To use a custom date format, you must also create a custom subpattern corresponding to the date format and then use it in the primary pattern.

For more information, see Sample date formats.

Date Locale

(Optional) You can use this setting to enable reading the date and time string based on the language selected. Note that this setting only applies to those portions of the date and time string that consist letters (digits are not considered).

By default, this value is set to English.

You can manually select a language to override the default locale. For a list of languages supported, see Language information.

Multi-line entry

Select this check box to capture raw data entries that continue on many lines.

This will enable you to see the entire data entry (all the lines) by expanding the entry in the search results area on the Search tab.

Preview

After specifying the primary pattern and date format, click Preview to validate the sample data entries and specify the field type for each of the fields that you specified in the primary pattern. When you click Preview, the Sample Log Validation Results box is displayed with the following information:

  • The Sample log entries provide a list of sample data entries extracted.
  • In the Choose Field Type area on the right, a list of extracted field names is displayed. Raw data entries in the name=value format are automatically extracted as fields. You need to assign the way in which these fields must be stored in the data store. For more information regarding the various options, see About field extraction. Storing fields with a particular field type enables you to use particular search commands to search fields effectively.
Manage Subpatterns

Subpattern information (regular expressions) that parse a portion (one or more fields) of your data; constructed in the format, %{subPatternName1}{%{subPatternName2}.

Search subpatterns

Search for a subpattern from the list of default subpatterns by entering a name or value (or both) in the respective search bar and then clicking Search .

Suppose you want to see if a subpattern exists for a particular type of information in your data file; you can copy that entry as the value and see the subpatterns that match that value.

In the Actions column, click Add to Primary Pattern to add that subpattern to the primary pattern.

To remove an already added subpattern, click Remove next to the subpattern.

Add New Subpattern

You can add a new subpattern in the following way:

  1. Subpattern Name: Enter a logical name to identify the subpattern.
  2. Subpattern Expression: Enter the regular expression for the subpattern.
  3. Actions: Enter sample text for validating the regular expression.
  4. Click Test to verify if the sample text matches the subpattern regular expression.
  5. Click Add .

For more information, see Sample subpatterns.

UpdateClick Update to save your changes.

Notes about editing or cloning a data pattern

The following notes are important to keep in mind while editing or cloning a data pattern and will help you understand the impact on the search capabilities:

ActionDescription
1Creating a custom date format

If you create a custom date format, then you must create a corresponding subpattern and use it in the primary pattern that you are constructing.

Impact: Without this, you cannot collect data using the particular data pattern.

2Using internal fields

The following fields are internal fields and might not be available for previewing to validate the sample data entries.

  • timestamp
  • details
  • SEQUENCE_ID
  • _ignore

Impact: These fields are not searchable.

3Using more than one subpattern for defining the timestamp field

While constructing a primary pattern, you cannot assign more than one subpattern for extracting the timestamp (field).

Instead of using more than one subpattern in the primary pattern, you can create a more complex subpattern that provides the unified value that you were trying to achieve with multiple subpatterns.

Impact: A data pattern containing such a primary pattern is invalid and is not usable for data-collection purposes.

Example of an invalid primary pattern
%{Data:_ignore}\s*
%{%MonthDay}/%{Month}/%{Year} %{Time}:timestamp}
\s*%{MultilineEntry:details}
Example of a valid pattern example

Primary pattern:

%{Mytimestamp:timestamp} \[%{Data:debuglevel}\] 
%{Data:component} - \[Thread=%{Data:threadid}\]
%{Ip:clientip} - %{MultilineEntry:details}

Supporting subpattern:

Mytimestamp: %{DigitDay:day}\s+%{Month:month}\s+
%{FullYear}\s+%{Hour}:%{Minute}:%{Second}
4Using the details field for categorizing miscellaneous information in your data file.

You can assign the details field for miscellaneous information that you do not want to categorize with a specific field. All name=value pairs in the section to which this field is applied are extracted as fields.

Impact: At the time of indexing, the details field is ignored. If you do not specify the details field in your primary pattern, then the product looks for name=value pairs in the entire raw data record and extracts them as fields.

5Using the _ignore field for ignoring certain portions of data in your data file

You can assign the _ignore field to the the portion of your data that you want to ignore and not categorize with a specific field. For example, if you want to ignore the extra digits (the milliseconds) in the custom date and timestamp 2014 Thu May 14 05:25:14.12321, you can assign this field to the extra digits. In this case, you can use the following subpattern to ignore the last two digits:

%{extraDigits:_ignore}

where,

extraDigits = \d{2}

Impact: The portion of data to which this field is applied is not categorized with a field.

6Using the letter X while creating a custom date format.

For a custom date format, the letter X that indicates the ISO 8601 time zone is not supported. To enable you to capture the time zone, when you create a data collector, select an option in the Time Zone field.

Impact: You cannot collect data.