Creating data patterns
The Administration > Data Patterns page provides a list of default data patterns for most of the common log formats. For more information, see Default-data-patterns.
You can directly use these data patterns at the time of creating a data collector. However, if you do not find a data pattern that suits your needs, you can either clone an existing data pattern and customize it or create a new data pattern.
This topic contains the following information:
The following video (3:29) illustrates the data pattern creation process with an example.
The [confluence_iframe] macro is a standalone macro and it cannot be used inline. Click on this message for details. https://youtu.be/rumO-6iywNg
Data pattern creation process
At a high level, the data pattern creation process is made up of the following major tasks:
- Creation of date format: Determines the format in which the date and time string is read and extracted.
- Extraction of fields: Determines the custom fields that must be extracted from the data.
Creation of the date format can be done both, while creating a data collector and while creating a data pattern. The following table analyzes the benefits of creating the date format during the data collector creation versus the data pattern creation.
Analyzing when to create the date format
Creating a new data pattern
To create a new data pattern, access the data pattern wizard by navigating to Administration > Data Patterns > Add Data Pattern , and then follow these steps:
1. Provide sample data
This step allows you to provide sample text from your data file by using one of the following methods:
Copy and paste a few lines from your data file as sample text.
- Click Choose File and select the data file that you want to index.
The first 100 lines are displayed as sample text.
Note that by default the file encoding considered is UTF-8. If your data file uses a character set encoding other than UTF-8, then on the top-right of your screen, select an option available in the File Encoding list.
Click Next at the bottom-right of the wizard, to proceed to the next step.
2. Select date format
This step allows you to construct the date format – the format in which the date and time string must be interpreted and displayed on the Search page. This step forms one of the major steps involved in creation of a data pattern.
The wizard automatically detects the date format based on the sample data provided. You can decide to keep the date format suggested by the wizard or customize it to suit your needs.
You can customize the date format by changing the following selections:
When you are satisfied with the date format selection, click Next at the bottom-right of the wizard to proceed to the next step.
3. Select data pattern
This step is concerned with performing advanced functions such as extraction of fields and controlling the way in which the data will be processed. Search results are displayed based on how the data is processed.
If you want to create a basic data pattern containing the date format only, click Skip to move to the next step without advanced processing of fields. When you skip this step, the date and time string is extracted as per the date format that you defined in step 2, while rest of the data is extracted as free text. Also, default fields and name=value pairs available in the data are automatically extracted. For more information, see About-field-extraction.
If you want to continue with the field extraction, proceed as follows:
- In the sample data box, click the portion of data that you want to extract as a field.
On doing this, you can see that the selected sample is added under the sample data box. Next to the selected sample, provide the following details:
- Field type: Define the way in which these fields must be stored in the data store.
Storing fields with a field type enables you to use particular search commands to search fields effectively. For more information regarding the various options, see About-field-extraction. - Field name: Provide a name by which you want to identify the selected field.
The same name is displayed in the search results area.
For more information about best practices for adding fields, see Understanding-fields.
- Field type: Define the way in which these fields must be stored in the data store.
- Click Add Field
to confirm your selection.
- After adding all the fields, click Next at the bottom-right of the wizard to proceed to the next step.
Note that the product automatically detects the data portions that follow some pattern in the data. These portions are clickable in the sample data box and can be added as fields. The rest of the data is treated as miscellaneous details and is automatically extracted as free text; you cannot assign particular fields for this portion of the data. Name=value pairs occurring in the miscellaneous details are also automatically extracted as fields.
If you want to perform an even more advanced field extraction, then you can save and later edit the data pattern. In the edit mode, you can customize the primary pattern to suit your needs. To be able to customize the primary pattern, you need the knowledge of Java regular expressions. For more information about editing the data pattern, see Editing-or-cloning-data-patterns.
4. Review and save
This step allows you to review the data pattern information and save the data pattern
Use this step to validate details of the data pattern – such as the date format, the date locale, and the fields to be extracted. These details indicate the pattern that you just defined.
If you are satisfied with the pattern, provide the inputs described in the following table and click Save. Otherwise, click Previous to navigate backwards and make further modifications.
Data pattern naming inputs