The Administration > Data Patterns page provides a list of default data patterns for most of the common log formats. For more information, see Default data patterns.
You can directly use these data patterns at the time of creating a data collector. However, if you do not find a data pattern that suits your needs, you can either clone an existing data pattern and customize it or create a new data pattern.
This topic contains the following information:
Data pattern creation process
At a high level, the data pattern creation process is made up of the following major tasks:
Related topics
Suggested reading
- Creation of date format: Determines the format in which the date and time string is read and extracted.
- Extraction of fields: Determines the custom fields that must be extracted from the data.
Creation of the date format can be done both, while creating a data collector and while creating a data pattern. The following table analyzes the benefits of creating the date format during the data collector creation versus the data pattern creation.
Analyzing when to create the date format
| During data collector creation | During data pattern creation | Resource |
|---|---|---|
| You know the exact format in which the date and time string must be extracted. | You want to see the date format suggested by the wizard and then decide whether you want to keep the same format or customize it. | (See the individual pages based on the kind of data collector that you want to create). |
(Basic) You want to extract the date and time string only. Additional information: In addition to the date and time string, default fields and name=value pairs available in the data are automatically extracted by the product. For more information, see Understanding fields. | (Advanced) You want to extract the date and time string along with some custom fields. | Creating a new data pattern |
Note
If the data that you want to collect occurs in multiple formats or if you want greater control over the field extraction, you can edit the data pattern and customize the primary pattern to suit your needs. To be able to customize the primary pattern, you need the knowledge of Java regular expressions.
For more information about editing the data pattern, see Editing or cloning data patterns.
Creating a new data pattern
To create a new data pattern, access the data pattern wizard by navigating to Administration > Data Patterns > Add Data Pattern 
, and then follow these steps:
1. Provide sample data
This step allows you to provide sample text from your data file by using one of the following methods:
Copy and paste a few lines from your data file as sample text.
Recommendation
BMC recommends you to provide enough samples of the data that you want to collect, to be able to produce a proper set of fields.
Note that the variations occurring in your data might not be present in the first few lines and the last few lines of the file. For example, the first and the last sections of an application log file usually indicate the initialization and shut down of the application, and such data might not be a good sample.
- Click Choose File and select the data file that you want to index.
The first 100 lines are displayed as sample text.
Note that by default the file encoding considered is UTF-8. If your data file uses a character set encoding other than UTF-8, then on the top-right of your screen, select an option available in the File Encoding list.
Click Next at the bottom-right of the wizard, to proceed to the next step.
2. Select date format
This step allows you to construct the date format – the format in which the date and time string must be interpreted and displayed on the Search page. This step forms one of the major steps involved in creation of a data pattern.
The wizard automatically detects the date format based on the sample data provided. You can decide to keep the date format suggested by the wizard or customize it to suit your needs.
You can customize the date format by changing the following selections:
| Selection | Description |
|---|---|
| Sample date and time string | The date and time string used for detecting the date format is highlighted in the sample data. If the correct date and time string is not highlighted, change it by selecting one of the correct date and time strings displayed in the sample data, and click Detect. |
| Units | The date and time string is divided into logical portions. Under the box displaying the highlighted date and time string, the units representing the logical portions in the date and time string are displayed. The units are displayed in the same sequence as it appears in the sample text (from left to right). If you are not satisfied with the units suggested by the wizard, select the correct unit for each of the logical portions. |
| Date Locale | You can use this setting to enable reading the date and time string based on the language selected. Note that this setting only applies to those portions of the date and time string that consist letters (digits are not considered). By default, this value is set to English. You can manually select a language to override the default locale. For a list of languages supported, see Language information. |
When you are satisfied with the date format selection, click Next at the bottom-right of the wizard to proceed to the next step.
3. Select data pattern
This step is concerned with performing advanced functions such as extraction of fields and controlling the way in which the data will be processed. Search results are displayed based on how the data is processed.
If you want to create a basic data pattern containing the date format only, click Skip to move to the next step without advanced processing of fields. When you skip this step, the date and time string is extracted as per the date format that you defined in step 2, while rest of the data is extracted as free text. Also, default fields and name=value pairs available in the data are automatically extracted. For more information, see Understanding fields.
If you want to continue with the field extraction, proceed as follows:
- In the sample data box, click the portion of data that you want to extract as a field.
On doing this, you can see that the selected sample is added under the sample data box. - Next to the selected sample, provide the following details:
- Field type: Define the way in which these fields must be stored in the data store.
Storing fields with a field type enables you to use particular search commands to search fields effectively. For more information regarding the various options, see About field extraction. - Field name: Provide a name by which you want to identify the selected field.
The same name is displayed in the search results area.
- Field type: Define the way in which these fields must be stored in the data store.
- Click Add Field to confirm your selection.
- After adding all the fields, click Next at the bottom-right of the wizard to proceed to the next step.
Note that the product automatically detects the data portions that follow some pattern in the data. These portions are clickable in the sample data box and can be added as fields. The rest of the data is treated as miscellaneous details and is automatically extracted as free text; you cannot assign particular fields for this portion of the data. Name=value pairs occurring in the miscellaneous details are also automatically extracted as fields.
If you want to perform an even more advanced field extraction, then you can save and later edit the data pattern. In the edit mode, you can customize the primary pattern to suit your needs. To be able to customize the primary pattern, you need the knowledge of Java regular expressions. For more information about editing the data pattern, see Editing or cloning data patterns.
4. Review and save
This step allows you to review the data pattern information and save the data pattern
Use this step to validate details of the data pattern – such as the date format, the date locale, and the fields to be extracted. These details indicate the pattern that you just defined.
If you are satisfied with the pattern, provide the inputs described in the following table and click Save. Otherwise, click Previous to navigate backwards and make further modifications.
Data pattern naming inputs
| Field | Description |
|---|---|
| Date Format Name | Name to identify the date format. Defines the way in which the date and time string must be indexed and displayed in your search results. |
| Data Pattern Name | Name to identify the data pattern. Defines the way in which the data will be parsed and the fields will be extracted. |
| Data Pattern Category | Category in which you want to include this data pattern. Can be useful for administering and searching the data patterns by category. |
| Collect Multi-line Records (or Events) | (Optional) Indicates whether you want to capture data records (or events) that continue on many lines. By selecting this check box, you can view the entire data record (all the lines) in your search results. You need to expand the data record in the search results to see all the lines. By default, this check box is already selected. |
Overview
Content Tools
Apps
Reporter Replacement
com.bmc.atlassian.bmc-util-plugin-2.section.label
Tasks