Creating data patterns

The Administration > Data Patterns page provides a list of default data patterns for most of the common log formats. For more information, see Default-data-patterns.

You can directly use these data patterns at the time of creating a data collector. However, if you do not find a data pattern that suits your needs, you can either clone an existing data pattern and customize it or create a new data pattern.

This topic contains the following information:

The following video (3:29) illustrates the data pattern creation process with an example.

Note

The following video displays screens from an earlier version, however, the information provided in the video is still relevant to the current version of the product.

Related topics

Setting-up-data-patterns-to-extract-fields 

Collecting-data-into-the-system

Editing-or-cloning-data-patterns

Suggested reading

Sample-data-patterns

Sample subpatterns

Sample-date-formats

The [confluence_iframe] macro is a standalone macro and it cannot be used inline. Click on this message for details.
icon-play.png https://youtu.be/rumO-6iywNg

Data pattern creation process

At a high level, the data pattern creation process is made up of the following major tasks:

  • Creation of date format: Determines the format in which the date and time string is read and extracted.
  • Extraction of fields: Determines the custom fields that must be extracted from the data.

Creation of the date format can be done both, while creating a data collector and while creating a data pattern. The following table analyzes the benefits of creating the date format during the data collector creation versus the data pattern creation.

Analyzing when to create the date format

Note

If the data that you want to collect occurs in multiple formats or if you want greater control over the field extraction, you can edit the data pattern and customize the primary pattern to suit your needs. To be able to customize the primary pattern, you need the knowledge of Java regular expressions.

For more information about editing the data pattern, see Editing-or-cloning-data-patterns.

Creating a new data pattern

To create a new data pattern, access the data pattern wizard by navigating to Administration > Data Patterns > Add Data Pattern Plus icon.jpg, and then follow these steps:

dp wizard.png


1. Provide sample data

This step allows you to provide sample text from your data file by using one of the following methods:

  • Copy and paste a few lines from your data file as sample text.

    Recommendation

    BMC recommends you to provide enough samples of the data that you want to collect, to be able to produce a proper set of fields.

    Note that the variations occurring in your data might not be present in the first few lines and the last few lines of the file. For example, the first and the last sections of an application log file usually indicate the initialization and shut down of the application, and such data might not be a good sample.

  • Click Choose File and select the data file that you want to index.
    The first 100 lines are displayed as sample text.

Note that by default the file encoding considered is UTF-8. If your data file uses a character set encoding other than UTF-8, then on the top-right of your screen, select an option available in the File Encoding list.

1. Provide sample data

2. Select date format

3. Select data pattern

4. Review and save

Click Next at the bottom-right of the wizard, to proceed to the next step.

 Back to top

2. Select date format

This step allows you to construct the date format – the format in which the date and time string must be interpreted and displayed on the Search page. This step forms one of the major steps involved in creation of a data pattern.

The wizard automatically detects the date format based on the sample data provided. You can decide to keep the date format suggested by the wizard or customize it to suit your needs.

You can customize the date format by changing the following selections:

1. Provide sample data

2. Select date format

3. Select data pattern

4. Review and save

When you are satisfied with the date format selection, click Next at the bottom-right of the wizard to proceed to the next step.

Back to top

3. Select data pattern

This step is concerned with performing advanced functions such as extraction of fields and controlling the way in which the data will be processed. Search results are displayed based on how the data is processed.

If you want to create a basic data pattern containing the date format only, click Skip to move to the next step without advanced processing of fields. When you skip this step, the date and time string is extracted as per the date format that you defined in step 2, while rest of the data is extracted as free text. Also, default fields and name=value pairs available in the data are automatically extracted. For more information, see About-field-extraction.

1. Provide sample data

2. Select date format

3. Select data pattern

4. Review and save

If you want to continue with the field extraction, proceed as follows:

  1. In the sample data box, click the portion of data that you want to extract as a field.
    On doing this, you can see that the selected sample is added under the sample data box.

  2. Next to the selected sample, provide the following details:

    • Field type: Define the way in which these fields must be stored in the data store.
      Storing fields with a field type enables you to use particular search commands to search fields effectively. For more information regarding the various options, see About-field-extraction.
    • Field name: Provide a name by which you want to identify the selected field.
      The same name is displayed in the search results area.

    For more information about best practices for adding fields, see Understanding-fields.

  3. Click Add Field Plus icon.jpgto confirm your selection.
  4. After adding all the fields, click Next at the bottom-right of the wizard to proceed to the next step.

Note that the product automatically detects the data portions that follow some pattern in the data. These portions are clickable in the sample data box and can be added as fields. The rest of the data is treated as miscellaneous details and is automatically extracted as free text; you cannot assign particular fields for this portion of the data. Name=value pairs occurring in the miscellaneous details are also automatically extracted as fields.

If you want to perform an even more advanced field extraction, then you can save and later edit the data pattern. In the edit mode, you can customize the primary pattern to suit your needs. To be able to customize the primary pattern, you need the knowledge of Java regular expressions. For more information about editing the data pattern, see Editing-or-cloning-data-patterns.

Back to top

4. Review and save

This step allows you to review the data pattern information and save the data pattern

Use this step to validate details of the data pattern – such as the date format, the date locale, and the fields to be extracted. These details indicate the pattern that you just defined.

If you are satisfied with the pattern, provide the inputs described in the following table and click Save. Otherwise, click Previous to navigate backwards and make further modifications.

1. Provide sample data

2. Select date format

3. Select data pattern

4. Review and save

 Data pattern naming inputs

Back to top 

 

 

Tip: For faster searching, add an asterisk to the end of your partial query. Example: cert*