Understanding fields

Fields represent small portions of your data displayed as name=value pairings, such as Source=<host-name>.

Fields add meaning to the data and help you search more effectively. They help you classify particular portions in your data that might otherwise go unnoticed. Fields act as the building blocks for running search commands and creating dashboards.

Fields can be identified in the following ways:

Automatically identified by the product
Defined by a user while creating a data pattern
Defined by a user at search-time (virtual fields)
Defined in the data patterns included in the content packs that you are using

About field extraction

At the time of data indexing, fields are automatically extracted. This process is known as field extraction.

Fields can be extracted in the following ways:

Field extraction performed during the time of data collection and indexing.
Field extraction performed during search time.

During data collection, the product automatically discovers and extracts name=value pairs from the data and displays it as fields in your search results. In addition to this, for every data entry that is indexed, the product assigns certain fields based on the inputs specified at the the time of creating a data collector or by certain default settings. These fields are treated as default fields and are displayed under the Filters panel > Fields section, on the Search page (after you perform a search).

When you assign a data pattern to a data collector, the tokens used in the data pattern are also extracted as fields and are searchable as name=value pairs. If the data pattern contains the details token, the product looks for the equals sign (=) to use as a delimiter to extract the name=value pairs.

Fields can also be extracted during search-time by using search commands such as extract and extractkv. Fields extracted during search-time are virtual fields that cannot be added to the Filters panel on the Search page.

Recommendation

If you want to capture and extract particular portions in your data that are important to your needs, but which are not automatically discovered and extracted, then you need to define fields for such portions. You can define fields by performing a data pattern based field extraction while creating a data pattern or you can perform a search-time field extraction by using particular search commands.

BMC recommends you to plan your fields beforehand and perform a data pattern based field extraction.

For more information about the various ways in which fields are extracted, see About-field-extraction.

Learning how to identify fields in the data

Before you begin to extract fields by creating a data pattern, it is important that you learn how to identify fields that might be useful for performing effective searches.

Begin with analyzing your data file to see if the file follows any patterns that can be captured by means of fields. After finding the pattern, you need to identify if this pattern can be divided into small logical portions and grouped. For every group identified, you can create a field. This ensures that all information that can be categorized into a group is indexed and is available for search.

Suppose you want to extract fields from the following data:

Apr 24, 2014 03:16:40 PM configservice WARN: No configuration found.

Apr 24, 2014 03:16:44 PM dbservice INFO: Starting Schema Apr 24, 2014 03:16:44 PM
dbservice INFO: CONFIGURATIONS table exists in the database.

Apr 24, 2014 03:16:44 PM dbservice INFO: Executing Query to check init property:
select * from CONFIGURATIONS where userName = 'admin' and propertyName ='init'

Apr 24, 2014 03:16:44 PM dbservice INFO: init property exists in CONFIGURATIONS table.

In the preceding lines, every new line starts with the time stamp. And you will notice that the file follows a consistent pattern.

The following information (groups) appears in the preceding lines from left to right:

Time stamp
Component name
Debug information
Application message

For each of the preceding groups, you can assign a field.

Tip

If your data contains miscellaneous details that cannot be categorized under a specific field, then you can assign the details field for such miscellaneous details. At the time of indexing, the details field is ignored, but all name=value pairs appearing in the section to which this field is applied are automatically extracted as fields.

Additional information

In the preceding sample lines, suppose you think that extracting the word "CONFIGURATIONS" as a field might be useful. But this word does not appear consistently on each line.

If you decide to extract this portion as a field, the product might skip the other lines that do not contain the word, in which case those lines will not be available for searching. In this scenario, you can use one of the following options:

Extract this field at search-time by using search commands such as extract.
Extract this field while creating the data pattern. And then while creating the data collector, ensure that you enable the Best Effort Collection setting under the advanced options.

Searching with fields

Fields are displayed on the Search page, under the Filters panel, and in the Fields section; fields with multiple values are displayed as a comma-separated list. For every field, a count of occurrences is displayed in parentheses () next to the field name. If the number of occurrences is too large, an approximate count is displayed with a plus sign (+). When you expand such a field, the values show the approximate count with an asterisk (*) next to them. The plus sign next to the field name and the asterisk next to the field values indicate that the count for those fields or values is an approximate number, not an exact number. If you add one of the field values to the search criteria and click Search, the accurate count is displayed next to that value.

You can delete all fields that you added manually to the Fields section under the Filters panel, except for the defaults.

In addition to the default fields, you can specify other custom fields to display under the Fields section. You can use these fields in your search query for narrowing down results. For more information, see Filtering-your-search-results.

Note

Fields are extremely useful but have a cost; therefore, it is recommended that you do not add too many fields to the Fields section, available under the Filters panel on the search page.

Adding too many fields can cost a fair amount of CPU, memory, and disk I/O overhead.

Best practices for adding fields

Use the following best practices while adding fields to create better searches:

Use appropriate names while adding fields because field names are case sensitive.
Note
By default, field names and tag names are case sensitive. While searching, you cannot control case sensitivity for field names and tag names. However, you can control case sensitivity for field values and tag values. For more information, see Case-sensitive search and case-insensitive search.
Avoid using special characters while naming fields.
Avoid creating duplicate fields with conflicting field types across two different data patterns.
The following scenario provides guidelines to avoid duplicate fields with conflicting field types:
Scenario: Creating fields for capturing particular errors
Suppose you are creating a data pattern for the following logs.
Log 1
2012-08-16 03:15:34.65 spid60 Error: 1105, Severity: 17, State: 2.
2012-08-16 03:15:34.65 spid60 Could not allocate space for object 'dbo.T8'.'PK__T8__02C769E9' in database 'ARSystem' because the 'PRIMARY' filegroup is full.
Log 2
[6/22/12 6:30:01:228 PDT] 00000024 webapp Error: com.ibm.ws.webcontainer.webapp.WebApp logServletError SRVE0293E: [Servlet Error]-[petstore]: org.springframework.transaction.CannotCreateTransactionException: Could not open JDBC Connection for transaction.
[6/22/12 10:29:23:611 PDT] 00000024 servlet Error: com.ibm.ws.webcontainer.servlet.ServletWrapper service SRVE0014E: Uncaught service() exception root cause petstore: org.springframework.web.util.NestedServletException: Request processing failed.
Log 1 contains an error code while Log 2 contains an error message.
In this scenario, while creating the field for error in the data pattern, follow these guidelines:
Correct:
- Name the field indicating the error code as ErrorCode and assign the INTEGER field type.
- Name the field indicating the error message as ErrorMessage and assign the STRING field type.
Incorrect:
- Name both the fields as Error and proceed as follows:
  Assign the INTEGER field type to the field indicating the error code.
  Assign the STRING field type to the field indicating the error message.

The following scenario provides guidelines to avoid duplicate fields with conflicting field types: