Understanding fields

The product automatically identifies portions of the data that appear in the name=value format and displays them as fields. To effectively filter the indexed data and to perform advanced search operations (by using search commands), it is recommended that you define fields. Additionally, if you plan to analyze certain portions of your data, then it is recommended that you identify those portions and define fields for those portions of the data. You can define fields, at the time of creating a data pattern or at search-time by using search commands.

The following information provides an overview about how fields can be extracted and used:

About field extraction

At the time of data indexing, fields are automatically extracted. This process is known as field extraction.

Fields can be extracted in the following ways:

• Field extraction performed during the time of data collection and indexing.
• Field extraction performed during search time.

During data collection, the product automatically discovers and extracts name=value pairs from the data and displays it as fields in your search results. In addition to this, for every data entry that is indexed, the product assigns certain fields based on the inputs specified at the the time of creating a data collector or by certain default settings. These fields are treated as default fields and are displayed under the Filters panel > Fields section, on the Search page (after you perform a search).

When you assign a data pattern to a data collector, the tokens used in the data pattern are also extracted as fields and are searchable as name=value pairs. If the data pattern contains the details token, the product looks for the equals sign (=) to use as a delimiter to extract the name=value pairs.

Fields can also be extracted during search-time by using search commands such as extract and extractkv. Fields extracted during search-time are virtual fields that cannot be added to the Filters panel on the Search page.

For more information about the various ways in which fields are extracted, see About field extraction.

Learning how to identify fields in the data

Before you begin to extract fields by creating a data pattern, it is important that you learn how to identify fields that might be useful for performing effective searches.

Begin with analyzing your data file to see if the file follows any patterns that can be captured by means of fields. After finding the pattern, you need to identify if this pattern can be divided into small logical portions and grouped. For every group identified, you can create a field. This ensures that all information that can be categorized into a group is indexed and is available for search.

Suppose you want to extract fields from the following data:

In the preceding lines, every new line starts with the time stamp. And you will notice that the file follows a consistent pattern.

The following information (groups) appears in the preceding lines from left to right:

1. Time stamp
2. Component name
3. Debug information
4. Application message

For each of the preceding groups, you can assign a field.

Searching with fields

Fields are displayed on the Search page, under the Filters panel, and in the Fields section; fields with multiple values are displayed as a comma-separated list. For every field, a count of occurrences is displayed in parentheses () next to the field name. If the number of occurrences is too large, an approximate count is displayed with a plus sign (+). When you expand such a field, the values show the approximate count with an asterisk (*) next to them. The plus sign next to the field name and the asterisk next to the field values indicate that the count for those fields or values is an approximate number, not an exact number. If you add one of the field values to the search criteria and click Search, the accurate count is displayed next to that value.

You can delete all fields that you added manually to the Fields section under the Filters panel, except for the defaults.

In addition to the default fields, you can specify other custom fields to display under the Fields section. You can use these fields in your search query for narrowing down results. For more information, see Filtering your search results.

Best practices for adding fields

Use the following best practices while adding fields to create better searches:

• Use appropriate names while adding fields because field names are case sensitive.

  Note

  By default, field names and tag names are case sensitive. While searching, you cannot control case sensitivity for field names and tag names. However, you can control case sensitivity for field values and tag values. For more information, see Case-sensitive search and case-insensitive search.

• Avoid using special characters while naming fields. 

• Avoid creating duplicate fields with conflicting field types across two different data patterns.
  The following scenario provides guidelines to avoid duplicate fields with conflicting field types:

  Scenario: Creating fields for capturing particular errors

  Suppose you are creating a data pattern for the following logs.

  Log 1

  2012-08-16 03:15:34.65 spid60 Error: 1105, Severity: 17, State: 2.

  2012-08-16 03:15:34.65 spid60 Could not allocate space for object 'dbo.T8'.'PK__T8__02C769E9' in database 'ARSystem' because the 'PRIMARY' filegroup is full.

  Log 2

  [6/22/12 6:30:01:228 PDT] 00000024 webapp Error: com.ibm.ws.webcontainer.webapp.WebApp logServletError SRVE0293E: [Servlet Error]-[petstore]: org.springframework.transaction.CannotCreateTransactionException: Could not open JDBC Connection for transaction.

  [6/22/12 10:29:23:611 PDT] 00000024 servlet Error: com.ibm.ws.webcontainer.servlet.ServletWrapper service SRVE0014E: Uncaught service() exception root cause petstore: org.springframework.web.util.NestedServletException: Request processing failed.

  Log 1 contains an error code while Log 2 contains an error message.

  In this scenario, while creating the field for error in the data pattern, follow these guidelines:

  Correct:

  • Name the field indicating the error code as ErrorCode and assign the INTEGER field type.
  • Name the field indicating the error message as ErrorMessage and assign the STRING field type.

  Incorrect:

  • Name both the fields as Error and proceed as follows:
    
    • Assign the INTEGER field type to the field indicating the error code.
    • Assign the STRING field type to the field indicating the error message.

The following scenario provides guidelines to avoid duplicate fields with conflicting field types: