Field Spec Delimiter and Normalization options

Clicking the Delimiter and Normalization Options hyperlink at the top of the Pivot Item Configuration Screen (discussed in the previous section) opens up a new table of special options that determine how the sample messages are parsed. The operator can specify special delimiters that affect the fields of the sample messages and can perform special pre-processing on each message, to modify the message format before it is parsed. These options control how the class of messages is parsed into separate fields. By default, the system parses words at spaces and phrases at double-quote marks. The operator can select other parsing options using check boxes.

The system distinguishes between words and phrases. Specifically, a phrase can have multiple field definitions. A phrase might contain multiple words, and the operator can configure the parser to select any particular word within the phrase (where words are delimited by space, comma, semi-colon or colon, or any combination of these). By default, the entire message is considered one phrase. However, the operator can break the message up into several phrases and operate on each phrase separately. This provides simplicity in the general case, as well as considerable power in parsing messages that might contain multiple phrases. A discussion of this feature is provided in further sections.

• Word delimiters—The operator can check either the space, comma, semi-colon, or colon of the word delimiters. The message is broken up into separate fields based upon the word delimiters.
• Phrase delimiters—Phrase delimiters are a special case of delimiters that break the message up into phrases based upon either double quote, bracket, parenthesis, or brace, characters. Each phrase can then be further parsed by word delimiters. (See Selecting-phrases-and-sub-fields-within-the-message.)

• Additional match expression—The operator can inspect the match expression (input on the previous Edit screen) and make adjustments to filter out messages that might be in the selected input thread that do not match a particular structure. This enables the operator with more flexibility in selecting messages of a particular type or class, necessary to consistently parse the message into fields. 

  Note

  This is the same match expression available on the parent Edit screen, included on the Config screen for convenience.

• Preprocessing options—The operator can click the Preprocessing Options hyperlink to apply a global match and replace pattern to each message immediately before it is parsed by the Word and Phrase delimiters. This screen allows the operator to specify a From pattern and To pattern, and specify how the From pattern is matched, either First, Last, or All occurrences of the From pattern. This allows the operator to insert delimiter characters in the message prior to any parsing, that might be required for certain highly complex messages.