Using parse expressions

The parsing functions of the BMC Defender Server are used in a variety of places throughout the system. This section contains a detailed explanation of these Parse Expressions, their usage, their syntax, and general application notes.

Parse Expressions allow the program to extract information from a string or message, such as a particular field value, identified by a simple field number, match expression, or a more complex function. 

Parse Expressions are used in the following locations in the system.

• Catalog Analyze Screen—The operator can interactively specify a Parse Expression to extract data from any catalog. The operator clicks the Analyze link at the top of the Messages > Devices, Messages > Users, Correlation > Threads or other locations, then clicks the Parse link. The operator can then enter a Parse Expression to extract, enumerate, and tabulate data from the set of selected messages.
• Reports > Query Screen—The operator can specify a Parse Expression to extract data from any query generated by the Query function. The operator clicks on the Analyze link after running a query and then clicks the Parse link. The operator can then enter a Parse Expression to extract, enumerate, and tabulate data from the query results.
• Excel Reports—The operator can specify a Parse Expression to the Reports > Excel facility, clicking the Message Parsing Rules on the Add New and Edit screens. This allows you to populate a spreadsheet with the values parsed from a set of messages.
• Pivot Reports—The operator can specify a Parse Expression to the Reports > Pivot screen. Parse expressions provide one way of populating a pivot report with parsed values (the other way being to parse message data by field numbers, selected on the Add New or Edit screen).
• Dashboard Gadgets—The dashboard facility can display lists of parsed values using the Parse-Thread-Gadget.exe program that allows the operator to depict an occurrence count of parsed values from a set of messages.
• Thread (and other) Match Expressions—The operator can include a parse expression within a match expression, so that a set of messages is matched only if a certain parsed function exists within the message, for instance, if the fifth space-delimited word of the message contains an expected text string.
• User Discovery—The User Discovery Match Patterns, accessed through the Messages > Users > Advanced screen, permit the operator to specify parse expressions in order to discover user names within messages.

• Filter And Override Screens—The Message > Config > Filters and Messages > Config Overrides screens allow the operator to use parse expressions to match specific keywords. 

  Note

  This should be used cautiously to avoid unnecessary processing expense for these high-speed match patterns.

• Match Expression Evaluation Tool—BMC Defender includes a simple Match Expression Evaluation Tool, accessed through the More > Match Expr menu. This tool permits the operator to test expressions against user input strings, including the ability to test parse expressions.
• CPars.exe Command-Line Tool—All the functions in this section are supported by the CPars.exe command-line tool, found in the cli-bin folder of BMC Defender installations. This command-line tool allows simple scripting of complex parse functions for report generation, forensics, or test.

 This topic provides the following information:

• • How parsing works
  • Parse by field number
  • Parse by match expression
  • Parse expression examples

How parsing works

Parsing is applied to a set of target strings on the system. These target strings are just textual data, typically received messages (but possibly other types of textual data). In all cases, the parse function you specify is replaced by the parsed value as the first step in processing or matching messages.

Three distinct types of parsing can occur on the system, documented in detail in the various parts of this section:

• Parse By Field Number—You can simply specify the field number that is extracted from the target string, such as $3 being the third word of the target string. The leading $ can generally be omitted, but is always appropriate to include in the parse specification. This type of parsing is useful if the operator is working with highly structured data where the position of a particular word has a standard significance. In practice, this generally works with only a small set of very standardized messages.

• Parse By Match Expression—You can specify a match expression, where the asterisk character in the match expression specifies the particular text string segment that is to be extracted. 

  Employing a parse expression such as User Name: * returns the value that matches the asterisk character.This works with many different loosely structured data formats, especially messages in the standard BMC Defender Server format, as well as CEF, LEEF, and other common information formats. In practice, this technique gets used a lot for all types of structured and unstructured data.

• Parse By Function—You can specify a more complex $parse() function that performs special processing of the message. A majority of the information in this section is related to this technique; each parse function supported by the BMC Defender Server system is documented in detail.

In all cases, the parse expression is replaced by the parsed value. Herein, this is called the returned value of the parse expression; this simply means that the parsed value replaces the user furnished parse expression. 

The remaining portion of this section provides a description of each parse expression, its purpose and usage both as a parser and within a correlation rule. You should closely look at the examples of each second and might experiment with the match pattern using any of the facilities described above.

Parse by field number

A parse expression can be a simple numeric value. In this case, the number specifies the particular space-delimited word that is returned by the expression. If the numeric value is bigger than the number of space-delimited words in the expression, no value returns.

Parse by match expression

A parse expression can be a match expression, containing a single asterisk * character. The value returned is the portion of the string that matches the asterisk. If the parse expression does not match the target string, then no value is returned.

Parse expression examples

You can match any particular field using $N notation as described previously. Other more sophisticated parsing functions exist that permit a particular field to be matched only if the field has some property such as being an integer value, or being a URL. For more information about parse specifications and the multiple parsing functions, see Parse-expressions.