Correlating messages into groups and patterns

Previous sections elaborated upon the basic usage of the BMC Defender Server to view and inspect message data. This new section changes the focus of the discussion to identify one of the most distinctive and powerful features of the BMC Defender Server program, that is how to correlate messages into groups and patterns, and how to take action when these patterns are detected.

Correlation is defined simply as finding arbitrary associations between messages. The term can be used broadly, or quite specifically. In the case of the BMC Defender Server, tools are provided to perform semantic correlation (as opposed to a statistical correlation) to detect the specific meaning of events in real time, so that consequential real-time actions (such as sending notifications, or executing other actions) can reliably occur. Additionally, the BMC Defender Server correlation feature is very useful in reducing the amount of incoming data to provide a clear picture of what is happening with managed devices.

The Message tab of the program, discussed in previous sections, performs minor correlation functions (such as cataloging messages by device, facility, and severity). The Correlation facility of the BMC Defender Server, described in this section, takes this basic correlation to a much higher and more sophisticated level, allowing users to detect complex patterns and trends.

Section summary and additional notes about correlating messages

• BMC Defender Server provides semantic correlation to detect the specific meaning of a stream of messages in real time, so that consequential real-time actions can occur.
• The purpose of the Correlation / Threads component is to categorize event messages, arriving at the BMC Defender Server, into Threads based upon simple or complex expressions
• The purpose of the Alerts / Counters component is to detect when thread counters (or other counters on the system) increment more or less a certain rate.
• The purpose of the Correlation / Trigger component is to detect when a particular event occurs, and then open a gate that enables further threading and actions.
• The preceding components work together, connected by the event log, to achieve a high degree of correlation. Users connect these components together, using the syslog as the common area of communication.
• Users add a correlation rule by clicking AddNew. Users can view, modify or delete a correlation rule by clicking Edit at the left of the correlation rule.
• Correlation screens all permit complex expressions that can be simple keywords or wildcards or might be a more complex expression supporting and, or, xor, not, and containing field matches and parenthetical nesting of expressions.
• All correlation expressions are case insensitive, and permit partial matches.
• The Threads and Actions components contain the ability to match a Trigger state of either Set or Clear. This furnishes a special ability to gate on or gate off the correlation counters and actions based upon a previously received event message, that establishes a context for subsequent messages.
• The Alerts function permits the user to establish thresholds on system counters, that sends syslog messages back into the system and opens tickets. This function includes Threshold hints and a Suggest function for the actual alert message (and ticket) text.
• The user can define macro expressions, that simplify the setup and maintenance of match expressions. The user creates a macro (representing a match expression) and then uses this macro in the Match Expression field of Threads, Triggers,  Actions or all.
• The user can define Address Groups that are used in Thread and Actions screens. These are similar to macros, but represent lists of addresses (possibly wildcarded) that are included or excluded from the match.
• Both macros and Address Groups use the naming convention @@name@@, where double at signs delimit the start and the end of the macro expression. However, macros cannot be used as address groups, and address groups cannot be used as macros.
• When implementing address groups, the user should be careful not to overuse this facility. Messages can be managed by device groups, but more typically, messages are managed by their textual content.
• The user cannot delete a correlation item that has dependents, without first removing or fixing these dependencies. The user can view the dependents of Threads, Triggers, macros, and other items by clicking Edit, and then clicking the View Dependents hyperlink at the bottom of the resulting edit screen.

This section provides information about the following topics:

• Basic correlation components of BMC Defender Server correlation
• Correlation rules and rule systems
• Configuration of correlation rules
• Match expressions
• Parse expression examples
• Macro expressions
• List macro expressions
• Address groups
• Thread groups
• Alert types and functions
• Alerting parameters
• Inserting alert variables
• Trigger, device alert, user alert, and pattern instances
• Custom alerts
• Correlation rule dependencies
• Correlation Speedups