Query tab
You do not have to wait on the tab for the background query to finish. You can leave the Query tab at any time and then return to view the latest results or termination status of the query. The product retains the query results until you run the next query or until you click Clear, which allows you to run a long-running query, and then leave the page to perform other activities such as web browsing. You can return to the tab later to obtain the query results. You can run only one query at a time and you must terminate the current query before starting a new one. Each signed-in user can run their own queries without interfering with other users of the BMC Defender Server system.
The Query function is a rigorous way of extracting data from the BMC Defender Server system. It complements the Messages > Search function, but supports forensics and analytics more effectively than that function.
The default LogFile query is the most rigorous way of extracting data from the system, but because it scans all log files on the system for data, it is the slowest. For faster results, you can search a particular correlation thread. For more information, see Query file types.
The Query function is an excellent way to create ad hoc evidence associated with audits, produce evidence of suspicious behavior, produce evidence of failures of a particular application, or other situations requiring a static report for later reference.
The Reports > Query tab displays the following information:
Feature | Description |
|---|---|
Time | Date and time when the message was received |
Address | IP address of the system from which the message was received. Click the hyperlink in the Address column to view the Device Information tab, on which you can see more information about the system and hyperlinks to additional actions. |
Facility | Facility type of the message |
Matched Message | Messages that match the parameters that you define when running a query |
On the Reports > Query tab, click the following hyperlinks to perform additional functions:
- Search These Results—Fill the Match field, select the required options, and click Apply to search within the query results for a specific keyword, and sort the query results in ascending or descending order.
- Graph Results—Displays a bar graph with the number of messages received over a specified period. Click on the graph to see the Query Results For Selected Time tab that displays the query results for the selected time which is useful for forensics.
- History—Opens the Query History tab that displays the last 10 queries that you executed. On the Query History tab, click the numbered button against a past query status to see the past query and results. Enter a keyword in the Search Results History field and click Apply to run a search for the keyword on all the results on the Query History tab.
- Analyze—Opens the Analyze Results tab that groups the query results into a list of devices, users, facilities, severities, frequencies, and parse specifications contained in the messages. The Analyze Results tab displays an output that is similar to what you see on the Messages > Catalogs tab.
Saved Queries—Opens the Saved Queries tab that displays a list of queries that you saved earlier. The saved queries are specific to the signed-in user and simplify repetitive query operations. You can use the Saved Queries function to save queries for later use, and share saved queries. This provides a mechanism for configuring site-specific queries that might apply to certain complex forensic situations.
- Download HTML Report—Downloads the query report in HTML format
- Text Report—Downloads the query report in raw text format
- CSV Report—Downloads the query report in CSV format
- Generate PDF Report—Downloads the query report in PDF format
The Analyze and Search These Results hyperlinks, and the review and search history make the Query function a formidable application in diagnosing and documenting evidence.
This topic contains the following sections:
Fields on the Reports Query tab
To run a query, modify the following fields that are applicable to your query file type:
Field | Description |
|---|---|
Query Name | (Optional) Name of the query
|
Query File Type | Type of files to search Each value searches a different area of BMC Defender Server. For more information about the different query file types, see Query file types. |
Query Thread Name | Thread on which the query runs To view or edit available correlation threads, click Go To Threads Screen that displays the Correlation > Threads tab. |
Query User Name | Name of the user on which the query runs To view or edit available users, click Go To User Catalogs Screen that displays the Messages > Catalogs > Users tab. |
Query Device Name / Address | Device name or IP address on which the query runs To view or edit available devices, click Go To Device Catalogs Screen that displays the Messages > Catalogs > Devices tab. |
Query Start Date | Start day for the query |
Archive Start Date | Start day of the archive for the query |
Archive Span Days | Number of days of archives to run the query Enter the number of days before to the Archive Start Date. |
Start Aux File | Aux File on which the query starts |
Ticket Archive Start Date | Start day of the archived tickets |
Ticket Archive Span Days | Number of days of ticket archives to run the query Enter the number of days before to the Ticket Archive Start Date. |
Start File | Microsoft Windows file name to start the External and Replay type queries To edit the message parameters, click Go to Parms that displays the Messages > Config > Parms tab. |
Span Files | Number of days of files to run the External and Replay type queries Enter the number of days before to Start File. |
Query Span Days | For most query file types, the range of days that you want to query For example, if the start date is 2014-01-31 and Query Span Days is set to one day, then only messages on the selected date are scanned. |
Query Max Results | Maximum number of results that the query should display The default value is 50. The default setting is useful to quickly show results on systems in which a large number of results might exist for the Match Expression . When the specified number of results is achieved, the query tool terminates normally. |
Match IP Addr / Group | Range of messages matching a specific IP address or wildcard For example, if you had defined the @@windows_boxes@@ address group on the Correlation > Config > Address Groups page, then you can specify the @@windows_boxes@@ address group. |
Match Expression | Expression with which each message is compared. Messages that match are listed as results. The expression can be a keyword, wildcard, or logical combination of keywords and wildcards that are potentially parenthetically nested. The match expressions are identical to those found on the Correlation > Threads tab, with the exception that macros are not allowed as part of the expression. |
Query file types
When you run a query, you must select one of the following data sources in the Query File Type field to determine how the Query function operates:
Query file type | Description |
|---|---|
LogFiles | Operates on all the messages in the BMC Defender Server\logs folder, which contains a current list of all the messages received during the Keep Days interval (by default, 30 days). When you select LogFile as the query file type, the Query function operates like the Messages > Search function, except that the search takes place as a background process and the search uses complex match expressions. |
Thread-Catalog | Operates on a user-selected thread appearing on the Correlation > Threads tab. This might be the fastest way to run a query (given that the messages being queried all reside in a single defined thread on the system). |
User-Catalog | Operates on a managed user name appearing on the Messages > Catalogs > Users tab. The operator specifies the name of a valid user on the Query tab. |
Device-Catalog | Operates on a managed device name. The operator specifies the name of a valid device exactly as it appears on the Messages > Catalogs > Devices tab. |
Archives | Operates on all the messages contained in the gzipped archives residing in the BMC Defender Server\archive folder. When you select Archive as the query file type, the Query function searches all the archives on the system for the specified message. This can take a long time (even longer than a day) given that the BMC Defender Server archives can contain a 1,000 terabytes or more of message data. |
AuxFiles | Operates on the system Aux files, which is the filtered data. You can search this data by using the Messages > Aux tab. However, the Query function performs a more complete job of searching this data and uses complex match expressions to locate specific messages in these files. In this case, the Span Files setting spans the Aux files (and not the days, because the Aux files are always deleted at midnight). |
Tickets | Operates on tickets in the system. You can search this data by using the Advanced Ticket Search tab on the top-level Tickets tab. This provides an alternate method, including the searching of archived tickets on the system. |
External | Changes the mode of operation of the Query function. Rather than searching message data, the tool simply searches the .log, and .txt files of an external directory. By default, this is the directory in the BMC Defender Server\external folder, but the administrator can change this folder by using the Message > Config > Parms tab to be any folder on the system, including shared drives. This function expands the role of BMC Defender Server to include non-message data. In this case, the Span Files setting spans the external file names (and not the days). The Match IP Addr / Group input is not available for this type of file. |
Replay | Operates on BMC Defender log files (not arbitrary data, as is the case with the External setting). The Replay function permits the operator to review historical data associated with BMC Defender log files, such as to support analysts re-creating or investigating long past events or reviewing data that is not part of the BMC Defender system. Specifically, this selection operates on .log files placed in the external directory, as described earlier in this topic. The files must be generated by some copy of BMC Defender Server, and therefore contain the date, time, device, facility, severity, and message content. |
BigData-Logs | Allows large volumes of log data to be quickly searched by using different search methodology from the LogFiles search. BigData-Logs searches the log files on the system for the selected date range. Fewer search options are displayed compared with other Query File Type searches. The match expression supports regular expressions, which allows for very complex searches. BigData-Logs searches do not support lists or macros, which are available for other query types. You can duplicate search criteria from other query-type fields as part of a regular expression query. |
BigData-Archives | Searches the system log file for the selected date range. This search is similar to the Archives search, but uses a different search methodology that allows large volumes of log data to be quickly searched. Fewer search options are displayed compared with other Query File Type searches. The match expression supports regular expressions, which allows for very complex searches. BigData-Archives searches do not support lists or macros that are available for other query types. You can duplicate search criteria from other query-type fields as part of a regular expression query. |
The time taken for the completion of a query depends on several factors. If you search for a rarely occurring (or non-occurring) message across all log data or archive data on the system, the query might take a long time to complete. But if you search for a common message across a limited number of files, then you might receive the results within a few seconds.
When you run a BigData query, all results are returned. Unless you terminate the query, it runs until all the data is searched and all the results are returned.
Log files are searched directly, but before searching archives, you must uncompress them. The BigData searches do not have the same time, size, or line limitations that affect other searches. For long-running searches, the results page automatically refreshes periodically. New results are visible as they are found in approximately five-second intervals. When the search is complete, the results are available and searchable, just like for other query types.
Searching files on a solid-state drive (SSD) device has significant advantages over a standard hard drive device. The BigData search method is very disk intensive and maximizes the read/write speed while running. Searches that return fewer results return faster, while result files that take up several gigabytes increase the search time.
To run a query
- On the Reports > Query tab, click Generate.
- On the Run Forensics Search Query tab, modify the applicable fields. For more information, see Fields on the Reports Query tab.
To refine the query results, click Additional Query Parameters and modify the following fields.
Field
Description
Match Start Time
Specifies the start and end times of the search in HH : MM : SS format
The messages returned are delimited as after the match start and before the match end times (inclusive).
Match End Time
Specifies the start and end times of the search in HH : MM : SS format
The messages returned are delimited as after the match start and before the match end times (inclusive).
Match Facility
Specifies the match facility
Match Severity
Specifies the match severity
Trigger Expression
Finds an initial match expression and the search begins after the match expression is located in the log file
The trigger expression allows the user to search for messages within a specific context of a previous message. For example, the user might wish to find all messages associated with login failures that have been preceded by a specific connection to a VPN. The program first finds the trigger expression, and then finds all messages that follow.
The trigger expression is any valid match expression in a format identical to the main match expression. When you use a trigger expression, search results are limited to the same day as the trigger expression (that is, the Query function does not span multiple days).
Stop Expression
Final match expression, after which the query stops
Query Seek Order
Order in which the message data is searched.
The options are Newer to Older (the default) or Older to Newer.
This setting can be significant because Max Results limits the number of matches. If Max Results is set to 50, for example, and Query Seek Order is set to Newer to Older, then the 50 most recent matches are listed. Conversely, if set to Older to Newer, then the 50 oldest matches are listed. This also affects the trigger expression (if used).
Screen Auto-Refresh
How often you want the screen to refresh while the query is running
Each time the screen refreshes, the latest results, if any, are displayed. The value does not affect the update of the background process or the status line indicating the progress of the background process.
The default is 15 seconds. That is, new results are displayed every 15 seconds.
To enable email notifications for the query, click Query Report E-Mail Notifications and modify the following fields:
Field
Description
Enable E-Mail Notifications
Enables or disables the email notification for the generated query report
Send To E-Mail Address
Enter the email address to which you need the product to send the query report.
- Click Confirm.