Deriving insights from logs
On the Discover tab, you can view log records according to your permissions. Administrators can restrict access to log records by using collection policies to restrict access to log records. If there are anomalous logs, the log records also display the log severity of the log anomaly.
The following video (2:53) illustrates how to analyze and visualize logs:
Overview of the Discover tab
In BMC Helix Log Analytics, use the Explorer > Discover tab to derive insights from logs. The following figure displays the Discover tab on the Explorer page:
You can perform the following actions on the Discover tab:
- Analyze logs
- Search for and filter logs
- Export logs
- View the index pattern
Analyzing logs
Log analysis involves searching, analyzing, and visualizing machine data generated by your IT systems and technology infrastructure to gain operational insights. Traditional data analytics tools are simply not built to handle the variety and volume of rapidly proliferating machine data.
In the Explorer > Discover tab, use the log messages to analyze logs.
After logs are collected through collection policies, they appear as log messages on the Discover tab. You can analyze these messages by performing the following actions:
- Searching for relevant logs
Use the search and filter options to locate relevant logs. You can refine the search results to perform analysis on the right logs. - Extracting fields
Extract fields from the log messages to effectively search for relevant log messages. - Enriching logs
Add meaningful information to log messages so that operators can resolve issues faster. For example, add host details such as the location of the host. - Detecting anomalies
Anomalies are rare patterns or abnormalities that indicate a deviation from the normal behavior of system performance. BMC Helix Log Analytics provides automated analysis with machine learning (ML)-based anomaly detection of abnormal or rare log patterns. You can analyze anomalous logs to debug application errors and ensure optimum performance. You can proactively find concerns or errors before they become a problem.
Searching for and filtering logs
Use the Discover tab to search for specific log messages and filter them.
Searching for alphanumeric strings in logs
On the Discover tab, use the following methods to search for a specific alphanumeric string:
- In the Search field, enter the string that you are looking for in a log field. The format is: field_name:"search string".
For example, to search for all logs that reported status 501 is reported, enter status:501. - Click Add Filter and select a field. Operators are available according to the data type of the field that you selected. Enter the string and save the filter.
For example, loglevel.keyword is error.
Filtering search results by time range and date
Use the following options to filter logs according to the time range and date:
- Specify days or hours for which you want to search results.
For example, search results for last 15 minutes or last 7 days. - Set specific date and time (absolute or specific).
For example, search results for Jul 18, 2022 18:00 hours till Jul 19, 2022 18:00 hours.
Supported time formats
The log generation time is saved in the @timestamp field. The time of the collected logs must be in the ISO 8601 ZULU format, for example, 2022-02-20T12:21:32.756Z. If the log generation time is specified in any other format, it is saved in the @@timestamp field, and the log collection time is saved in the @timestamp field. The log collection time is available in the Greenwich Mean Time (GMT) time zone.
If you are collecting logs by using external agents such as Logstash and Filebeat, the Epoch time format is supported. However, if you are collecting logs by using the Docker, Windows, or Linux connector, the Epoch time format is not supported.
Filtering search results based on fields
You can filter logs based on their fields. The fields identified in logs are displayed in the Available fields section. Click a field and select a value to filter logs based on the field.
Example
If you want to filter all logs that have a specific IP Address, perform the following steps:
- In the Available fields section, click the ipAddress field.
- Select an IP Address and click Filter.
Saving searches
Save the search query that you have created by using the search field, available fields, and time period fields. In future, access the saved search to get similar results.
- Click Save.
- Enter a name.
- To access the saved search, click Open.
Exporting logs as reports
Use the Explorer page to export logs in the CSV format. Use the exported log reports for enhanced data analysis, effective troubleshooting, and improved collaboration by sharing the reports with other stakeholders.
The export logs feature is supported only on OpenSearch 2.x.
You can perform the following actions from the Explorer page:
- Generate and download reports
- View previously generated reports
To generate a report and download it in the CSV format
- In BMC Helix Log Analytics, navigate to the Explorer tab.
Perform one of the following actions:
- If you have a saved search, click Open and click the saved search.
- If you don't have a saved search, click Save.
For more information about saving the search, see Deriving-insights-from-logs.
- Click Reporting.
- In the GENERATE AND DOWNLOAD menu, click Generate CSV.
The report is saved in the CSV format in the local directory that you specify.
You can export a maximum of 10000 logs per report.
To view previously generated reports
- In BMC Helix Log Analytics, navigate to the Explorer tab.
- Click Reporting.
- In the GENERATE AND DOWNLOAD menu, click View reports to view a list of generated reports.
You can download the generated reports in the CSV format.
Configuring report definitions
Customize log reports by using report definitions, where you can specify parameters such as the log data sources and time ranges to gather logs. You can also use report definitions to schedule reports to run later or multiple times at a scheduled cadence for consistent insights and operational efficiency improvements.
On the Reporting page, use the Report definitions section to select a report and edit or delete it.
To configure report definitions:
- In BMC Helix Log Analytics , navigate to the Explorer tab.
- Select Reporting > View reports.
The Reports page is displayed. - Under the Report definition section, click Create to create a report definition to generate customized and automated reports for effectively analyzing log data.
The create report definition page is displayed. - From the Reports settings section, enter the following parameters:
- In the Name field, enter the report name.
- (Optional) In the Description field, enter the report description.
- From the Select saved search list, select a saved search for which you want to generate the report.
- In the Time range field, select the time range for generating reports. You can choose the custom date range.
The default time range is last 30 minutes. - In the Report trigger field, you can trigger the report immediately or schedule the reports according to your requirements.
Overview of index patterns
An index pattern refers to the configuration that defines how logs and data are ingested, organized, and indexed within the system efficient log analysis. A default index pattern is already created in BMC Helix Log Analytics. All logs are collected under this index pattern. You can neither delete it nor create a new pattern.
By default, the rollover duration of the default index pattern is 3 days. Therefore, you can see data from the last 3 days on the Explorer page.
The following index patterns are also available:
- logarc_*
- logml-*
The logarc_* index pattern
After you enable log archiving, a new index pattern is added to the Discover page in the format logarc_*. All the logs collected since the time archiving is enabled for your tenant are shown in the new index pattern. The data before archiving is enabled continues to show in the earlier index pattern. The archived and restored data is available only in the new index pattern. Therefore, to analyze logs that are collected after archiving is enabled, use the logarc_* index pattern.
The logml-* index pattern
After an anomaly or rare pattern is detected in logs, it is reported in a new index pattern whose format is logml-*.
Learn more
Read the following blog to learn how logs help you understand the health of your environment, identify issues, and track their root cause Observability with logs to accelerate MTTR.