Collecting AWS logs

Gather all application and services logs that are collected by the Amazon Cloudwatch service for search and analysis. CloudWatch monitors the Amazon Web Services (AWS) resources and the applications that run on AWS in real time.

The following image shows how logs are collected from your AWS accounts:

Before you begin

Here are the steps that you must perform before configuring logs collection from AWS:

Download and install a connector. For more information, see Installing-and-managing-connectors. You can use any of the following connectors:
- Linux
- Windows
Get access and secret keys for your AWS account and ensure that you have access to the Cloudwatch service.
Plan and decide which logs you want to collect. You can collect logs at the region, group, or stream levels. Ensure that you have access to the appropriate regions, groups, and streams.
Create a parsing rule. For more information, see Creating-a-parsing-rule.
Create a filtering rule. For more information, see Creating-a-filtering-rule.

To collect AWS logs

In BMC Helix Log Analytics, use the Collection > Collection Policies > Create button and perform the following steps:

Add the policy information by performing the following steps:
1. Enter a unique name and description.
2. From the Collection Type list, select AWS
3. Enter the access and secret keys.
In the Connector configurations section, perform the following steps:
1. From the Connector Type list, select Linux Connector or Windows Connector.
2. In Connector Selection Criteria, create the connector selection criteria to identify connectors for collection.
  When you click in the box, you are prompted to make a selection. Each time you make a selection, you are progressively prompted to make another selection.
  The selection criteria consist of an opening parenthesis, followed by the slot name, the operator, the slot value (which can be a string based on the type of slot selected), and the closing parenthesis. You can optionally select the logical operator AND or OR to add additional conditions. Specifying the opening and closing parentheses is optional.
  The connector fields available to create the selection criteria are status, name, version, host_name, ip, and tags.
  Important
  The values that you enter for a field in the selection criteria are case-sensitive. For example, if the host name is WebServer.example.com, add the selection criteria as ( host_name Equals WebServer.example.com ). If you enter ( host_name Equals webserver.example.com ), the connector is not selected. To add case-insensitive values, use the Equals ignore case operator.

In the Configuration section, perform the following steps:

In the Configuration step, click Configure.
In the Customize Logs Data panel, enter how frequently (a value in the range of 60 to 3600 seconds) )you want to refresh the collection interval.
To filter the logs for collection, ensure that the Region/Group Filter check box is selected.

Select a region and enter a group within the region, and a stream within the group from where you want to collect logs.

To collect these logs	Instructions
Log streams whose names begin with East_Apps in the Apache_logs group.	Enter Apache_logs and East_Apps in the Log Group Pefix and Log Stream Pefix fields.
All logs of a region	Leave asterisks in the Log Group Prefix and Log Stream Prefix fields.
All logs of a stream in a group	Enter the group name in the Log Group Prefix field and asterisks in the Log Stream Prefix field.
All logs of a group or stream whose names begin with a common prefix	Enter the prefixes in these fields. For example, to collect logs from all groups whose names begin with BMC, enter BMC in the Log Groups Prefix field.

If the region for which you want to collect logs is not present in the list, contact BMC Support.

Important

Ensure that the log files that you will collect by configuring the log group and stream prefixes can be parsed by using the parsing rule that you have created. If the log files require different parsing rules, create different collection policies.

To add multiple regions, groups, or streams, click the + sign.
Save the configurations.
In the Tags field, add the tags to identify the collected logs, such as AWS_Apache_logs.
In Fields, enter custom information that you want to add to logs in the form of key-value pairs.
For example, Key: applicationContext; Value: Apache.

In the Parsing Rule step, select the parsing rule that you have created.
If you have not created a parsing rule, see Creating-a-parsing-rule for instructions.
From the Filtering Rule list, select the filtering rule that you have created.
If you have not created a filtering rule, see Creating-a-filtering-rule for instructions.
From the User group list, select one or more user groups to assign to the collection policy.
Users associated with this user group can see the data collected by this collection policy.
Enable and save the policy by performing the following steps:
1. To start collecting logs, select the Enable Collection Policy check box.
  You can choose to enable the collection policy later.
2. Click Save.
  The created policy is shown on the Collection Policies page. Use the Actions menu to edit, enable (or disable), and delete the policy.

To verify the log collection

To verify whether the log collection has started, select Explorer > Discover. Use the tags or time range to view the collected logs.

To verify whether the parameters are correctly populated in the fluentd pipeline, go to /opt/td-agent/etc/data/<integration_ID>/pipeline. Open the file_log_pipeline.conf file by running the cat aws_logs_class_pipeline.conf command.

To view the out-of-the-box AWS dashboard in BMC Helix Dashboards

Click the Dashboards menu.
Select Manage Dashboards.
The Log Analytics folder is displayed.
Click the AWS dashboard.

Learn more

Read the following blog to learn how you can enhance observability by using the AWS logs that you collect AWS cloud observability with Log Analytics.

Where to go from here

Configuring-logs

Generating-alerts-from-logs

Deriving-insights-from-logs