Collecting logs from cloud (AWS)

Gather all application and services logs that are collected by the Amazon Cloudwatch service for analysis in BMC Helix Log Analytics. CloudWatch monitors the Amazon Web Services (AWS) resources and the applications that run on AWS in real time.

The following image shows how logs are collected from your AWS accounts:

Before you begin

Here are the steps that you must perform before configuring logs collection from AWS:

• Download and install a connector in a Linux environment. For more information, see Downloading and installing connectors for third-party integrations.
• Get the access and secret keys for your AWS account and ensure that you have access to the Cloudwatch service.
• Plan and decide which logs you want to collect. You can collect logs at the region, group, or stream levels. Ensure that you have access to all those regions, groups, and streams.

To collect logs from AWS

The following video (2:40) illustrates the steps to configure log collection from AWS.

 https://youtu.be/zydF4utVdyM

1. In BMC Helix Developer Tools, open the Available integrations tab if it is not open by default.

2. On the Amazon Web Services tile, click Configure.
   On this tile, the build number of the product and the name of the entity that made the product available for configuration are displayed. 

3. Enter an integration name.

4. Select the connector that you downloaded and saved.
   If no connector is available, click Add Connector to download and install a connector.

5. Enter the access and secret keys.

6. In the Customize Entity Configuration section, click Configure.

7. In the Customize Logs Data window, enter how frequently you want to refresh the collection interval.
   Enter a value in the range of 60 to 3600 seconds.

8. In the Tags field, add the tags to identify the collected logs such as AWS_Apache_logs.

9. To filter the logs for collection, select the Region/Group Filter check box.

10. Select a region and enter a group within the region, and a stream within the group from where you want to collect logs.
    For example, in the US East (Ohio) region, you have a log group called Apache_logs in which you have log streams whose names begin with East_Apps. In this case, select the region and enter Apache_logs and East_Apps in the Log Group Pefix and Log Stream Pefix fields.  

    To collect all the logs of a region, leave asterisks in the Log Group Prefix and Log Stream Prefix fields. Similarly, to collect all logs of a stream in a group, enter the group name in the Log Group Prefix field and asterisks in the Log Stream Prefix field. To collect the logs of a group or stream whose names begin with a common prefix, enter the prefixes in these fields. For example, to collect logs from all groups whose names begin with BMC, enter BMC in the Log Groups Prefix field.
    If the region for which you want to collect logs is not present in the list, contact BMC Support.

11. To add multiple regions, groups, or streams, click the + sign.

12. Select the logs format present in your logs, such as Apache.

    The expression and time format are provided by default. Change if required. For more information, see Logs parsing and filtering.

13. To filter the logs, from the Log Filter list, select grep.
    For more information, see Logs parsing and filtering.

14. Save the entity configuration and then save the integration.
    You can view the added configuration by clicking Integrations and then Configured Integrations. 
    You can also see the logs being received, the connector status, and other details on the integration tile:
    

    Troubleshooting tip

    Scenario: The configured integration is showing a disconnected state.

    Solution: The server on which the connector is downloaded is down or it is not sending the heartbeats. Go to your virtual machine and ensure that the docker container is up and running.

To verify log collection

To verify whether log collection has started, in BMC Helix Log Analytics, select Log Explorer > Discover. Use the tags or time range to view the collected logs.

If the log format expression that you have configured is incorrect, in the /opt/bmc/connectors/<connector_name>/logs/fluent.log file, you get Response: 201.

To verify whether the parameters are correctly populated in the fluentd pipeline, go to /opt/bmc/connectors/<connector_name>/data/<integration_ID>/pipeline. Open the file_log_pipeline.conf file by running the cat file_log_pipeline.conf command.