Creating collection policies to collect logs from AWS

Gather all application and services logs that are collected by the Amazon Cloudwatch service for search and analysis. CloudWatch monitors the Amazon Web Services (AWS) resources and the applications that run on AWS in real time.

The following image shows how logs are collected from your AWS accounts:

AWS_Log_Collection.png

Before you begin

Here are the steps that you must perform before configuring logs collection from AWS:

  • Download and install a connector. For more information, see Installing-and-managing-connectors. You can use any of the following connectors:
    • Linux
    • Windows 
  • Get the access and secret keys for your AWS account and ensure that you have access to the Cloudwatch service.
  • Plan and decide which logs you want to collect. You can collect logs at the region, group, or stream levels. Ensure that you have access to the appropriate regions, groups, and streams.

To collect logs from AWS

  1. Click the Collection menu and select Collection Policies.
  2. On the Collection Policies page, click Create.
  3. Enter the policy information:
    1. Enter a unique name and description for the policy.
    2. From the Collection Type list, select AWS.
    3. Enter the access and secret keys.
  4. Enter the following information about connector:
    1. From the Connector Type list, select Linux Connector or Windows Connector.
    2. In Connector Selection Criteria, create the connector selection criteria to identify connectors for collection.
      When you click in the box, you are prompted to make a selection. Each time you make a selection, you are progressively prompted to make another selection. 
      The selection criteria consist of an opening parenthesis, followed by the slot name, the operator, the slot value (which can be a string based on the type of slot selected), and the closing parenthesis. You can optionally select the logical operator AND or OR to add additional conditions. Specifying the opening and closing parentheses is optional.
      The connector fields available to create the selection criteria are status, name, version, host_name, ip, and tags. 
  5. Enter the following information about log collection:
    1. In the Configuration step, click Configure.
    2. In the Customize Logs Data panel, enter how frequently (a value in the range of 60 to 3600 seconds) )you want to refresh the collection interval.
    3. To filter the logs for collection, ensure that the Region/Group Filter check box is selected.

    4. Select a region and enter a group within the region, and a stream within the group from where you want to collect logs.

      To collect these logs

      Instructions

      Log streams whose names begin with East_Apps in the Apache_logs group.

      Enter Apache_logs and East_Apps in the Log Group Pefix and Log Stream Pefix fields.

      All logs of a region

      Leave asterisks in the Log Group Prefix and Log Stream Prefix fields.

      All logs of a stream in a group

      Enter the group name in the Log Group Prefix field and asterisks in the Log Stream Prefix field.

      All logs of a group or stream whose names begin with a common prefix

      Enter the prefixes in these fields.

      For example, to collect logs from all groups whose names begin with BMC, enter BMC in the Log Groups Prefix field.

      If the region for which you want to collect logs is not present in the list, contact BMC Support.

      Important

      Ensure that the log files that you will collect by configuring the log group and stream prefixes can be parsed by using the parsing rule that you have created. If the log files require different parsing rules, create different collection policies.

    5. To add multiple regions, groups, or streams, click the + sign.
    6. Save the configurations.
    7. In the Tags field, add the tags to identify the collected logs, such as AWS_Apache_logs.
  6. In the Parsing Rule step, select the parsing rule that you have created.
    If you have not created a parsing rule, click Create New. For more information, see Creating-a-parsing-rule.
  7. From the Filtering Rule list, select the filtering rule that you have created.
    If you have not created a filtering rule, click Create New. For more information, see Creating-a-filtering-rule.
  8. To start collecting logs, select the Enable Collection Policy check box.
  9. Click Save.
    The created policy is shown on the Collection Policies page.

  10. To edit, enable, disable, or delete a policy, use the Actions menu.

    Important

    When you edit a collection policy, ensure that you enter the access and secret keys again before saving it.

To verify log collection

To verify whether log collection has started, select Log Explorer > Discover. Use the tags or time range to view the collected logs.

To verify whether the parameters are correctly populated in the fluentd pipeline, go to /opt/td-agent/etc/data/<integration_ID>/pipeline. Open the file_log_pipeline.conf file by running the cat aws_logs_class_pipeline.conf command.

Learn more

Read the following blog to learn how you can enhance observability by using the AWS logs that you collect AWS cloud observability with Log Analytics.

Where to go from here

Configuring-logs

Generating-events-from-logs

Deriving-insights-from-logs

 

Tip: For faster searching, add an asterisk to the end of your partial query. Example: cert*