Amazon Web Services connector

Use the Amazon Web Services cloud connector to collect the resource utilization data of the services that are provisioned in the Amazon Web Services (AWS) cloud. You can use this connector to:

  • Collect the cost data of all the services
  • Collect the cost and usage data of your virtual machines (EC2 instances)

BMC Helix Cloud Costuses these data points to provide cost insights and forecasting estimations to optimize your cloud costs by providing recommendations.

The connector supports data collection for the following AWS subscription types:

  • AWS default
  • AWS GovCloud (US)

The following video (8:24) provides information about configuring the Amazon Web Services connector in BMC Helix Cloud Cost.

 https://youtu.be/3WyIdmviMiI

Collecting data by using the AWS cloud connector

To collect data by using the AWS cloud connector, do the following tasks:

I. Complete the preconfiguration tasks.

II. Configure the connector.

Step I. Complete the preconfiguration tasks

The connector requires the following information to connect to AWS and collect data:

  • S3 bucket name
  • Name of the daily billing report and its prefix
  • Access key and secret key of the IAM account created

To fetch these details, complete the following preconfiguration tasks.

StepDetails

Create an S3 bucket to store the daily billing reports of your AWS resources that are generated by AWS.

 About S3 buckets
Amazon S3 is a repository to store data objects in the AWS cloud. Buckets are containers for data objects in Amazon S3. Therefore, you must first create a bucket and upload your data objects to the bucket. You can create multiple buckets to store related data objects.

The usage and billing details for the AWS GovCloud (US) and standard AWS accounts are combined. The S3 bucket that you create with a standard AWS account stores these combined reports. Therefore, always use the S3 bucket created with the standard AWS account for AWS GovCloud accounts also.

For more information, see AWS GovCloud (US-West) Billing and Payment .

 Steps to create a bucket

    1. Log in to the Amazon S3 console at https://console.aws.amazon.com/s3/.
    2. Click Create bucket.
    3. On the Name and region page, configure these properties:
      1. In the Bucket Name box, type a name for your bucket.
        Ensure that the name conforms to the bucket naming guidelines. For more information, see Rules for bucket naming .
      2. From the Region list, select a region for the bucket.
      3. (Optional) From the Copy settings from an existing bucket list, select the bucket. The settings of this bucket will be applied to the bucket that you are creating.
      4. If you have copied the settings from your existing bucket, click Create. Else, click Next.
    4. (Optional) On the Set properties page, enable the following properties. By default, these properties are disabled.
      1. Versioning for the objects in your bucket.
      2. Logging to track details of access requests to the data objects in the bucket.
      3. Tags to organize costs according to projects in the billing report. To add tags, click Add tag, and specify a key-value pair for the tag.
      4. Collection of the object-level API activity by using CloudTrail data events.
      5. Encryption of data objects that will be stored in the bucket.
      6. Click Next.
    5. On the Set permissions page, grant the following permissions:
      1. Bucket owner for managing objects in the bucket
      2. (Optional) Other AWS accounts for managing objects in the bucket
      3. (Not recommended) General public for accessing objects in the bucket
      4. (Optional) Amazon S3 Log Delivery group for accessing objects in the bucket
    6. On the Review page, verify the configuration settings, and click Create bucket. To change a setting, click Edit corresponding to the page where you want to make changes.

For more information about creating an S3 bucket, see  Creating a bucket .

Grant permissions to the S3 bucket to store the AWS Cost and Usage report from AWS.
 Steps to grant permissions

    1. Log in to the Amazon S3 console at https://console.aws.amazon.com/s3/.
    2. From the list of buckets, select the S3 bucket where you want to store the report.
    3. Click Permissions > Bucket Policy, and add the following code in the Bucket policy editor:

      {
        "Version": "2012-10-17",
        "Statement": [
        {
          "Effect": "Allow",
          "Principal": {
            "AWS": "386209384616"
          },
          "Action": [
            "s3:GetBucketAcl",
            "s3:GetBucketPolicy"
          ],
          "Resource": "arn:aws:s3:::bucketname"
        },
        {
          "Effect": "Allow",
          "Principal": {
            "AWS": "386209384616"
          },
          "Action": "s3:PutObject",
          "Resource": "arn:aws:s3:::bucketname/*"
        }
        ]
      }
    4. Replace bucketname with the name of your bucket. Do not change the Principal number 386209384616. AWS uses it to send reports to your bucket.
    5. Save the changes.

Schedule the AWS Cost and Usage report to be generated daily.

 About AWS Cost and Usage report

The AWS Cost and Usage report provides information about the usage of your AWS resources and the estimated cost for the usage. The report contains the details, such as AWS services that are used, the duration of usage, the amount of data transfer, and the used storage space.

If you use the consolidated billing feature, the report is available only to the master account that includes the cost and usage details of the member accounts associated with the master account.

 Steps to generate the AWS Cost and Usage report

    1. Log in to the Amazon S3 console: https://console.aws.amazon.com/s3
    2. Open the Billing and Cost Management console: https://console.aws.amazon.com/billing/
    3. Click Reports > Create report.
    4. On the Select Content page, configure the following properties:
      1. Report name: Type a name for the report.
      2. Time unit: Select Daily to aggregate report data every day.
      3. Include: Select the Resource IDs check box to associate the resources with the business services.
      4. Enable support for: Select whether you want to upload the report to Amazon Redshift or Amazon QuickSight.
      5. Click Next.
    5. On the Report details page, configure the following properties:
      1. In the S3 bucket box, type the name of the S3 bucket that is created with the standard AWS account and click Verify to check whether the bucket has appropriate permissions to store the reports. The reports are sent to this bucket.

      2. In the Report path prefix box, type the prefix that you want to append to the report name.
      3. In Report versioning, select Overwrite existing report.
        Overwriting reports can save on Amazon S3 storage costs and the processing time.
      4. Click Next.
    6. Review the settings, and click Review and Complete.

For more information, see Turn on daily reports .

Ensure that you have the required permissions to collect cost and utilization data for each AWS service
AWS Service namePermissions required for collecting cost dataPermissions required for collecting utilization data
S3
  • "S3:GetObject"
  • "S3:HeadBucket"
  • "S3: ListBucket"


Organizations"organizations:ListAccounts"
IAM
  • "iam:GetUser"
  • "iam:ListAccountAliases"
  • "iam:ListAttachedUserPolicies"
  • "GetPolicyVersion"
  • "GetPolicy"
  • "ListGroupsForUser"
  • "ListAttachedUserPolicies"
  • "GetUser"
EC2
  • "DescribeInstances"
  • "DescribeInstancesTypes"
RDS
  • "Describe"
CloudWatch
  • "GetMetricData"
  • "ListMetrics"
If you want to create a custom policy for collecting your cost data, you can use the sample json file
 Sample JSON snippet for collecting cost data
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "organizations:ListAccounts",
                "iam:ListAccountAliases",
                "iam:GetUser",
				"iam:listAttachedUserPolicies",
                "s3:HeadBucket"
            ],
            "Resource": "*"
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": "s3:GetObject",
            "Resource": [
                "arn:aws:s3:::dso-bill-bucket",
                "arn:aws:s3:::dso-bill-bucket/*"
            ]
        },
        {
            "Sid": "VisualEditor2",
            "Effect": "Allow",
            "Action": "s3:ListBucket",
            "Resource": [
                "arn:aws:s3:::dso-bill-bucket",
                "arn:aws:s3:::dso-bill-bucket/*"
            ]
        }
    ]
}

Note: Replace dso-bill-bucket with billing S3 bucket.

If you want to create a custom policy for collecting your utilization data, you can use the sample json file
 Sample JSON snippet for collecting utilization data
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "iam:GetPolicyVersion",
                "iam:GetPolicy",
                "iam:ListGroupsForUser",
                "iam:ListAttachedUserPolicies",
                "iam:GetUser"
            ],
            "Resource": [
                "arn:aws:iam::*:policy/*",
                "arn:aws:iam::*:user/*"
            ]
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeInstances",
				"ec2:DescribeInstanceTypes"
				"rds:Describe",
                "cloudwatch:GetMetricData",
                "cloudwatch:ListMetrics"
            ],
            "Resource": "*"
        }
    ]
}

You can use the CloudWatch agent to collect the system-level metrics from your AWS EC2 instances. These metrics are useful for investigating the utilization related issues that might occur in your AWS cloud environment. The CloudWatch agent collects these metrics and sends them to Amazon CloudWatch. When you run the AWS connector, these metrics are imported into BMC Helix Cloud Cost.

Collecting Metrics and Logs from Amazon EC2 Instances and On-Premises Servers with the CloudWatch Agent

Create an IAM user. You will need to specify the access key ID and the secret key of this user while configuring the connector.

 Steps to create an IAM user
    1. Open the IAM console and sign in with your AWS account credentials: https://console.aws.amazon.com/iam/
    2. From the left navigation pane, select Users > Add user.
    3. Enter a user name.
    4. Under Select AWS access type, select Programmatic access.
    5. Click Next Permissions.
    6. Select Attach existing policies directly.
    7. In the Filter box, search for the custom policies that you created for collecting cost and utilization data, and select it.
    8. Click Review.
    9. Click Create User.
      The policy is associated with the newly created IAM user.
    10. Note down the access key ID and the secret access key.
      Click Download.csv to download the access key ID and the secret key of the newly added user.

Configure an AWS IAM user account with specific privileges to access billing reports from the S3 bucket.

If you already have an IAM user account with the necessary permissions to access S3, you can use the access key ID and the secret key of this user during connector configuration. In such a case, you can skip this step.

 Steps to create an IAM user account with permissions to access S3

    1. Open the IAM console and sign in with your AWS account credentials: https://console.aws.amazon.com/iam/
    2. Click Users > Add user.
    3. On the Add user page, configure the following properties:
      1. In the User Name box, type a name for the IAM user.
      2. Under Select AWS access type, select the Programmatic access check box.
      3. Click Next: Permissions.
    4. Click Create group.
    5. On the Create group page, specify these details:
      1. In the Group name box, type a name for the group.
      2. From the list of policies, select the check box corresponding to the AmazonS3ReadOnlyAccess policy. Select AmazonEC2ReadOnly, AmazonRDSReadOnly, CloudWathcReadOnly if you want to collect utilization data as well.
      3. Click Create group.
        The group is created, and the specified user is added to this group.
    6. Click Next Review, review the configured settings, and then click Create user. The user is created with permissions to access the S3 bucket.
    7. Review the specified configuration settings, and click Create user. The user is created with permissions to access the S3 bucket.
    8. Note down the access key ID and the secret access key.

      Tip

      Click Download.csv to download the access key ID and the secret key of the newly created user.

Step II. Configure the connector

You must configure the connector to connect to Amazon Web Services for collecting the cost and usage data of AWS services.

To configure the connector:

  1. In the BMC Helix Cloud Cost dashboard, navigate to Connectors Add a Connector > and select AWS Cloud Connector from the cloud based connectors.
  2. On the Configure Connector page, configure the following properties:

    PropertyDescription
    Connector nameA unique name for the connector.
    AWS configurationSpecify whether you want to import data from the AWS GovCloud (US) account. The default selection is a standard AWS account.
    • AWS (default)
    • AWS GovCloud (US)
    Select the type of data that you want to collect

    Depending on your AWS subscription, select the type of data that you want to collect:

    • AWS (default)
      • Select Security and Compliance to collect all resource meta information and evaluate them for compliance & security. For more information, see Amazon Web Services cloud connector in the BMC Helix Cloud Security documentation. This option is available only if you are licensed to use BMC Helix Cloud Security.
      • Select Manage & Monitor AWS Costs to monitor and receive cost and utilization data of your AWS account.
      • Select Monitor utilization data of your AWS resources.
    • AWS GovCloud (US)
      • Select Security and Compliance to collect all resource meta information and evaluate them for compliance & security. For more information, see Amazon Web Services cloud connector in the BMC Helix Cloud Security documentation.
      • Select Monitor utilization data of your AWS resources. You must have at least one connector that is collecting cost data to view utilization data in your account.
    Cost Data S3 BucketEnter the name of the S3 bucket where you store the billing reports.
    Report NameSpecify the billing report name.
    Report PrefixSpecify the prefix that is attached to the report. (The prefix corresponds to the directory level in the S3 bucket hierarchy.)
    AWS account access key

    Specify the access key ID of the IAM user that you have created. For example, a typical access key ID looks like: AMAZONACSKEYID007EXAMPLE.

    To get the access key:

    • Open the IAM console and sign in with your AWS account credentials: https://console.aws.amazon.com/iam/
    • Click Users > select your user name.
    • Click Security > Credentials tab > Access key section.
    AWS account secret keySpecify the secret access key that is associated with the access key ID. For example, a typical secret access key looks like: wSecRetAcsKeYY712/K9POTUS/BCZthIZIzprvtEXAMPLEKEY.
  3. Collection mode: By default, the data collection cycle is set to On Demand collection. You can select an appropriate unit of time (days, minutes, hours) to schedule the data collection frequency along with event driven collection cycle where the collection is triggered when an event is identified in the selected account.
  4. On the Select Policies page, select the policies that you want to import from the policy library. This option is available only if you are licensed to use BMC Helix Cloud Security. For more information, see Managing policies .
  5. Click Continue. A confirmation message about the request for data collection processing is displayed.
    The Manage Connectors page shows the details of the newly configured AWS Cloud Connector.

Step III. Verify data collection

Verify that the connector ran successfully and check whether the AWS data is refreshed on the Dashboard.

To verify whether the connector ran successfully:

  1. On the Manage Connectors page, the state of the newly configured connector is updated to Running.
    When you run the connector for the first time, the connector recovers data for the past 6 months. The data collection begins immediately but depending on the number of resources in your environment the data is displayed after some time in BMC Helix Cloud Cost.
  2. On the BMC Helix Cloud Cost dashboard, the AWS connector tab is displayed.
  3. Select the AWS tab from the Dashboard.
  4. In the Summary tab, verify that the total cost, historical cost, and total resources are displayed. Also, Recommendations are displayed only when you have configured the connector to collect utilization data and you have efficiency issues in your infrastructure. Recommendations are not generated if all the resources are utilized efficiently.
  5. Resource pool information is not available by default. You must create a resource pool to view the resource pool details like name, resource count, budget, actual cost, and the projected cost. For information about how to create a resource pool, see, Resource Pools.
  6. In the Accounts tab, verify that the account details like name, actual cost, change in cost (in US dollars and percent), percent total cost, and number of resources are displayed for the accounts you own.
  7. In the Services tab, verify that the service details like name, actual cost, change in cost (in US dollars and percent), percent total cost, and number of resources are displayed.
  8. In the Explore Bill tab, verify that the resource name, actual cost, resource type, region, account name, and the service name are displayed.


Was this page helpful? Yes No Submitting... Thank you

Comments