Amazon Web Services connector

Use the Amazon Web Services cloud connector to collect the resource utilization data of the services that are provisioned in the Amazon Web Services (AWS) cloud. You can use this connector to:

Collect the cost data of all the services
Collect the usage data of your virtual machines (EC2 instances) and relational database instances

BMC Helix Cloud Cost uses these data points to provide cost insights and forecasting estimations to optimize your cloud costs by providing recommendations. Recommendations are displayed if you have configured the connector to collect both cost and utilization data.

The connector supports data collection for the following AWS subscription types:

AWS default
AWS GovCloud (US)

All communication between BMC Helix Cloud Cost and Amazon Web Services is secure over HTTPS. The connector uses the following APIs to collect data from AWS:

The following video (8:24) provides information about configuring the Amazon Web Services connector in BMC Helix Cloud Cost.

https://youtu.be/3WyIdmviMiI

License utilization

A product license gets consumed when the connector is used to collect data from the following asset types:

Amazon Elastic Compute Cloud (EC2)
Amazon Relational Database Service (Amazon RDS)
Amazon DynamoDB
Amazon Neptune
Amazon Redshift
Amazon Elastic Container Service (Amazon ECS)
Amazon Elastic Kubernetes Service (Amazon EKS)
Amazon API Gateway
Amazon ElastiCache
Amazon Simple Queue Service (Amazon SQS)
Amazon Elasticsearch Service

Collecting data by using the AWS cloud connector

To collect data by using the AWS cloud connector, do the following tasks:

I. Complete the preconfiguration tasks.

II. Configure the connector.

III. Verify data collection.

Step I. Complete the preconfiguration tasks

Depending on the type of data you want to collect, select a tab and complete the steps.

The connector requires the following information to connect to AWS and collect data:

S3 bucket name
Name of the daily billing report and its prefix
Access key and secret key of the IAM account

Note

If you want to import cost data from multiple AWS accounts, perform the following steps on your parent AWS account.

Step Details

Create an S3 bucket to store the daily billing reports of your AWS resources that are generated by AWS.

About S3 buckets

Amazon S3 is a repository to store data objects in the AWS cloud. Buckets are containers for data objects in Amazon S3. Therefore, you must first create a bucket and upload your data objects to the bucket. You can create multiple buckets to store related data objects.

The usage and billing details for the AWS GovCloud (US) and standard AWS accounts are combined. The S3 bucket that you create with a standard AWS account stores these combined reports. Therefore, always use the S3 bucket created with the standard AWS account for AWS GovCloud accounts also.

For more information, see AWS GovCloud (US-West) Billing and Payment Open link .

Steps to create a bucket

1. Log in to the Amazon S3 console at https://console.aws.amazon.com/s3/.
2. Click Create bucket.
3. On the Name and region page, configure these properties:
  1. In the Bucket Name box, type a name for your bucket.
    Ensure that the name conforms to the bucket naming guidelines. For more information, see Rules for bucket naming .
  2. From the Region list, select a region for the bucket.
  3. (Optional) From the Copy settings from an existing bucket list, select the bucket. The settings of this bucket will be applied to the bucket that you are creating.
  4. If you have copied the settings from your existing bucket, click Create. Else, click Next.
4. (Optional) On the Set properties page, enable the following properties. By default, these properties are disabled.
  1. Versioning for the objects in your bucket.
  2. Logging to track details of access requests to the data objects in the bucket.
  3. Tags are used to organize costs according to projects in the billing report. To add tags, click Add tag, and specify a key-value pair for the tag.
  4. Collection of the object-level API activity by using CloudTrail data events.
  5. Encryption of data objects that will be stored in the bucket.
  6. Click Next.
5. On the Set permissions page, grant the following permissions:
  1. Bucket owner for managing objects in the bucket
  2. (Optional) Other AWS accounts for managing objects in the bucket
  3. (Not recommended) General public for accessing objects in the bucket
  4. (Optional) Amazon S3 Log Delivery group for accessing objects in the bucket
6. On the Review page, verify the configuration settings, and click Create bucket. To change a setting, click Edit corresponding to the page where you want to make changes.

For more information about creating an S3 bucket, see Creating a bucket .

Grant permissions to the S3 bucket to store the AWS Cost and Usage report from AWS.

Steps to grant permissions

Log in to the Amazon S3 console at https://console.aws.amazon.com/s3/.
From the list of buckets, select the S3 bucket where you want to store the report.

Click Permissions > Bucket Policy, and add the following code in the Bucket policy editor:

{
  "Version": "2012-10-17",
  "Statement": [
  {
    "Effect": "Allow",
    "Principal": {
      "AWS": "386209384616"
    },
    "Action": [
      "s3:GetBucketAcl",
      "s3:GetBucketPolicy"
    ],
    "Resource": "arn:aws:s3:::bucketname"
  },
  {
    "Effect": "Allow",
    "Principal": {
      "AWS": "386209384616"
    },
    "Action": "s3:PutObject",
    "Resource": "arn:aws:s3:::bucketname/*"
  }
  ]
}

Replace bucketname with the name of your bucket. Do not change the Principal number 386209384616. AWS uses it to send reports to your bucket.
Save the changes.

Schedule the AWS Cost and Usage report to be generated daily.

About AWS Cost and Usage report

The AWS Cost and Usage report provides information about the usage of your AWS resources and the estimated cost for the usage. The report contains the details, such as AWS services that are used, the duration of usage, the amount of data transfer, and the used storage space.

If you use the consolidated billing feature, the report is available only to the master account that includes the cost and usage details of the member accounts associated with the master account.

Steps to generate the AWS Cost and Usage report

1. Log in to the Amazon S3 console: https://console.aws.amazon.com/s3
2. Open the Billing and Cost Management console: https://console.aws.amazon.com/billing/
3. Click Cost and Usage Reports > Create report.
4. On the Report Content page, configure the following properties:
  1. Report name: Type a name for the report.
  2. Include resource IDs: Select the check box to associate the resources with the business services.
  3. Data refresh settings: Select the check box to automatically refresh your Cost & Usage Report when charges are detected for previous months with closed bills.
  4. Click Next.
  5. On the Delivery options page, configure the following properties:
    1. Configure the S3 bucket. Type the name of the S3 bucket that is created with the standard AWS account and click Verify to check whether the bucket has appropriate permissions to store the reports. The reports are sent to this bucket.
    2. Report path prefix: Type the prefix that you want to append to the report name.
    3. Time granularity: Select Daily to aggregate report data every day.
    4. Report versioning: Select Overwrite existing report.
      Overwriting reports can save on Amazon S3 storage costs and the processing time.
    5. Enable report data integration for: (Optional) Select whether you want to upload the report to Amazon Redshift or Amazon QuickSight.
    6. Compression type: Select GZIP.
    7. Click Next.
5. On the Review page, review the settings, and click Review and Complete.

For more information, see Turn on daily reports Open link .

Ensure that you have the required permissions to collect cost data for each AWS service

AWS Service name	Permissions required for collecting cost data
S3	"S3:GetObject" "S3:HeadBucket" "S3: ListBucket"
Organizations	"organizations:ListAccounts"
IAM	"iam:GetUser" "iam:ListAccountAliases" "iam:ListAttachedUserPolicies"

If you want to create a custom policy for collecting your cost data, you can use the sample json file

Sample JSON snippet for collecting cost data

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "organizations:ListAccounts",
                "iam:ListAccountAliases",
                "iam:GetUser",
				"iam:listAttachedUserPolicies",
                "s3:HeadBucket"
            ],
            "Resource": "*"
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": "s3:GetObject",
            "Resource": [
                "arn:aws:s3:::dso-bill-bucket",
                "arn:aws:s3:::dso-bill-bucket/*"
            ]
        },
        {
            "Sid": "VisualEditor2",
            "Effect": "Allow",
            "Action": "s3:ListBucket",
            "Resource": [
                "arn:aws:s3:::dso-bill-bucket",
                "arn:aws:s3:::dso-bill-bucket/*"
            ]
        }
    ]
}

Note: Replace dso-bill-bucket with billing S3 bucket.

Create an IAM user. You will need to specify the access key ID and the secret key of this user while configuring the connector.

Steps to create an IAM user

1. Open the IAM console and sign in with your AWS account credentials: https://console.aws.amazon.com/iam/
2. From the left navigation pane, select Access Management > Users > Add user.
3. Enter a user name.
4. Under Select AWS access type, select Programmatic access.
5. Click Next Permissions.
6. Select Attach existing policies directly.
7. In the Filter box, search for the custom policies that you created for collecting cost data and select it.
8. Click Review.
9. Click Create User.
  The policy is associated with the newly created IAM user.
10. Note down the access key ID and the secret access key.
  Click Download.csv to download the access key ID and the secret key of the newly added user.

Configure an AWS IAM user account with specific privileges to access billing reports from the S3 bucket.

If you already have an IAM user account with the necessary permissions to access S3, you can use the access key ID and the secret key of this user during connector configuration. In such a case, you can skip this step.

Steps to create an IAM user account with permissions to access S3

1. Open the IAM console and sign in with your AWS account credentials: https://console.aws.amazon.com/iam/
2. From the left navigation pane, select Access Management > Users > Add user.
3. On the Add user page, configure the following properties:
  1. In the User Name box, type a name for the IAM user.
  2. Under Select AWS access type, select the Programmatic access check box.
  3. Click Next: Permissions.
4. Click Create group.
5. On the Create group page, specify these details:
  1. In the Group name box, type a name for the group.
  2. From the list of policies, select the check box corresponding to the AmazonS3ReadOnlyAccess policy. Select AmazonEC2ReadOnly, AmazonRDSReadOnly, CloudWatchReadOnly if you want to collect utilization data as well.
  3. Click Create group.
    The group is created, and the specified user is added to this group.
6. Click Next Review, review the configured settings, and then click Create user. The user is created with permissions to access the S3 bucket.
7. Review the specified configuration settings, and click Create user. The user is created with permissions to access the S3 bucket.
8. Note down the access key ID and the secret access key.
  
  Tip
  
  Click Download.csv to download the access key ID and the secret key of the newly created user.

Depending on your AWS account setup, select a tab and complete the steps:

Step Details

Create an IAM user. You will need to specify the access key ID and the secret key of this user while configuring the connector.

Steps to create an IAM user

1. Open the IAM console and sign in with your AWS account credentials: https://console.aws.amazon.com/iam/
2. From the left navigation pane, select Access Management > Users > Add user.
3. Enter a user name.
4. Under Select AWS access type, select Programmatic access.
5. Click Next Permissions.
6. Select Attach existing policies directly.
7. In the Filter box, search for the custom policies that you created for collecting utilization data, and select it.
8. Click Review.
9. Click Create User.
  The policy is associated with the newly created IAM user.
10. Note down the access key ID and the secret access key.
  Click Download.csv to download the access key ID and the secret key of the newly added user.

Ensure that you have the required permissions to collect utilization data for each AWS service

AWS Service name	Permissions required for collecting utilization data
IAM	"GetPolicyVersion" "GetPolicy" "ListGroupsForUser" "ListAttachedUserPolicies" "GetUser"
EC2	"DescribeInstances" "DescribeInstance Types" "DescribeVolumes"
RDS	"DescribeDBInstances"
CloudWatch	"GetMetricData" "ListMetrics"

If you want to create a custom policy for collecting your utilization data, you can use the sample json file

Sample JSON snippet for collecting utilization data

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "iam:GetPolicyVersion",
                "iam:GetPolicy",
                "iam:ListGroupsForUser",
                "iam:ListAttachedUserPolicies",
                "iam:GetUser"
            ],
            "Resource": [
                "arn:aws:iam::*:policy/*",
                "arn:aws:iam::*:user/*"
            ]
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeInstances",
				"ec2:DescribeInstanceTypes",
				"ec2:DescribeVolumes",
				"rds:DescribeDBInstances",
                "cloudwatch:GetMetricData",
                "cloudwatch:ListMetrics"
            ],
            "Resource": "*"
        }
    ]
}

You can use the CloudWatch agent to collect the system-level metrics from your AWS EC2 instances. These metrics are useful for investigating the utilization related issues that might occur in your AWS cloud environment. The CloudWatch agent collects these metrics and sends them to Amazon CloudWatch. When you run the AWS connector, these metrics are imported into BMC Helix Cloud Cost.

Collecting Metrics and Logs from Amazon EC2 Instances and On-Premises Servers with the CloudWatch Agent Open link

(Optional) If you want to configure a single AWS connector for collecting cost and utilization data, ensure that you have the required permissions

AWS Service name	Permissions required for collecting cost data	Permissions required for collecting utilization data
S3	"S3:GetObject" "S3:HeadBucket" "S3: ListBucket"
Organizations	"organizations:ListAccounts"
IAM	"iam:GetUser" "iam:ListAccountAliases" "iam:ListAttachedUserPolicies"	"GetPolicyVersion" "GetPolicy" "ListGroupsForUser" "ListAttachedUserPolicies" "GetUser"
EC2		"DescribeInstances" "DescribeInstancesTypes" "DescribeVolumes"
RDS		"DescribeDBInstances"
CloudWatch		"GetMetricData" "ListMetrics"

In multiple AWS account setup, the owner of the parent AWS account must perform the preconfiguration tasks.

Step Details

Create an IAM user. You will need to specify the access key ID and the secret key of this user while configuring the connector.

Steps to create an IAM user

1. Open the IAM console and sign in with your AWS account credentials: https://console.aws.amazon.com/iam/
2. From the left navigation pane, select Access Management > Users > Add user.
3. Enter a user name.
4. Under Select AWS access type, select Programmatic access.
5. Click Next Permissions.
6. Select Attach existing policies directly.
7. In the Filter box, search for the custom policies that you created for collecting utilization data, and select it.
8. Click Review.
9. Click Create User.
  The policy is associated with the newly created IAM user.
10. Note down the access key ID and the secret access key.
  Click Download.csv to download the access key ID and the secret key of the newly added user.

Ensure that you have the required permissions to collect utilization data for each AWS service

AWS Service name	Permissions required for collecting utilization data
IAM	"GetPolicy" "GetPolicyVersion" "GetUser" "ListAccessKeys" "ListAttachedRolePolicies" "ListAttachedUserPolicies" "ListAttachedGroupPolicies" "ListEntitiesForPolicy" "ListPolicies" "ListRoles" "ListUserPolicies" "ListGroupPolicies" "ListGroupsForUser"
EC2	"DescribeInstances" "DescribeVolumes" "DescribeInstancesTypes"
RDS	"DescribeDBInstances"
CloudWatch	"GetMetricData" "ListMetrics"

If you want to create a custom policy for collecting your utilization data, you can use the sample json file

Sample JSON snippet for collecting utilization data

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "iam:getPolicy",
				"iam:getPolicyVersion",
		        "iam:getUser",
		        "iam:listAccessKeys",
		        "iam:listAttachedRolePolicies",
		        "iam:listAttachedUserPolicies",
		        "iam:listAttachedGroupPolicies",
		        "iam:listEntitiesForPolicy",
		        "iam:listPolicies",
		        "iam:listRoles",
		        "iam:listUserPolicies",
		        "iam:ListGroupPolicies",
		        "iam:ListGroupsForUser"
            ],
            "Resource": [
                "arn:aws:iam::*:policy/*",
                "arn:aws:iam::*:user/*"
            ]
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeInstances",
				"ec2:DescribeInstanceTypes",
				"ec2:DescribeVolumes",
				"rds:DescribeDBInstances",
                "cloudwatch:GetMetricData",
                "cloudwatch:ListMetrics"
            ],
            "Resource": "*"
        }
    ]
}

Obtain the parent (trusted) AWS account ID and note it down.

Steps to obtain the account ID of the parent (trusted) account

In the AWS Management Console header, click the account name and select My Account.

The Account Settings information displays the Account ID.

Create a role in the child (trusting) accounts and provide the required permissions.

To use the AWS connector on multiple AWS accounts you must delegate access to all AWS accounts using IAM roles. This includes using role and establishing trust-based access. Trust relationship is created between parent account and a group of tenant accounts.

Steps to create a role in the child (trusting) AWS accounts

Open the IAM console and sign in with your AWS account credentials: https://console.aws.amazon.com/iam/
In the IAM service, select the Roles option and click Create role.
Select Another AWS account role type.
In Account ID, type the AWS account ID of your parent AWS account.

In the Attach Policy step, depending on your requirements, select the custom policies that you have created for collecting utilization data. Each child account must have the following permissions:

AWS service name	Permissions required
EC2	"DescribeInstances" "DescribeInstancesTypes" "DescribeVolumes"
RDS	"DescribeDBInstances"
CloudWatch	"GetMetricData" "ListMetrics"

Specify the role name and click Create role.
The role is created.
Select Management > Roles. Select the newly created role.
On the Trust relationships tab, the parent account is displayed as the trusted entity.

These steps must be performed on each child AWS account that you want to add as trusting account.

Steps to obtain the Role ARN

Open the IAM console and sign in with your AWS account credentials: https://console.aws.amazon.com/iam/
In the IAM service, select the Roles option and search for the role that you created in the earlier step.
Click the role name and note down the Role ARN. You need to use the ARNs that you have created in the parent account assume role policy.

Grant access to the role to the parent (trusted) account

The parent (trusted) account groups have permissions to access resources in the child (trusting) accounts. To add the required permissions to the user you want to be able to access resources, modify the policy of the user/group in the parent (trusted) account that is going to access the resources.

When you create or modify the policy, copy the the ARNs of the connector Role1 and connector Role2 that are obtained from the earlier step. When you create a connector, you need the credentials of the user who has this policy attached.

Steps to grant role access to the parent (trusted) account

Providing access to individual child (trusting) accounts
Create or update the policy in the parent (trusted) account and attach the policy to the user that will be used for configuring the connector. You must add the necessary permissions to access the resources of all the trusting accounts.

Example: Assume role policy with individual account details

{

"Version": "2020-10-19",
"Statement": {
"Effect": "Allow",
"Action": "sts:AssumeRole",
"Resource": [ "arn:aws:iam::12345654321:role/trustedrolename",
"arn:aws:iam::98798743211:role/trustedrolename"]
}
}
Providing access to all child accounts simultaneously
Create or update the policy in the master AWS account and attach the policy to the user that will be used for configuring the connector. Use the * character instead of the account ID in the Resource ARN and create the same role in each child (trusting) accounts.

Example: Assume role policy using *

{

"Version": "2020-10-19",
"Statement": {
"Effect": "Allow",
"Action": "sts:AssumeRole",
"Resource": ["arn:aws:iam::*:role/trustedrolename"]

}
}

You can use the CloudWatch agent to collect the system-level metrics from your AWS EC2 instances. These metrics are useful for investigating the utilization related issues that might occur in your AWS cloud environment. The CloudWatch agent collects these metrics and sends them to Amazon CloudWatch. When you run the AWS connector, these metrics are imported into BMC Helix Cloud Cost.

Collecting Metrics and Logs from Amazon EC2 Instances and On-Premises Servers with the CloudWatch Agent Open link

(optional) If you want to use the role-based authentication, do the following:

Create an IAM role with the BMC AWS account as a trusted entity.
Do the following:
1. Log in to your AWS account.
2. Under Security, click IAM. You are directed to the Identity and Access Management (IAM) dashboard.
3. Click Roles > Create role.
4. Select Another AWS account, and do the following:
  1. In the Account ID box, type the identifier of the BMC AWS account.
  2. Select the Require external ID check box, and enter any alphanumeric string as a value for the external ID. This ID is used to grant access to collect the AWS resources data.
  3. Click Next.
5. On the permissions page, select the appropriate option, and click Next.
6. (optional) Create tags and click Next.
7. Click Create role. A confirmation message is displayed.
Note down the role ARN and external ID.
Do the following:
1. Search for the role that you created and select it to view the details.
2. From the Summary section, note down the role ARN value.
3. Click the Trust relationships tab, and note down the external ID value.

Step II. Configure the connector

You must configure the connector to connect to Amazon Web Services for collecting the cost and usage data of AWS services.

To configure the connector:

In the BMC Helix Cloud Cost dashboard, navigate to Connectors > Add a Connector > and select AWS Cloud Connector from the cloud based connectors.

On the Configure Connector page, configure the following properties:

Property	Description
Connector name	A unique name for the connector.
AWS configuration	Specify whether you want to import data from the AWS GovCloud (US) account. The default selection is a standard AWS account. AWS (default) AWS GovCloud (US)
Select the type of data that you want to collect	Depending on your AWS subscription, select the type of data that you want to collect: AWS (default) Select Security and Compliance to collect all resource meta information and evaluate them for compliance & security. For more information, see Amazon Web Services cloud connector in the BMC Helix Cloud Security documentation. This option is available only if you are licensed to use BMC Helix Cloud Security. Select Manage & Monitor AWS Costs to monitor and receive cost and utilization data of your AWS account. Select Monitor utilization of AWS Resources to monitor and receive utilization data of your AWS resources. AWS GovCloud (US) Select Security and Compliance to collect all resource meta information and evaluate them for compliance & security. For more information, see Amazon Web Services cloud connector in the BMC Helix Cloud Security documentation. Select Monitor utilization of AWS Resources to monitor and receive utilization data of your AWS resources. You must have at least one connector that is collecting cost data to view utilization data in your account. Note: Recommendations are displayed if you have configured the connector to collect both cost and utilization data.
Cost Data S3 Bucket	Enter the name of the S3 bucket where you store the billing reports.
Report Name	Specify the billing report name.
Report Prefix	Specify the prefix that is attached to the report. (The prefix corresponds to the directory level in the S3 bucket hierarchy.)
Credential Type	Configure the authentication method to authenticate with your AWS account: Key Based: This authentication uses your AWS keys. AWS Account Access Key: Specify the access key ID of the IAM user that you have created. For example, a typical access key ID looks like: AMAZONACSKEYID007EXAMPLE. To get the access key: Open the IAM console and sign in with your AWS account credentials: https://console.aws.amazon.com/iam/ Click Users > select your user name. Click Security > Credentials tab > Access key section. AWS Account Secret Key: Specify the secret access key that is associated with the access key ID. For example, a typical secret access key looks like: wSecRetAcsKeYY712/K9POTUS/BCZthIZIzprvtEXAMPLEKEY

Collection Mode: By default, the data collection cycle is set to On Demand collection. You can select an appropriate unit of time (days, minutes, hours) to schedule the data collection frequency along with event driven collection cycle where the collection is triggered when an event is identified in the selected account.
On the Select Policies page, select the policies that you want to import from the policy library. This option is available only if you are licensed to use BMC Helix Cloud Security. For more information, see Managing policies .
Click Continue. A confirmation message about the request for data collection processing is displayed.
The Manage Connectors page shows the details of the newly configured AWS Cloud Connector.

Step III. Verify data collection

Verify that the connector ran successfully and check whether the AWS data is refreshed on the Dashboard.

To verify whether the connector ran successfully:

On the Manage Connectors page, the state of the newly configured connector is updated to Running.
When you run the connector for the first time, the connector recovers data for the past 6 months. The data collection begins immediately but depending on the number of resources in your environment the data is displayed after some time in BMC Helix Cloud Cost.
On the BMC Helix Cloud Cost dashboard, the AWS connector tab is displayed.
Select the AWS tab from the Dashboard.
In the Summary tab, verify that the total cost, historical cost, and total resources are displayed. Also, recommendations are displayed if you have configured the connector to collect both cost and utilization data; and you have efficiency issues in your infrastructure. Recommendations are not generated if all the resources are utilized efficiently.
Resource pool information is not available by default. You must create a resource pool to view the resource pool details like name, resource count, budget, actual cost, and the projected cost. For details, see Resource Pools.
In the Accounts tab, verify that the account details like name, actual cost, change in cost (in US dollars and percent), percent total cost, and number of resources are displayed for the accounts you own.
In the Services tab, verify that the service details like name, actual cost, change in cost (in US dollars and percent), percent total cost, and number of resources are displayed.
In the Explore Bill tab, verify that the resource name, actual cost, resource type, region, account name, and the service name are displayed.

Amazon Web Services connector

License utilization

Collecting data by using the AWS cloud connector

Comments