Amazon Web Services connector
Use the Amazon Web Services cloud connector to collect the resource utilization data of the services that are provisioned in the Amazon Web Services (AWS) cloud. You can use this connector to:
- Collect the cost data of all the services
- Collect the usage data of your virtual machines (EC2 instances) and relational database instances
BMC Helix Cloud Cost uses these data points to provide cost insights and forecasting estimations to optimize your cloud costs by providing recommendations. Recommendations are displayed if you have configured the connector to collect both cost and utilization data.
The connector supports data collection for the following AWS subscription types:
- AWS default
- AWS GovCloud (US)
All communication between BMC Helix Cloud Cost and Amazon Web Services is secure over HTTPS. The connector uses the following APIs to collect data from AWS:
- Describe EC2 instances
- Describe Volumes
- Describe DB instances
- Collect EC2 metrics
- Collect EC2 metrics using the CloudWatch Agent
- Cost and usage reports in AWS S3
The following video (8:24) provides information about configuring the Amazon Web Services connector in BMC Helix Cloud Cost.
License utilization
A product license gets consumed when the connector is used to collect data from the following asset types:
- Amazon Elastic Compute Cloud (EC2)
- Amazon Relational Database Service (Amazon RDS)
- Amazon DynamoDB
- Amazon Neptune
- Amazon Redshift
- Amazon Elastic Container Service (Amazon ECS)
- Amazon Elastic Kubernetes Service (Amazon EKS)
- Amazon API Gateway
- Amazon ElastiCache
- Amazon Simple Queue Service (Amazon SQS)
- Amazon Elasticsearch Service
Collecting data by using the AWS cloud connector
To collect data by using the AWS cloud connector, do the following tasks:
I. Complete the preconfiguration tasks.
III. Verify data collection.
Depending on the type of data you want to collect, select a tab and complete the steps.
The connector requires the following information to connect to AWS and collect data:
- S3 bucket name
- Name of the daily billing report and its prefix
- Access key and secret key of the IAM account
Note
If you want to import cost data from multiple AWS accounts, perform the following steps on your parent AWS account.
Step | Details | ||||||||
---|---|---|---|---|---|---|---|---|---|
Create an S3 bucket to store the daily billing reports of your AWS resources that are generated by AWS. The usage and billing details for the AWS GovCloud (US) and standard AWS accounts are combined. The S3 bucket that you create with a standard AWS account stores these combined reports. Therefore, always use the S3 bucket created with the standard AWS account for AWS GovCloud accounts also. For more information, see AWS GovCloud (US-West) Billing and Payment . |
|
||||||||
Grant permissions to the S3 bucket to store the AWS Cost and Usage report from AWS. |
|
||||||||
Schedule the AWS Cost and Usage report to be generated daily. |
|
||||||||
Ensure that you have the required permissions to collect cost data for each AWS service |
|
||||||||
If you want to create a custom policy for collecting your cost data, you can use the sample json file |
Note: Replace |
||||||||
Create an IAM user. You will need to specify the access key ID and the secret key of this user while configuring the connector. |
|
||||||||
Configure an AWS IAM user account with specific privileges to access billing reports from the S3 bucket. If you already have an IAM user account with the necessary permissions to access S3, you can use the access key ID and the secret key of this user during connector configuration. In such a case, you can skip this step. |
|
Depending on your AWS account setup, select a tab and complete the steps:
Step | Details | |||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Create an IAM user. You will need to specify the access key ID and the secret key of this user while configuring the connector. |
|
|||||||||||||||||||||
Ensure that you have the required permissions to collect utilization data for each AWS service |
|
|||||||||||||||||||||
If you want to create a custom policy for collecting your utilization data, you can use the sample json file |
|
|||||||||||||||||||||
You can use the CloudWatch agent to collect the system-level metrics from your AWS EC2 instances. These metrics are useful for investigating the utilization related issues that might occur in your AWS cloud environment. The CloudWatch agent collects these metrics and sends them to Amazon CloudWatch. When you run the AWS connector, these metrics are imported into BMC Helix Cloud Cost. |
||||||||||||||||||||||
(Optional) If you want to configure a single AWS connector for collecting cost and utilization data, ensure that you have the required permissions |
|
In multiple AWS account setup, the owner of the parent AWS account must perform the preconfiguration tasks.
Step | Details | ||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|
Create an IAM user. You will need to specify the access key ID and the secret key of this user while configuring the connector. |
|
||||||||||
Ensure that you have the required permissions to collect utilization data for each AWS service |
|
||||||||||
If you want to create a custom policy for collecting your utilization data, you can use the sample json file |
|
||||||||||
Obtain the parent (trusted) AWS account ID and note it down. |
|
||||||||||
Create a role in the child (trusting) accounts and provide the required permissions. To use the AWS connector on multiple AWS accounts you must delegate access to all AWS accounts using IAM roles. This includes using role and establishing trust-based access. Trust relationship is created between parent account and a group of tenant accounts. |
|
||||||||||
Grant access to the role to the parent (trusted) account The parent (trusted) account groups have permissions to access resources in the child (trusting) accounts. To add the required permissions to the user you want to be able to access resources, modify the policy of the user/group in the parent (trusted) account that is going to access the resources. When you create or modify the policy, copy the the ARNs of the connector Role1 and connector Role2 that are obtained from the earlier step. When you create a connector, you need the credentials of the user who has this policy attached. |
|
||||||||||
You can use the CloudWatch agent to collect the system-level metrics from your AWS EC2 instances. These metrics are useful for investigating the utilization related issues that might occur in your AWS cloud environment. The CloudWatch agent collects these metrics and sends them to Amazon CloudWatch. When you run the AWS connector, these metrics are imported into BMC Helix Cloud Cost. |
(optional) If you want to use the role-based authentication, do the following:
Create an IAM role with the BMC AWS account as a trusted entity.
Note down the role ARN and external ID.
You must configure the connector to connect to Amazon Web Services for collecting the cost and usage data of AWS services.
To configure the connector:
- In the BMC Helix Cloud Cost dashboard, navigate to Connectors > Add a Connector > and select AWS Cloud Connector from the cloud based connectors.
On the Configure Connector page, configure the following properties:
Property Description Connector name A unique name for the connector. AWS configuration Specify whether you want to import data from the AWS GovCloud (US) account. The default selection is a standard AWS account. - AWS (default)
- AWS GovCloud (US)
Select the type of data that you want to collect Depending on your AWS subscription, select the type of data that you want to collect:
- AWS (default)
- Select Security and Compliance to collect all resource meta information and evaluate them for compliance & security. For more information, see Amazon Web Services cloud connector in the BMC Helix Cloud Security documentation. This option is available only if you are licensed to use BMC Helix Cloud Security.
- Select Manage & Monitor AWS Costs to monitor and receive cost and utilization data of your AWS account.
- Select Monitor utilization data of your AWS resources.
- AWS GovCloud (US)
- Select Security and Compliance to collect all resource meta information and evaluate them for compliance & security. For more information, see Amazon Web Services cloud connector in the BMC Helix Cloud Security documentation.
- Select Monitor utilization data of your AWS resources. You must have at least one connector that is collecting cost data to view utilization data in your account.
Note: Recommendations are displayed if you have configured the connector to collect both cost and utilization data.
Cost Data S3 Bucket Enter the name of the S3 bucket where you store the billing reports. Report Name Specify the billing report name. Report Prefix Specify the prefix that is attached to the report. (The prefix corresponds to the directory level in the S3 bucket hierarchy.) Credential Type Configure the authentication method to authenticate with your AWS account:
- Key Based: This authentication uses your AWS keys.
- AWS Account Access Key: Specify the access key ID of the IAM user that you have created. For example, a typical access key ID looks like: AMAZONACSKEYID007EXAMPLE.
To get the access key:
- Open the IAM console and sign in with your AWS account credentials: https://console.aws.amazon.com/iam/
- Click Users > select your user name.
- Click Security > Credentials tab > Access key section.
- AWS Account Secret Key: Specify the secret access key that is associated with the access key ID. For example, a typical secret access key looks like: wSecRetAcsKeYY712/K9POTUS/BCZthIZIzprvtEXAMPLEKEY
- AWS Account Access Key: Specify the access key ID of the IAM user that you have created. For example, a typical access key ID looks like: AMAZONACSKEYID007EXAMPLE.
- AWS (default)
- Collection Mode: By default, the data collection cycle is set to On Demand collection. You can select an appropriate unit of time (days, minutes, hours) to schedule the data collection frequency along with event driven collection cycle where the collection is triggered when an event is identified in the selected account.
- On the Select Policies page, select the policies that you want to import from the policy library. This option is available only if you are licensed to use BMC Helix Cloud Security. For more information, see
Managing policies
.
- Click Continue. A confirmation message about the request for data collection processing is displayed.
The Manage Connectors page shows the details of the newly configured AWS Cloud Connector.
Verify that the connector ran successfully and check whether the AWS data is refreshed on the Dashboard.
To verify whether the connector ran successfully:
- On the Manage Connectors page, the state of the newly configured connector is updated to Running.
When you run the connector for the first time, the connector recovers data for the past 6 months. The data collection begins immediately but depending on the number of resources in your environment the data is displayed after some time in BMC Helix Cloud Cost. - On the BMC Helix Cloud Cost dashboard, the AWS connector tab is displayed.
- Select the AWS tab from the Dashboard.
- In the Summary tab, verify that the total cost, historical cost, and total resources are displayed. Also, recommendations are displayed if you have configured the connector to collect both cost and utilization data; and you have efficiency issues in your infrastructure. Recommendations are not generated if all the resources are utilized efficiently.
- Resource pool information is not available by default. You must create a resource pool to view the resource pool details like name, resource count, budget, actual cost, and the projected cost. For details, see Resource Pools.
- In the Accounts tab, verify that the account details like name, actual cost, change in cost (in US dollars and percent), percent total cost, and number of resources are displayed for the accounts you own.
- In the Services tab, verify that the service details like name, actual cost, change in cost (in US dollars and percent), percent total cost, and number of resources are displayed.
- In the Explore Bill tab, verify that the resource name, actual cost, resource type, region, account name, and the service name are displayed.
Comments
Hello, thanks for creating the documentation. I noticed that AWS have changed their interfaces which makes the AWS side instructions here a bit obsolete. Would you mind updating them? Especially the ones for user creation and scheduling the report. Thanks!
Thank you for bringing this to our attention, Przemek Tomczuk.
I have updated the instructions for Steps to create an IAM user and Steps to generate the AWS Cost and Usage report sections.
Thanks,
Shweta
Log in or register to comment.