Amazon Web Services - AWS API Extractor

Use the Amazon Web Services - AWS API Extractor to integrate with Amazon Web Service (AWS) to discover and import the AWS infrastructure data for capacity planning. The extractor makes API calls to the following AWS services:

  • EC2: To import EC2 instances and EBS volumes.
  • CloudWatch: To import metrics for EC2 instances, EBS volumes, and Auto Scaling groups.
  • Auto Scaling: To import the Auto Scaling groups.

Depending on your requirement, you can configure one ETL module for a single or multiple AWS accounts. For multiple accounts, one AWS account is used as the main account to retrieve data from all the AWS services and to connect to other accounts.

The extractor collects tagging information from your AWS resources. If you use tags to organize your resources by related business services, you can configure the ETL to display business services and their related resources in a hierarchy in the Workspace. This hierarchy enables you to sort and view capacity management metrics by business service.

This integration uses the AWS Java SDK version 1.11.60. 
For entities, lookup information, and metrics for the AWS - API Extractor, see Entities, lookup information, and metrics for AWS API Extractor.

This topic explains how to configure and run the ETL.



AWS resources

Release Notes: AWS SDK for Java 1.11.60

About IAM policies

Prerequisites

If you want to configure an ETL to retrieve data from multiple AWS accounts, click the Multiple AWS accounts tab. Else, complete the prerequisite steps under the Single AWS account tab.

Single account

Prerequisite step Reference instructions/topics

Configure a policy to specify the permissions for the IAM user.

  Steps to configure the Access Privilege policy.
  1. Open the IAM console and sign in with your AWS account credentials: https://console.aws.amazon.com/iam/
  2. From the left navigation pane, select Policies > Create policy > Create your own policy.
  3. Specify a name for the policy. For example: tsco-aws-etl-policy
  4. In Policy Document section, enter the following JSON example:

    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Sid": "Stmt1484736991000",
                "Effect": "Allow",
                "Action": [
                    "cloudwatch:GetMetricData",
                    "cloudwatch:GetMetricStatistics",
                    "cloudwatch:ListMetrics",
                    "ec2:DescribeVolumes",
                    "ec2:DescribeHosts",
                    "ec2:DescribeRegions",
                    "ec2:DescribeAvailabilityZones",
                    "ec2:DescribeInstances",
                    "ec2:DescribeAccountAttributes",
                    "ec2:DescribeSnapshots",
                    "autoscaling:DescribeAutoScalingInstances",
                    "autoscaling:DescribeAutoScalingGroups",
                    "autoscaling:DescribePolicies",
                    "autoscaling:DescribeLaunchConfigurations",
                ],
                "Resource": [
                    "*"
                ]
            }
        ]
    }
  5. Click Validate Policy to ensure that the policy is syntactically correct.
  6. Click Create Policy.

For more information, see Creating a new policy.

Create an IAM user. You need the access key details of this user during ETL configuration. The access key details include an access key ID and a secret key. The AWS SDK requires these keys to automatically sign the requests that the ETL sends to AWS.

  Steps to create an IAM user.
  1. Open the IAM console and sign in with your AWS account credentials: https://console.aws.amazon.com/iam/
  2. From the left navigation pane, select Users > Add user.
  3. In the User name field, type a user name that is used as a sign-in name for AWS.
  4. (Optional) Click Add another user to create an additional user. You can create up to 10 users.
  5. Under Select AWS access type, select Programmatic access to access API, AWS, CLI, or Tools for Windows Powershell.

3. Click Next Permissions.

4. Select Attach existing policies directly

5. In the Filter box, search for the policy that you created, and select it.

6. Click Review.

7. Click Create User.

The policy is associated with the newly created IAM user.

5. Note down the Access key ID and Secret access key details.

Tip

Click Download .csv to get the access key ID and secret key of the newly added user in a file of .CSV format.

Tag your resources by using a business service tag key name such as Service to organize the resources by business services. You need to specify this business service tag key name during ETL configuration.

For information about tagging your resources, see Tagging your Amazon EC2 resources.

Multiple accounts

Click a box in the workflow diagram to view the corresponding step details.

aws_multiacct


Basic requirements

  • Generate an external ID that you need to use when you configure the additional AWS accounts. The external ID is an alphanumeric string. Use any alphanumeric string or use a tool such as the GUID UNIX tool to generate it.

  • To organize your resources by business services, ensure that you tag your resources by using a business service tag key name such as Service. You need to specify this business service tag key name during ETL configuration.


    For more information, see Tagging your Amazon EC2 resources.

Configure the main AWS account

Prerequisite step Reference instructions/topics

Access the main AWS account.

Log on to the  AWS Management Console.

Obtain the AWS account ID and note it down.

  Steps to obtain AWS account ID

In the AWS Management Console header, click the account name and select My Account.

The Account Settings information is displayed that shows the Account ID.

Configure a policy (tsco-aws-etl-policy) to specify the permissions for the user of the main AWS account.

  Steps to configure the Access Privilege policy.
  1. Select Policies > Create policy > Create your own policy.
  2. Specify a name for the policy. For example: tsco-aws-etl-policy
  3. In Policy Document section, enter the following JSON example:

    {
    	"Version": "2012-10-17",
    	"Statement": [
    		{
    			"Action": [
    				"cloudwatch:GetMetricData",
    				"cloudwatch:GetMetricStatistics",
    				"cloudwatch:ListMetrics",
    				"ec2:DescribeVolumes",
    				"ec2:DescribeHosts",
    				"ec2:DescribeRegions",
    				"ec2:DescribeAvailabilityZones",
    				"ec2:DescribeInstances",
    				"ec2:DescribeAccountAttributes",
    				"ec2:DescribeSnapshots",
    				"autoscaling:DescribeAutoScalingInstances",
    				"autoscaling:DescribeAutoScalingGroups",
    				"autoscaling:DescribePolicies",
    				"autoscaling:DescribeLaunchConfigurations",
    			],
    			"Resource": [
    				"*"
    			],
    			"Effect": "Allow",
    			"Sid": "Stmt1484736991000"
    		}
    	]
    }
  4. Click Validate Policy to ensure that the policy is syntactically correct.
  5. Click Create Policy.

For more information, see Creating a new policy.

Create an IAM user (tsco-etl-user) in the main account and assign the policy (tsco-aws-etl-policy) to the user.

You need the access key details of this account during ETL configuration.

The access keys include a key ID and a secret key. The AWS SDK requires these keys to automatically sign the requests that the ETL sends to AWS. For more information about managing access keys for IAM users, see  Managing Access Keys for IAM Users .  

  Steps to configure an IAM user.
  1. In the Set user details section on the Add user page, click Add another user, and then specify a user name for the new IAM account. For example, tsco-etl-user.
  2. Under Select AWS access type, select Programmatic access.

3. Click Next Permissions.

4. Select Attach existing policies directly

5. In the Filter box, search for the policy that you created in the earlier step (tsco-aws-etl-policy) and select it.

6. Click Review.

7. Click Create User.

The policy (tsco-aws-etl-policy) is associated with the newly created IAM user (tsco-etl-user).

5. Note down the Access key ID and Secret access key details.

Tip

Click Download .csv to get the access key ID and secret key of the newly added user in a file of .CSV format.

Configure the additional AWS account

You must repeat these steps for every additional AWS account.

Prerequisite step Reference instructions/topics

Access the additional AWS account.

Log on to the AWS Management Console.  

Obtain the account ID and note it down.

You need to enter this account ID when configuring a policy in the main AWS account to include the additional account details. You also need to enter it during the ETL configuration.

See Obtain the AWS account ID step in the Configure the Main AWS account section.

Configure a policy (tsco-aws-etl-policy) to specify the permissions for the user of the additional AWS account.

See Configure a policy step in the Configure the main AWS account section.

Create a cross-account access role (tsco-cross-account-role).

This step enables the main AWS account user (tsco-etl-user) to have federated read-only access to the AWS services in the additional account and to enable account switching.

  Steps to create a cross-account access role.
  1. In the IAM service, select the Roles tab and click Create new role.
  2. Select Role for cross-account access and click Select for the option: Provide access between your AWS account and a 3rd party AWS account.

  3. Enter the account ID of the main AWS account and the external ID that you generated in an earlier step.



  4. In the Attach Policy step, select the Access Privilege policy (tsco-aws-etl-policy).
  5. Specify the role name as tsco-cross-account-role and click Create role. 
    The role is created.

6. Select the role that you just created and in the Trust relationships tab, click Edit trust relationship.

7. In the Edit Trust Relationship page, replace the "root" element with the IAM user name that you created in the main account (tsco-etl-user)

8. Click Update Trust Policy to save the updates.

Access the main AWS account again.

Log on to the AWS Management Console.  

Configure a policy file (tsco-assume-role-policy.json) to include the additional account details.

If you are configuring the first additional AWS account, you need to create a policy file. Else, you need to update the existing file with the additional AWS account details.

Information

One policy JSON file can include details of all the additional AWS accounts.


  Steps to create a policy file
  1. Open a new file in any text editor such as Notepad.
  2. Copy the following content in the file and replace ADDITIONAL_ACCOUNT_ID by the account ID of the additional account that you obtained in the previous step.

    {
    	"Version": "2012-10-17",
    	"Statement": [
    		{
         		"Sid": "Stmt1500499562000",
         		"Effect": "Allow",
         		"Action": [
           			"sts:AssumeRole"
         		],
         		"Resource": [
            		"arn:aws:iam::ADDITIONAL_ACCOUNT_ID:role/tsco-cross-account-role"
         		]
        	}
    	]
    }
  3. Save the file as tsco-assume-role-policy.json.
  Steps to update an existing policy file.
  1. In the already created JSON file (tsco-assume-role-policy.json), add the next additional account information on a new line, separated by a comma.

    {
    	"Version": "2012-10-17",
    	"Statement": [
    		{
         		"Sid": "Stmt1500499562000",
         		"Effect": "Allow",
         		"Action": [
        			"sts:AssumeRole"
         		],
         		"Resource": [
            		"arn:aws:iam::ADDITIONAL_ACCOUNT_ID1:role/tsco-cross-account-role",
            		"arn:aws:iam::ADDITIONAL_ACCOUNT_ID2:role/tsco-cross-account-role"
         		]
        	}
    	]
    }

2. Save the file.

Enable the policy (tsco-assume-role-policy.json), which includes additional AWS account details, in the main AWS account.

  Steps to enable the policy.
  1. Select the IAM service and select Users.
  2. Select the IAM user (tsco-etl-user) that you created in an earlier step.
  3. In the Summary page, select the Permissions tab and click + Add inline policy.
  4. Select Custom policy and in the Review Policy page, enter the contents of the policy file (tsco-assume-role-policy.json).
  5. Click Validate Policy and then, click Apply Policy.

 

 To configure and run the ETL

  1. In the TrueSight Capacity Optimization console, navigate to Administration ETL & SYSTEM TASKS > ETL tasks.
  2. In the ETL tasks page, under the Last run tab, click Add > Add ETL
    The Add ETL page is displayed.
  3. In the Run configuration tab, for the ETL module property, select Amazon Web Services - AWS API Extractor.
    Specify values for the properties under each expandable tab. For details about the common properties, see ETL common configuration properties

    Note

    By default, the most common, basic properties that you can set for an ETL are displayed in the Add ETL page. You can accept these default selections.

  4. In the Entity catalog tab, ensure that you select the same entity catalog that is used by the Amazon Web Services - AWS API Extractor ETL module.

    1. In the Amazon Web Services Connection tab, specify values for the following properties:

      Basic properties

      PropertyDescription
      AWS Account access mode

      Select the AWS account access mode depending on whether you want to retrieve data from a single account or multiple accounts and specify the corresponding property values. These values are based on the prerequisite steps that you performed.

      • Single: Specify the following property values: 

        Access Key IDSpecify the access key ID of the IAM user of the AWS account. For example, a typical Access Key ID might look like this: AMAZONACSKEYID007EXAMPLE.
        Secret Access KeySpecify the secret access key associated with the Access Key ID. For example, a typical Secret Access Key might look like this: wSecRetAcsKeYY712/K9POTUS/BCZthIZIzprvtEXAMPLEKEY .
      • Multiple: Specify the following property values:

        Access Key IDSpecify the access key ID of the IAM user of the AWS account. For example, a typical Access Key ID might look like this: AMAZONACSKEYID007EXAMPLE.
        Secret Access KeySpecify the secret access key associated with the Access Key ID. For example, a typical Secret Access Key might look like this: wSecRetAcsKeYY712/K9POTUS/BCZthIZIzprvtEXAMPLEKEY .
        Cross-account role nameSpecify the name of the cross-account access role that you created.
        Cross-account role external ID Specify the name of the external ID that you used while defining the cross-account role.
        List of additional (or linked) account IDsSpecify the account IDs of all the additional accounts that you have configured.
      Use Proxy

      Select Yes to configure a proxy server. By default, No is selected.

      If you selected Yes, provide the following information in the respective field:

      • Proxy server host: The host name of your proxy server.
      • Proxy server port: The port number of your proxy server.
      • Is authentication required?: If the proxy server requires user name and password for authentication, select Yes, and specify the following details:
        • Proxy server username: The name of a proxy server user.
        • Proxy server password: The password for the user.
        If the proxy server does not require authentication, select No, and skip the user name and password fields.
        The default selection is Yes.

      By default, the proxy server uses the HTTPS protocol for communication.

      Business Service Hierarchy
      Specify if you want the ETL to use your existing business service tag key to display data in a hierarchy of business services and related resources. The option, Create Business Service hierarchy based on specific tag, is selected by default, and business service key "Service' is displayed. In this option, you must specify the following property: Business Service Hierachy: Specify the tag key name that you use to mark business services. For example: Service.

      Based on your input, the ETL creates a business service entity, and maps resources to each business service.
      For example, if you have VMs tagged as follows:


        • AS1: {user=John, Purpose=Dev, Service=Data Solutions}
        • vl-pub-bco-qa35: {user=Adam, Purpose=Production, Service=Data Solutions}
        • vl-pun-bco-qa20: {user=Jane, Purpose=QA, Service=Data Solutions}

      Then the ETL displays data in a hierarchy as follows:

       Advanced properties

      Property

      Description
      Default regionThe region where your AWS resources are located.
      Instance type definition JSON file pathThe path where you saved the JSON file that have the instance type configuration metrics. For more information, see Collecting data for additional instance type configuration metrics.
      Additional CloudWatch metrics JSON file pathThe path where you saved the JSON files that have the additional metrics. For more information, see Collecting data for additional CloudWatch metrics.

  5. Click Save.
    You return to the Last run tab under the ETL tasks page.

  6. Run the ETL in the simulation mode, and validate the results:
    1. In the ETL tasks table under ETL tasks > Last run, locate your ETL (ETL task name), and click Run .
      The Last exit column in the ETL tasks table displays one of the following values:
      • OK: The ETL executed without any error in simulation mode.
      • WARNING: The ETL execution returned some warnings in simulation mode. Check the ETL log.
      • ERROR: The ETL execution returned errors and was unsuccessful. Edit the active Run configuration and try again.
    2. Check the log or edit the ETL configuration if the ETL did not run as expected or failed.
  7. After you verify that the ETL is running correctly, complete these steps to run the ETL in the production mode:
    1. In the ETL tasks table under ETL tasks > Last run, click the ETL name under the Name column.
    2. In the Run configurations table in the ETL details page, click Edit  to edit the active run configuration.
    3. On the Edit run configuration page, navigate to the Run configuration expandable tab and set Execute in simulation mode to No.
    4. Click Save.
  8. Locate the ETL in the ETL tasks table and either schedule the ETL run or click Run  to run it now.
    When the ETL is run, it extracts data from the source and transfer it to the TrueSight Capacity Optimization database.
Was this page helpful? Yes No Submitting... Thank you

Comments