Amazon Web Services - AWS API Extractor

Use the Amazon Web Services - AWS API Extractor to integrate with Amazon Web Service (AWS) to discover and import the AWS infrastructure data for capacity planning. The extractor makes API calls to the following AWS services:

EC2: To import EC2 instances and EBS volumes.
CloudWatch: To import metrics for EC2 instances, EBS volumes, and Auto Scaling groups.
Auto Scaling: To import the Auto Scaling groups.

Depending on your requirement, you can configure one ETL module for a single or multiple AWS accounts. For multiple accounts, one AWS account is used as the main account to retrieve data from all the AWS services and to connect to other accounts.

The extractor collects tagging information from your AWS resources. If you use tags to organize your resources by related business services, you can configure the ETL to display business services and their related resources in a hierarchy in the Workspace. This hierarchy enables you to sort and view capacity management metrics by business service.

The same hierarchy is further used by the Amazon Web Service - Cost and Usage extractor. For more information, see Collecting-business-service-data.

This integration uses the AWS Java SDK version 1.11.60.

For entities, lookup information, and metrics for the AWS - API Extractor, see Entities-lookup-information-and-metrics-for-AWS-API-Extractor.

This topic explains how to configure and run the ETL.

Prerequisites

If you want to configure an ETL to retrieve data from multiple AWS accounts, click the Multiple AWS accounts tab. Else, complete the prerequisite steps under the Single AWS account tab.

Single AWS account
Multiple AWS accounts

Single account

Multiple accounts

Click a box in the workflow diagram to view the corresponding step details.

Basic requirements

Generate an external ID that you need to use when you configure the additional AWS accounts. The external ID is an alphanumeric string. Use any alphanumeric string or use a tool such as the GUID UNIX tool to generate it.
To organize your resources by business services, ensure that you tag your resources by using a business service tag key name such as Service. You need to specify this business service tag key name during ETL configuration.
For more information, see Collecting-business-service-data.

Configure the main AWS account

Configure the additional AWS account

You must repeat these steps for every additional AWS account.

To configure and run the ETL

In the TrueSight Capacity Optimization console, navigate to Administration > ETL & SYSTEM TASKS > ETL tasks.
In the ETL tasks page, under the Last run tab, click Add > Add ETL.
The Add ETL page is displayed.
In the Run configuration tab, for the ETL module property, select Amazon Web Services - AWS API Extractor.
Specify values for the properties under each expandable tab. For details about the common properties, see ETL common configuration properties.
Note
By default, the most common, basic properties that you can set for an ETL are displayed in the Add ETL page. You can accept these default selections.
In the Entity catalog tab, ensure that you select the same entity catalog that is used by the ETL module.
1. In the Amazon Web Services Connection tab, specify values for the following properties:
  The [confluence_table-plus] macro is a standalone macro and it cannot be used inline. Click on this message for details.
  This macro generates standalone content. As a consequence you need to make sure to use a syntax that separates your macro from the content before and after it so that it's on a line by itself. For example in XWiki Syntax 2.0+ this means having 2 newline characters (a.k.a line breaks) separating your macro from the content before and after it.
Click Save.
You return to the Last run tab under the ETL tasks page.
Run the ETL in the simulation mode, and validate the results:
1. In the ETL tasks table under ETL tasks > Last run, locate your ETL (ETL task name), and click Run .
  The Last exit column in the ETL tasks table displays one of the following values:
  - OK: The ETL executed without any error in simulation mode.
  - WARNING: The ETL execution returned some warnings in simulation mode. Check the ETL log.
  - ERROR: The ETL execution returned errors and was unsuccessful. Edit the active Run configuration and try again.
2. Check the log or edit the ETL configuration if the ETL did not run as expected or failed.
After you verify that the ETL is running correctly, complete these steps to run the ETL in the production mode:
1. In the ETL tasks table under ETL tasks > Last run, click the ETL name under the Name column.
2. In the Run configurations table in the ETL details page, click Edit to edit the active run configuration.
3. On the Edit run configuration page, navigate to the Run configuration expandable tab and set Execute in simulation mode to No.
4. Click Save.
Locate the ETL in the ETL tasks table and either schedule the ETL run or click Run to run it now.
When the ETL is run, it extracts data from the source and transfer it to the TrueSight Capacity Optimization database.