Troubleshooting the AWS API ETL

The AWS API ETL might fail due to session timeout or other issues. Review the ETL logs to investigate and troubleshoot the issues.

Resolutions for common issues

The following table explains the common issues for the ETL failure and their resolutions.

SymptomCauseResolution
One or more performance metrics are not available for a virtual machine instance.

On the Monitoring tab of the AWS console, the metrics are not available for the virtual machine instance.

  • Review the AWS configuration by using the same user account that is configured for the ETL.
  • For details, see knowledge article 00352749 Open link (Support logon ID required).
Only a subset of metrics are collected for virtual machine instances.The AWS ETL is configured to collect only the basic set of metrics from the Amazon Elastic Compute Cloud (EC2) instances, such as CPU Utilization, Network Byte Rate, and Disk Transfer Rate.

Configure the CloudWatch agent on the virtual machine instance to collect the additional metrics. For details, see Collecting EC2 instance metrics by using the CloudWatch agent.

The ETL fails to collect data from the AWS GovCloud. The ETL log displays the following messages:

WARNING Error getting list of availability zones for region
us-gov-west-1

INFO Region us-gov-west-1 does not have any availability zones.

FAILED BCO_ETL_ERR011: Detected an abnormal ETL termination.
Reason: com.bmc.bco.aws.exception.AWSConnectionException:
Connection phase wasn't completed
at com.bmc.bco.aws.extractor.DMAWSExtractorE.connect
(DMAWSExtractorE.java:203)
at com.neptuny.cpit.etl.Engine.start(Engine.java:129) at com.neptuny.cpit.etl.ETLrun.main(ETLrun.java:171)

Caused by: com.amazonaws.SdkClientException: Unable to execute HTTP request

In the AWS ETL, the property (Is target AWS Government Cloud) to support the AWS GovCloud is not enabled.

Configure the Is target AWS Government Cloud property to enable support for the AWS GovCloud. For details, see Amazon Web Services - AWS API Extractor.

The ETL run ends with a Warning status. The ETL log displays the following message:

Unable to fetch all the instances due to AWS RequestLimitExceeded error in 5 attempts, hence skipping further calls for instances.

The default instance chunk size of 100 is insufficient for the size set in the AWS configuration (up to 1000).
  1. Navigate to Administration > ETL & system tasks > ETL tasks.
  2. Select the AWS API ETL.
  3. In the Run configurations table, click Edit  to modify the ETL task. 
  4. On the Edit run configuration page, click Advanced.
  5. Add the extract.aws.instance.chunksize property and specify a value matching the one configured in AWS.
  6. Save the changes and run the ETL.

The ETL run ends with a Warning status.

The ETL log displays the following message:

Unable to fetch all the volumes due to AWS RequestLimitExceeded error in 5 attempts, hence skipping further calls for volumes.

The default volume chunk size of 100 is insufficient for the size set in the AWS configuration (up to 500).
  1. Navigate to Administration > ETL & system tasks > ETL tasks.
  2. Select the AWS API ETL.
  3. In the Run configurations table, click Editto modify the ETL task. 
  4. On the Edit run configuration page, click Advanced.
  5. Add the extract.aws.volume.chunksize property and specify a value matching the one configured in AWS.
  6. Save the changes and run the ETL.

When you run the AWS API ETL to fetch resources data from multiple AWS accounts, the ETL fails with a session timeout error. This failure usually occurs after an hour, which is the default session timeout internal.

The ETL log displays the following messages:

Error Assuming Role " + roleName + " externalId:" + externalId + ". Reason:" + e.getMessage())

Error Connecting to Linked Account Id:" + accountId + " ExternalId:" + externalId + ". Error-Details:"+ StringTools.getStacktrace(e));

Unable to complete discovery of Amazon Web Service for 'Linked Account': " + accountId +". reason:"+e.getMessage());

This issue occurs when you have large number of instances provisioned in your AWS accounts or a slow network connection.
  1. Configure the ETL to increase the session timeout value:
    1. Navigate to Administration > ETL & system tasks > ETL tasks.
    2. Select the AWS API ETL.
    3. In the Run configurations table, click Edit to modify the ETL task. 
    4. On the Edit run configuration page, click Advanced.
    5. Add the extract.aws.linkedaccount.session.timeout.seconds property and specify a value greater than 3600, which is the default value.
      This value is set when you create a cross-account role in AWS. The minimum value that AWS recommends is 900.
    6. Save the changes and run the ETL.
  2. Configure cross-account roles to increase the session duration:
    1. Open the IAM console, and sign in with your AWS account credentials: https://console.aws.amazon.com/iam/
    2. Click Services > IAM > Roles.
    3. Click the cross-account role name.
    4. Increase the value for the maximum CLI/API session duration, and save the changes.
    5. Repeat steps 2b through 2d for each cross-account role that you created for the linked accounts.

      Important

      Always make sure that the session timeout value configured for the ETL is less than the session duration value configured in AWS for cross-account roles.

Was this page helpful? Yes No Submitting... Thank you

Comments