Microsoft Azure - Cost and Usage Extractor


Use the Microsoft Azure - Cost and Usage Extractor (ETL) to extract daily cost and usage data and metadata for your Azure accounts. The extracted data is saved in the Capacity Optimization database. The Cost Estimation backend service uses this data to compute the cost estimates for TrueSight Cloud Cost Control. 

You can configure the ETL to collect data from the following types of Azure subscriptions:

  • Pay-As-You-Go
  • Enterprise Agreement

To collect data from the EA subscriptions, apply hotfix CHF#021 or later to TrueSight Capacity Optimization. You can download the hotfix from this location.

Information

There is a data latency of up to three days for the cost and usage billing data to be available for extraction.

For more information, see the following sections:

 

Prerequisites

To configure the ETL, you need some parameter values about your Microsoft Azure subscriptions. The ETL uses these values to connect to the Azure cloud for extracting the billing and usage data of your Azure resources. 

Required parameters for ETL configuration

For Pay-As-You-Go Azure subscriptions:
To collect the billing details of your Azure resources, the ETL needs to make the REST API calls to Azure services. To authenticate with these APIs, you must specify the following details during the ETL configuration:

  • Azure subscription ID
  • Azure offer ID
  • Azure Active Directory (AAD) tenant ID
  • Application ID
  • Authentication key
  • Business service tag key

For Enterprise Agreement Azure subscriptions:
To collect the billing details of your Azure resources, the ETL needs to access the billing API. To authenticate with the billing API, you must specify the following details during the ETL configuration:

  • Enrollment number
  • API access key

Complete the prerequisite steps given in the following tables to obtain these details.

Prerequisites to collect data from the Pay-As-You-Go Azure subscriptions

Prerequisites to collect data from the EA Azure subscriptions

Additionally, ensure that the following requirements are met:

  • The Microsoft Azure - Azure API extractor is already configured to extract the metrics and relationships of Azure resources. 
  • If you want to view business service data in the Truesight Cloud Cost Control UI, configure the Microsoft Azure - Azure API extractor to collect tagging related data. After the ETL runs, the Workspace displays a hierarchy of business service entities and related resources. The Business Service Tag Key that is used in the Azure portal to tag the business services is used in this ETL to organize the cost and usage data as per the business services.

To configure the ETL module

To integrate TrueSight Capacity Optimization with the extractor, complete the following steps:

  1. In the TrueSight Capacity Optimization console, navigate to Administration ETL & SYSTEM TASKS > ETL tasks.
  2. In the ETL tasks page, under the Last run tab, click Add > Add ETL.
    The Add ETL page is displayed.
  3. In the Run configuration tab, for the ETL module property, select Microsoft Azure - Cost and Usage Extractor.
    Specify values for the properties under each expandable tab. For details about the common properties, see ETL-common-configuration-properties

    Note

    By default, the most common, basic properties that you can set for an ETL are displayed in the Add ETL page. You can accept these default selections.

  4. In the Entity catalog tab, ensure that you select the same entity catalog that is used by the Microsoft-Azure-Azure-API-Extractor ETL module.
  5. In the Microsoft Azure configuration tab, specify values for the following properties:

    The [confluence_table-plus] macro is a standalone macro and it cannot be used inline.

  6. Click Save.
    You return to the Last run tab under the ETL tasks page.
  7. In simulation mode, validate the results: In the ETL tasks table under ETL tasks > Last run, locate your ETL (ETL task name), click Run run_etl.png to run the ETL.
    After you run the ETL, the Last exit column in the ETL tasks table will display one of the following values:
    1. OK: The ETL executed without any error in simulation mode.
    2. WARNING: The ETL execution returned some warnings in simulation mode. Check the ETL log.
    3. ERROR: The ETL execution returned errors and was unsuccessful. Edit the active Run configuration and try again.
  8. After you verify that the ETL is running correctly, switch the ETL to production mode:
    1. In the ETL tasks table under ETL tasks > Last run, click the ETL name under the Name column.
    2. In the Run configurations table in the ETL details page, click Edit edit icon.png to edit the active run configuration.
    3. In the Edit run configuration page, navigate to the Run configuration expandable tab and set Execute in simulation mode to No.
    4. Click Save.
  9. Locate the ETL in the ETL tasks table and either schedule an ETL run or click Run run_etl.png to run it now.
    After you run the ETL, or schedule the ETL for a run, it will extract the data from the source and transfer it to the TrueSight Capacity Optimization database.

API calls for the Pay-As-You-Go Azure subscriptions

The ETL makes the following calls to the Azure Billing REST API to retrieve data from your Pay-As-You-Go Azure subscriptions.

Task

Resource Management API call

Authentication

POST https://login.windows.net/<tenant_id>/oauth2/token (Application ID, Authentication key)

List all subscription details

Get resource rate card

GET https://management.azure.com/subscriptions/<Subscription_ID>/providers/Microsoft.Commerce/RateCard?api-version=2015-06-01-preview&$filter=OfferDurableId eq 'Offer Code' and Currency eq 'USD' and Locale eq 'en-US' and RegionInfo eq 'US'

 

Get resource usage

GET https://management.azure.com/subscriptions/<Subscription ID>/providers/Microsoft.Commerce/UsageAggregates?api-version=2015-06-01-preview&reportedStartTime=<Reported start date>&reportedEndTime=<Reported end date>&aggregationGranularity=Daily

Get consumption data for enrollment number

API calls for the EA Azure subscriptions

The ETL makes the following call to the Azure consumption REST API to retrieve data from your EA subscriptions. 

Task

Resource Management API call

Get consumption data for enrollment number

For more information, see the Microsoft Azure Billing API Documentation.

 

Tip: For faster searching, add an asterisk to the end of your partial query. Example: cert*