Microsoft Azure - Cost and Usage Extractor

Use the Microsoft Azure - Cost and Usage Extractor (ETL) to extract daily cost and usage data and metadata for your Azure accounts. The extracted data is saved in the Capacity Optimization database. The Cost Estimation backend service uses this data to compute the cost estimates for TrueSight Cloud Cost Control. 

You can configure the ETL to collect data from the following types of Azure subscriptions:

  • Pay-As-You-Go
  • Enterprise Agreement

To collect data from the EA subscriptions, apply hotfix CHF#021 or later to TrueSight Capacity Optimization. You can download the hotfix from  this location Open link .

Information

There is a data latency of up to three days for the cost and usage billing data to be available for extraction.

For more information, see the following sections:


Related topics

Microsoft Azure - Azure API Extractor

Analyzing and forecasting multi-cloud costs

Defining and managing the on-premises costs

Working with ETLs Open link
Additional resources

Microsoft Azure API documentation Open link

Prerequisites

To configure the ETL, you need some parameter values about your Microsoft Azure subscriptions. The ETL uses these values to connect to the Azure cloud for extracting the billing and usage data of your Azure resources. 

For Pay-As-You-Go Azure subscriptions:
To collect the billing details of your Azure resources, the ETL needs to make the REST API calls to Azure services. To authenticate with these APIs, you must specify the following details during the ETL configuration:

  • Azure subscription ID
  • Azure offer ID
  • Azure Active Directory (AAD) tenant ID
  • Application ID
  • Authentication key
  • Business service tag key

For Enterprise Agreement Azure subscriptions:
To collect the billing details of your Azure resources, the ETL needs to access the billing API. To authenticate with the billing API, you must specify the following details during the ETL configuration:

  • Enrollment number
  • API access key

Complete the prerequisite steps given in the following tables to obtain these details.

Prerequisites to collect data from the Pay-As-You-Go Azure subscriptions

Prerequisite stepReference instructions/topics

Get your Azure subscription ID.

The subscription ID is a GUID that uniquely identifies your subscription to use Azure services.

1. Log on to the Azure portal.

2. In the left navigation panel, click Subscriptions. The list of your subscriptions is displayed along with the subscription ID.

Obtain the ID of the Azure pricing offer that is associated with the subscription.

1. Log on to the Azure portal.

2. In the left navigation panel, click Subscriptions and select the subscription that you want to get the Offer ID for.

3. On the Overview page, get the Offer ID.

If you want to retrieve the cost and usage data for multiple subscriptions through a single ETL, create a Subscription list file that contains all the subscription details.

The Subscription list file contains the following items for every subscription, separated by a (;):

  • Subscription ID
  • Pricing offer ID

The file can be in .txt or .csv format. You need a separate subscription file per tenant.

Important: If you are using this ETL along with the Microsoft Azure - Azure API extractor, you can use a common Subscription list file. The pricing offer ID values that you provide in the file are ignored by the Microsoft Azure - Azure API extractor ETL.

  1. Open a new .txt or .csv file.
  2. Add the subscription ID and offer ID in the following format and press Enter:
    <subscription ID1>;<Offer ID1>
  3. Repeat step 2 for every subscription.

Ensure that you have the required permissions to create an application in Azure Active Directory (AAD).  Check Azure Active Directory permissions Open link
Create an AAD application to gain access to Azure resources on behalf of the ETL.  Create an Azure Active Directory application Open link
Get the Application ID and generate an authentication key for this application.  Get application ID and authentication key Open link

Get the Tenant ID, which is the ID of the AAD directory in which you created the application.

A Tenant is representative of an organization within Azure Active Directory. It is a dedicated instance of the Azure AD service. An AAD tenant is required for defining an application and for assigning permissions so the application can make use of other Azure services' REST APIs.

  Get tenant ID Open link
Assign API access to your application.

  1. Log on to the Microsoft Azure Resource Manager portal. The Dashboard opens.
  2. In the left pane, select Azure Active Directory. The Overview page is displayed.

  3. In the left pane of Azure Active Directory, select App Registrations, and in the right pane, select the application that you created in AAD.

  4. Click Required Permissions +Add.
  5. In the Add API access page, complete the following steps:
    1. In the Select an API field, click Windows Azure Service Management API and click Select.
       
    2. In the Select permissions field, click Access Azure Service Management as organization users, and then click Select.
      Note: If you select the DELEGATE PERMISSIONS check box before selecting the permission, the Select button is not enabled.
       
  6. Click Done.

Assign reader role to the application.
  1. In the left pane of the Azure portal menu, select Subscriptions.
  2. Select your subscription.
  3. Select the Access Control (IAM) tab, and assign the Reader role to the application.

For details, see Assign application to role Open link .

Prerequisites to collect data from the EA Azure subscriptions

StepDetails

Obtain your enrollment number.

An enrollment is like a master account that is associated with your organization's EA. All your Azure subscriptions and bills are associated with this enrollment number. When enterprise administrators first sign up for Azure, they receive an enrollment number and an access key from Microsoft. Administrators can then sign in to the Azure EA portal and start performing the admin tasks.

  1. Log in to the Azure EA portal as an enterprise administrator.
    https://ea.azure.com
  2. Click Manage. The enrollment number is listed in the Enrollment Detail section.
Obtain the API access key that is required for authentication.
  1. Log in to the Azure EA portal as an enterprise administrator.
    https://ea.azure.com
  2. Click Reports > Download Usage > API Access Key.
  3. Do one of the following steps:
    • If the primary key is available, click expand key and copy it.
    • If the primary key is not available, click Generate key, and then copy it.
Grant rights to the account owners and department administrators for viewing cost data.
  1. Log in to the Azure EA portal as an enterprise administrator.
    https://ea.azure.com
  2. Click Reports.
  3. Under Enrollment, enable the following:
    • DA view changes
    • AO view changes

Additionally, ensure that the following requirements are met:

  • The Microsoft Azure - Azure API extractor is already configured to extract the metrics and relationships of Azure resources. 
  • If you want to view business service data in the Truesight Cloud Cost Control UI, configure the Microsoft Azure - Azure API extractor to collect tagging related data. After the ETL runs, the Workspace displays a hierarchy of business service entities and related resources. The Business Service Tag Key that is used in the Azure portal to tag the business services is used in this ETL to organize the cost and usage data as per the business services.

To configure the ETL module

To integrate TrueSight Capacity Optimization with the extractor, complete the following steps:

  1. In the TrueSight Capacity Optimization console, navigate to Administration ETL & SYSTEM TASKS > ETL tasks.
  2. In the ETL tasks page, under the Last run tab, click Add > Add ETL.
    The Add ETL page is displayed.
  3. In the Run configuration tab, for the ETL module property, select Microsoft Azure - Cost and Usage Extractor.
    Specify values for the properties under each expandable tab. For details about the common properties, see ETL common configuration properties

    Note

    By default, the most common, basic properties that you can set for an ETL are displayed in the Add ETL page. You can accept these default selections.

  4. In the Entity catalog tab, ensure that you select the same entity catalog that is used by the Microsoft Azure - Azure API Extractor ETL module.

  5. In the Microsoft Azure configuration tab, specify values for the following properties:

    Basic properties

    If you are using EA Azure subscriptions, configure the following properties to import the cost and usage details:

    PropertyDescription
    Enterprise Agreement accountSelect Yes.
    Enrollment numberSpecify the enrollment number that you obtained from Microsoft.
    API access keySpecify the API access key that you obtained during the preconfiguration tasks.

    If you are using Pay-As-You-Go Azure subscriptions, configure the following properties to import the cost and usage data.

    PropertyDescription
    Enterprise Agreement accountSelect No.
    Subscription access modeSelect the subscription depending on whether you want to retrieve data from a single subscription or many subscriptions.
    • Single: Specify the following property values:
      In the Subscription ID property, specify the ID of the subscription for which you want to retrieve cost and usage data.
      In the Offer ID property, specify the Azure pricing offer ID. The Offer ID code format consists of the “MS-AZR-“ prefix, plus the unique Offer ID number
    • Multiple: In the Multiple subscription file path box, specify the path of the file (.txt or .csv) that contains the information of all the subscriptions for which you want to retrieve cost and usage data.
      The file contains the following items, separated by a semi-colon (;), for each subscription:
      • Subscription ID
      • Pricing offer ID
    Tenant IDSpecify the Directory ID from your Active Directory properties.
    Application IDSpecify the Application ID from App registrations in Azure Active Directory.
    Authentication keySpecify the key that you generated when you created the web application in the Azure Active Directory.

    Additionally, configure the following properties:

    PropertyDescription
    Business Service Tag KeySpecify the business service tag key that you used in the Microsoft Azure - Azure API extractor for importing business services.

    The Microsoft Azure - Azure API extractor creates business service entities in the Workspace, and maps resources to each business service. By mapping the tags of the imported resources with those of the workspace entities, the Microsoft Azure - Cost and Usage Extractor associates the cost of a resource under a specific business service.

    The default tag key is Service.

    Use Proxy

    If you have configured a proxy server to route the internet traffic to and from your Azure environment, you can configure the ETL to connect with your environment via the proxy server.
    If you want to use a Proxy server, provide the Proxy server host and port. To authenticate the Proxy server, provide the user name and the password.
    The default selection is not to use a Proxy server.

    Advanced properties

    Property

    Description
    Extraction mode
    Depending on the period that you want to extract data for, select one of the following options:
    • Regular - daily import: Import daily data.
      In the first run, the ETL extracts data for the past six months. On subsequent runs, it extracts the daily data (after last counter value). Use this option for your daily runs.
    • Historical - import historical data (do not use for daily scheduling): Import historical data for the period that you specify in the Extraction months property.
      By default, the Extraction months field contains 12.

    Recommendation

    BMC recommends that you configure a separate ETL to extract historical data. Do not use this ETL for your daily runs.

    By default, Regular is selected. The last counter value (LAST_EXTRACTION_MAX_TIME) is set to the day when the data was last extracted.

  6. Click Save.
    You return to the Last run tab under the ETL tasks page.
  7. In simulation mode, validate the results: In the ETL tasks table under ETL tasks > Last run, locate your ETL (ETL task name), click Run  to run the ETL.
    After you run the ETL, the Last exit column in the ETL tasks table will display one of the following values:
    1. OK: The ETL executed without any error in simulation mode.
    2. WARNING: The ETL execution returned some warnings in simulation mode. Check the ETL log.
    3. ERROR: The ETL execution returned errors and was unsuccessful. Edit the active Run configuration and try again.
  8. After you verify that the ETL is running correctly, switch the ETL to production mode:
    1. In the ETL tasks table under ETL tasks > Last run, click the ETL name under the Name column.
    2. In the Run configurations table in the ETL details page, click Edit  to edit the active run configuration.
    3. In the Edit run configuration page, navigate to the Run configuration expandable tab and set Execute in simulation mode to No.
    4. Click Save.
  9. Locate the ETL in the ETL tasks table and either schedule an ETL run or click Run  to run it now.
    After you run the ETL, or schedule the ETL for a run, it will extract the data from the source and transfer it to the TrueSight Capacity Optimization database.

API calls for the Pay-As-You-Go Azure subscriptions

The ETL makes the following calls to the Azure Billing REST API to retrieve data from your Pay-As-You-Go Azure subscriptions.

TaskResource Management API call
AuthenticationPOST https://login.windows.net/<tenant_id>/oauth2/token (Application ID, Authentication key)
List all subscription detailsGET
Get resource rate card

GET https://management.azure.com/subscriptions/<Subscription_ID>/providers/Microsoft.Commerce/RateCard?api-version=2015-06-01-preview&$filter=OfferDurableId eq 'Offer Code' and Currency eq 'USD' and Locale eq 'en-US' and RegionInfo eq 'US'

 
Get resource usage

GET https://management.azure.com/subscriptions/<Subscription ID>/providers/Microsoft.Commerce/UsageAggregates?api-version=2015-06-01-preview&reportedStartTime=<Reported start date>&reportedEndTime=<Reported end date>&aggregationGranularity=Daily

Get consumption data for enrollment numberGET https://consumption.azure.com/v2/enrollments/<enrollment-number>/usagedetailsbycustomdate?startTime=<start-date>&endTime=<end-date>

API calls for the EA Azure subscriptions

The ETL makes the following call to the Azure consumption REST API to retrieve data from your EA subscriptions. 

For more information, see the Microsoft Azure Billing API Documentation Open link .

Was this page helpful? Yes No Submitting... Thank you

Comments