Microsoft Azure - Cost and Usage Extractor
Use the Microsoft Azure - Cost and Usage Extractor (ETL) to extract daily cost and usage data and metadata for your Azure accounts. The extracted data is saved in the Capacity Optimization database. The Cost Estimation backend service uses this data to compute the cost estimates for TrueSight Cloud Cost Control.
You can configure the ETL to collect data from the following types of Azure subscriptions:
- Pay-As-You-Go
- Enterprise Agreement
To collect data from the EA subscriptions, apply hotfix CHF#021 or later to TrueSight Capacity Optimization. You can download the hotfix from this location .
Information
There is a data latency of up to three days for the cost and usage billing data to be available for extraction.
For more information, see the following sections:
Prerequisites
To configure the ETL, you need some parameter values about your Microsoft Azure subscriptions. The ETL uses these values to connect to the Azure cloud for extracting the billing and usage data of your Azure resources.
Complete the prerequisite steps given in the following tables to obtain these details.
Prerequisites to collect data from the Pay-As-You-Go Azure subscriptions
Prerequisite step | Reference instructions/topics |
---|---|
Get your Azure subscription ID. The subscription ID is a GUID that uniquely identifies your subscription to use Azure services. | |
Obtain the ID of the Azure pricing offer that is associated with the subscription. | |
If you want to retrieve the cost and usage data for multiple subscriptions through a single ETL, create a Subscription list file that contains all the subscription details. The Subscription list file contains the following items for every subscription, separated by a (;):
The file can be in .txt or .csv format. You need a separate subscription file per tenant. Important: If you are using this ETL along with the Microsoft Azure - Azure API extractor, you can use a common Subscription list file. The pricing offer ID values that you provide in the file are ignored by the Microsoft Azure - Azure API extractor ETL. | |
Ensure that you have the required permissions to create an application in Azure Active Directory (AAD). | Check Azure Active Directory permissions |
Create an AAD application to gain access to Azure resources on behalf of the ETL. | Create an Azure Active Directory application |
Get the Application ID and generate an authentication key for this application. | Get application ID and authentication key |
Get the Tenant ID, which is the ID of the AAD directory in which you created the application. | Get tenant ID |
Assign API access to your application. | |
Assign reader role to the application. |
For details, see Assign application to role . |
Prerequisites to collect data from the EA Azure subscriptions
Step | Details |
---|---|
Obtain your enrollment number. | |
Obtain the API access key that is required for authentication. | |
Grant rights to the account owners and department administrators for viewing cost data. |
Additionally, ensure that the following requirements are met:
- The Microsoft Azure - Azure API extractor is already configured to extract the metrics and relationships of Azure resources.
- If you want to view business service data in the Truesight Cloud Cost Control UI, configure the Microsoft Azure - Azure API extractor to collect tagging related data. After the ETL runs, the Workspace displays a hierarchy of business service entities and related resources. The Business Service Tag Key that is used in the Azure portal to tag the business services is used in this ETL to organize the cost and usage data as per the business services.
To configure the ETL module
To integrate TrueSight Capacity Optimization with the extractor, complete the following steps:
- In the TrueSight Capacity Optimization console, navigate to Administration > ETL & SYSTEM TASKS > ETL tasks.
- In the ETL tasks page, under the Last run tab, click Add > Add ETL.
The Add ETL page is displayed. In the Run configuration tab, for the ETL module property, select Microsoft Azure - Cost and Usage Extractor.
Specify values for the properties under each expandable tab. For details about the common properties, see ETL common configuration properties.Note
By default, the most common, basic properties that you can set for an ETL are displayed in the Add ETL page. You can accept these default selections.
In the Entity catalog tab, ensure that you select the same entity catalog that is used by the Microsoft Azure - Azure API Extractor ETL module.
In the Microsoft Azure configuration tab, specify values for the following properties:
Basic properties
If you are using EA Azure subscriptions, configure the following properties to import the cost and usage details:Property Description Enterprise Agreement account Select Yes. Enrollment number Specify the enrollment number that you obtained from Microsoft. API access key Specify the API access key that you obtained during the preconfiguration tasks.
If you are using Pay-As-You-Go Azure subscriptions, configure the following properties to import the cost and usage data.Property Description Enterprise Agreement account Select No. Subscription access mode Select the subscription depending on whether you want to retrieve data from a single subscription or many subscriptions. - Single: Specify the following property values:
In the Subscription ID property, specify the ID of the subscription for which you want to retrieve cost and usage data.
In the Offer ID property, specify the Azure pricing offer ID. The Offer ID code format consists of the “MS-AZR-“ prefix, plus the unique Offer ID number - Multiple: In the Multiple subscription file path box, specify the path of the file (.txt or .csv) that contains the information of all the subscriptions for which you want to retrieve cost and usage data.
The file contains the following items, separated by a semi-colon (;), for each subscription:- Subscription ID
- Pricing offer ID
Tenant ID Specify the Directory ID from your Active Directory properties. Application ID Specify the Application ID from App registrations in Azure Active Directory. Authentication key Specify the key that you generated when you created the web application in the Azure Active Directory. Additionally, configure the following properties:
Property Description Business Service Tag Key Specify the business service tag key that you used in the Microsoft Azure - Azure API extractor for importing business services. The Microsoft Azure - Azure API extractor creates business service entities in the Workspace, and maps resources to each business service. By mapping the tags of the imported resources with those of the workspace entities, the Microsoft Azure - Cost and Usage Extractor associates the cost of a resource under a specific business service.
The default tag key is Service.
Use Proxy If you have configured a proxy server to route the internet traffic to and from your Azure environment, you can configure the ETL to connect with your environment via the proxy server.
If you want to use a Proxy server, provide the Proxy server host and port. To authenticate the Proxy server, provide the user name and the password.
The default selection is not to use a Proxy server.Advanced properties
Property
Description Extraction mode Depending on the period that you want to extract data for, select one of the following options:- Regular - daily import: Import daily data.
In the first run, the ETL extracts data for the past six months. On subsequent runs, it extracts the daily data (after last counter value). Use this option for your daily runs. - Historical - import historical data (do not use for daily scheduling): Import historical data for the period that you specify in the Extraction months property.
By default, the Extraction months field contains 12.
Recommendation
BMC recommends that you configure a separate ETL to extract historical data. Do not use this ETL for your daily runs.
By default, Regular is selected. The last counter value (
LAST_EXTRACTION_MAX_TIME
) is set to the day when the data was last extracted.- Single: Specify the following property values:
- Click Save.
You return to the Last run tab under the ETL tasks page. - In simulation mode, validate the results: In the ETL tasks table under ETL tasks > Last run, locate your ETL (ETL task name), click Run to run the ETL.
After you run the ETL, the Last exit column in the ETL tasks table will display one of the following values:- OK: The ETL executed without any error in simulation mode.
- WARNING: The ETL execution returned some warnings in simulation mode. Check the ETL log.
- ERROR: The ETL execution returned errors and was unsuccessful. Edit the active Run configuration and try again.
- After you verify that the ETL is running correctly, switch the ETL to production mode:
- In the ETL tasks table under ETL tasks > Last run, click the ETL name under the Name column.
- In the Run configurations table in the ETL details page, click Edit to edit the active run configuration.
- In the Edit run configuration page, navigate to the Run configuration expandable tab and set Execute in simulation mode to No.
- Click Save.
- Locate the ETL in the ETL tasks table and either schedule an ETL run or click Run to run it now.
After you run the ETL, or schedule the ETL for a run, it will extract the data from the source and transfer it to the TrueSight Capacity Optimization database.
API calls for the Pay-As-You-Go Azure subscriptions
The ETL makes the following calls to the Azure Billing REST API to retrieve data from your Pay-As-You-Go Azure subscriptions.
Task | Resource Management API call |
---|---|
Authentication | POST https://login.windows.net/<tenant_id>/oauth2/token (Application ID, Authentication key) |
List all subscription details | GET |
Get resource rate card | GET https://management.azure.com/subscriptions/<Subscription_ID>/providers/Microsoft.Commerce/RateCard?api-version=2015-06-01-preview&$filter=OfferDurableId eq 'Offer Code' and Currency eq 'USD' and Locale eq 'en-US' and RegionInfo eq 'US' |
Get resource usage | GET https://management.azure.com/subscriptions/<Subscription ID>/providers/Microsoft.Commerce/UsageAggregates?api-version=2015-06-01-preview&reportedStartTime=<Reported start date>&reportedEndTime=<Reported end date>&aggregationGranularity=Daily |
Get consumption data for enrollment number | GET https://consumption.azure.com/v2/enrollments/<enrollment-number>/usagedetailsbycustomdate?startTime=<start-date>&endTime=<end-date> |
API calls for the EA Azure subscriptions
The ETL makes the following call to the Azure consumption REST API to retrieve data from your EA subscriptions.
Task | Resource Management API call |
---|---|
Get consumption data for enrollment number | GET https://consumption.azure.com/v2/enrollments/<enrollment-number>/usagedetailsbycustomdate?startTime=<start-date>&endTime=<end-date> |
For more information, see the Microsoft Azure Billing API Documentation .
Comments
Log in or register to comment.