Page tree

Skip to end of metadata
Go to start of metadata

BMC TrueSight Capacity Optimization supports integration with Microsoft Azure through Microsoft Azure - Azure API Extractor. You can use the extractor to discover and import information that is useful for capacity planning into BMC TrueSight Capacity Optimization. You can obtain the following Microsoft Azure entities:

  • Cloud Service
  • Deployment
  • Role
  • Role Instance or Virtual Machine

The extractor connects with the Windows Azure Service Management REST API, which is used to manage Windows Azure resources. For more information about Microsoft Azure, see Introducing Microsoft Azure .

Remember

The TrueSight Capacity Optimization V10.5 supports only classic deployment model of Azure. Hence, Service Management REST API provides information of Azure classic resources only.


The topic has the following sections:

Before you begin

Before you can use the extractor, complete the following tasks:

  • Obtain a subscription ID: The subscription ID is a GUID that uniquely identifies your subscription to use the Azure services.
  • Upload the management certificate to the Microsoft Azure portal: A management certificate is an SSL certificate that is a small data file that digitally binds a cryptographic key to an organization's details. This certificate is associated with your subscription and is required to authenticate your API calls. Create the management certificate and upload it in the Azure portal. For detailed steps, see Creating and using a management certificate.
 


Integrating TrueSight Capacity Optimization with the Microsoft Azure extractor

To integrate BMC TrueSight Capacity Optimization with the extractor, complete the following steps:

  1. In the TrueSight Capacity Optimization console, navigate to Administration > ETL & SYSTEM TASKS > ETL tasks.
  2. In the ETL tasks page, under the Last run tab, click Add > Add ETL.
    The Add ETL page is displayed.
  3. In the Run configuration tab, for the ETL module property, select Microsoft Azure - Azure API Extractor.

  4. Specify values for the following properties under each expandable tab:


    Note

    By default, the most common, basic properties that you can set for an ETL are displayed in the Add ETL page. You can accept these default selections.

    Basic properties

    Property Description
    Run configuration
    ETL task name  Accept the default name based on the selected ETL module or specify a different name for the ETL task. Duplicate names are allowed.
    Run configuration name Review the default name. This field is used to differentiate configurations that you can specify for the ETL task. You can then run the ETL task based on the configuration name.
    Environment Select Production or Test to mark the ETL tasks. For example, you can start by marking the task as Test and change it to Production after you have seen that you are getting what you wanted.
    Description (Optional) Enter a brief description for this ETL.
    Log level Select how detailed you want the ETL log to be. The log includes Error, Warning and Info type of log information.
    • 1 - Light: Add bare minimum activity logs to the log file.
    • 5 - Medium: Add medium-detailed activity logs to the log file.
    • 10 - Verbose: Add detailed activity logs to the log file.

    Note: Log levels 5 and 10 are typically used for debugging or troubleshooting ETL issues. Using a log level of 5 is general practice; however, you may choose level 10 to get a high level of detail while troubleshooting.
    Execute in simulation mode Select Yes if you want to validate the connectivity between the ETL engine and the target, and to ensure that the ETL does not have any other configuration issues.
    When set to Yes, the ETL will not store actual data into the data warehouse. This option is useful while testing a new ETL task.
    ETL module

    Ensure that Microsoft Azure - Azure API Extractor is selected.

    Note: You updated this property in the earlier step.

    Module description Review the short description of the ETL module.
    Entity catalog
    Sharing status Set the sharing status by selecting one of the following options:
    • Shared entity catalog: Select this option if, for the same entities, data is coming from multiple sources, for example, BPA ETL.
      • Sharing with Entity Catalog: Available when you select Shared entity catalog. Select an entity catalog from the drop-down list.
    • Private entity catalog: Select this option if, for the same entity, data is coming from a single source.
    Object relationships
    Associate new entities to

    Specify the domain where you want to add the entities created by the ETL. You can select an existing domain or create a new one.

    info Info: By default, a new domain is created for each ETL, with the same name as the extractor module. As the ETL is created, a new hierarchy rule with the same name as the ETL task is created automatically, with an active status. If you update the ETL specifying a different domain, the hierarchy rule is updated automatically.

    Select any one of the following options:

    • New domain: Create a new domain. Specify the following properties:
      • Parent: Select a parent domain for your new domain from the domain selector control.
      • Name: Specify a name for your new domain.
    • Existing domain: Select an existing domain. Make a selection for the following property:
      • Domain: Select an existing domain from the domain selector control.
        If the selected domain is already used by other hierarchy rules, a Domain conflict option is displayed. Select one of the following options:
        • Enrich domain tree: Create a new independent hierarchy rule to add a new set of entities, relations, or both that are not defined by other ETLs (for example this ETL is managing storage while others are managing servers).
        • ETL Migration: This configuration is recommended if a  new ETL manages the same set of entities, relations, or both (already defined in current domain tree). A typical use case is the migration from one or more ETLs to a new ETL instance. It will stop all relations imported by ETL instances and restore only valid relations after the first run; this configuration reuses an existing hierarchy rule to correctly manage relation updates.
    Microsoft Azure Connection

    Subscription ID

    Specify the ID of the subscription to which your cloud service belongs.
    Keystore file path

    Specify the path to the keystore file that is stored on the computer where BMC TrueSight Capacity Optimization is installed. The keystore file contains the certificate entry for Azure authentication. 

    Keystore file password
    Enter the password that is required to access the keystore file.
    ETL task properties
    Task group (Optional) Select a task group by which to classify this ETL.
    Running on scheduler Select the scheduler over which you want to run the ETL. The type of schedulers available are:
    • Primary Scheduler: Runs on the Application Server.
    • Generic Scheduler: Runs on a separate machine.
    • Remote: Runs on different remote machines.
    Maximum execution time before warning Select the number of hours, minutes, or days to to execute the ETL for before generating warnings, if any.
    Frequency Select the frequency of ETL execution. Available options are:
    • Predefined: Select a Predefined frequency from Each Day, Each Week, or Each Month.
    • Custom: Enter a Custom frequency (time interval) as the number of minutes, hours, days, or weeks in which you want to run the ETL.
    Start timestamp: hour\minute
    (Applies to Predefined frequency)
    Select the HH:MM start timestamp to add to the ETL execution that is running on a Predefined frequency.
    Custom start timestamp Select a YYYY-MM-DD HH:MM timestamp to add to the ETL execution that is running on a Custom frequency.

    To view or configure advanced properties, click Advanced. You do not need to set or modify these properties unless you want to change the way the ETL works. These properties are for advanced users and scenarios only.

    Advanced properties

    Property Description
    Run configuration
    Datasets

    Select or clear metric groups for which data will be populated in the Available datasets. The connector allows you to choose only from the given list of datasets, and you cannot include additional datasets to the run configuration of the ETL.

    1. Click Edit.
    2. Select one (click) or more (shift+click) datasets that you want to exclude from Available datasets and click >> to move them to Selected datasets.
    3. Click Apply.
    Collection level
    Metric profile selection

    Select either of the following options:

    • Use Global metric profile: Select this option to use an out-of-the-box global profile that is available on the Metric profiles page. By default, all ETL modules use this profile.
    • Select a custom metric profile: Select from any of the metric profiles that you add in the Add metric profile page (Administration > DATAWAREHOUSE > Metric profiles).

    For more information, see Metric profiles.

    Levels up to

    Define the amount of metric imported into the data warehouse. Increase the level to add load to the data warehouse or decrease the metric level to reduce the number of imported metrics.

    Choose the metric level to apply on selected metrics:
    • [1] Essential
    • [2] Basic
    • [3] Standard
    • [4] Extended

    For more information, see Aging Class mapping.

    Microsoft Azure Connection
    Instance type definition JSON file path Specify the path where you saved the JSON file that has the instance type configuration metrics. For more information, see Microsoft Azure - Azure API Extractor#Collecting data for additional instance type configuration metrics.
    Additional properties
    List of properties

    Specify any additional properties for this ETL that act as user inputs during execution. You can specify values for these properties either at this time or from the "You can manually edit ETL properties from this page" link that is displayed for the ETL in view mode.

    1. Click Add.
    2. Add an additional property in the etl.additional.prop.n box.
    3. Click Apply.
      Repeat this task to add more properties.
    Loader configuration
    Empty dataset behavior Select one of the following actions to take if the loader encounters an empty dataset:
    • Warn: Warn about loading an empty dataset.
    • Ignore: Ignore the empty dataset and continue parsing.
    ETL log file name Type or verify the name of the file that contains the ETL execution log; the default value is: %BASE/log/%AYEAR%AMONTH%ADAY%AHOUR%MINUTE%TASKID
    Maximum number of rows for CSV output Select a number which limits the size of the output files.
    CSV loader output file name Type or verify the name of the file generated by the CSV loader; the default value is: %BASE/output/%DSNAME%AYEAR%AMONTH%ADAY%AHOUR%ZPROG%DSID%TASKID.
    Capacity Optimization loader output file name Type or verify the name of the file generated by the BMC TrueSight Capacity Optimization loader; the default value is: %BASE/output/%DSNAME%AYEAR%AMONTH%ADAY%AHOUR%ZPROG%DSID%TASKID.
    Detail mode Select the level of detail:
    • Standard: Data will be stored on the database in different tables at the following time granularities: Detail (configurable, by default: 5 minutes), Hourly, Daily, Monthly.
    • Raw also: Data will be stored on the database in different tables at the following time granularities: Raw (as available from the original data source), Detail (configurable, by default: 5 minutes), Hourly, Daily, Monthly.
    • Raw only: Data will be stored on the database in a table only at Raw granularity (as available from the original data source).

    For more information, see Accessing data using public views and Sizing and scalability considerations.

    Reduce priority

    Select either Normal or High.

    Remove domain suffix from datasource name  (Only for systems) Set to True to remove the domain name from the data source name. For example, server.domain.com will be saved as server.
    Leave domain suffix to system name  (Only for systems) Set to True to retain the domain name in the system name. For example: server.domain.com will be saved as the system name.
    Update grouping object definition (Only for systems) Set to True to allow the ETL to update the grouping object definition for a metric loaded by an ETL.
    Skip entity creation  (Only for ETL tasks sharing lookup with other tasks) Set to True to prevent this ETL from creating an entity. With this setting, the ETL discards data from its data source for entities not found in BMC TrueSight Capacity Optimization. It uses one of the other ETLs that share lookup to create the new entity.
    Scheduling options
    Hour mask Specify a value to execute the task only during particular hours within the day. For example, 0 – 23 or 1,3,5 – 12.
    Day of week mask Select the days so that the task can be executed only during the selected days of the week. To omit setting this filter, do not select any option for this field.
    Day of month mask Specify a value to execute the task only during particular days within a month. For example, 5, 9, 18, 27 – 31.
    Apply mask validation By default this property is set to True. Set it to False if you want to disable the preceding Scheduling options that you specified. Setting it to False is useful if you want to temporarily turn off the mask validation without removing any values.
    Execute after time Specify a value in the hours:minutes format (for example, 05:00 or 16:00) to wait before the task must be executed. This means that once the task is scheduled, the task execution starts only after the specified time passes.
    Enqueueable Select one of the following options:
    • False (Default): While a task is running, additional execution commands are ignored.
    • True: While a task is running, additional execution commands are placed in a queue and executed in order after the current task is finished.

     

     

  5. Click Save.
    You return to the Last run tab under the ETL tasks page.
  6. In simulation mode, validate the results: In the ETL tasks table under ETL tasks > Last run, locate your ETL (ETL task name), click Run  to run the ETL.
    After you run the ETL, the Last exit column in the ETL tasks table will display one of the following values:
    1. OK: The ETL executed without any error in simulation mode.
    2. WARNING: The ETL execution returned some warnings in simulation mode. Check the ETL log.
    3. ERROR: The ETL execution returned errors and was unsuccessful. Edit the active Run configuration and try again.
  7. After you verify that the ETL is running correctly, switch the ETL to production mode:
    1. In the ETL tasks table under ETL tasks > Last run, click the ETL name under the Name column.
    2. In the Run configurations table in the ETL details page, click Edit  to edit the active run configuration.
    3. In the Edit run configuration page, navigate to the Run configuration expandable tab and set Execute in simulation mode to No.
    4. Click Save.
  8. Locate the ETL in the ETL tasks table and either schedule an ETL run or click Run  to run it now.
    After you run the ETL, or schedule the ETL for a run, it will extract the data from the source and transfer it to the BMC Capacity Optimization database. You can see the entity hierarchy in the left Navigation pane under Workspace, as follows:

Metric mapping details

Configuration metrics for Deployment

BMC TrueSight Capacity Optimization metricAzure metricFormula

CLOUD_SERVICE_NAME

ServiceName

Not applicable

DEPLOYMENT_SLOT

DeploymentSlot

Not applicable

Configuration metrics for Roles

BMC TrueSight Capacity Optimization metricAzure metricFormula

CLOUD_SERVICE_NAME

ServiceName

Not applicable

DEPLOYMENT_NAMELabelNot applicable

DEPLOYMENT_SLOT

DeploymentSlot

Not applicable

Configuration metrics for Role Instances

BMC TrueSight Capacity Optimization metricAzure metricFormula

CPU_NUM

Cores

Not applicable

DISK_SIZE

VirtualMachineResourceDiskSizeInMb

VirtualMachineResourceDiskSizeInMb*1024*1024

TOTAL_REAL_MEM

MemoryInMb

MemoryInMb*1024*1024

REQUESTED_INSTANCE_TYPE

InstanceSize

Not applicable

 Performance metrics for Role and Role Instances

BMC TrueSight Capacity Optimization metricAzure metricFormula
CPU_UTILPercentage CPU(Percentage CPU)/100

DISK_READ_RATE

Disk Read Bytes/sec

Not applicable

DISK_TRANSFER_RATEDisk Read Bytes/sec; Disk Write Bytes/sec((Disk Read Bytes/sec)+( Disk Write Bytes/sec))

DISK_WRITE_RATE

Disk Write Bytes/sec

Not applicable

NET_BYTE_RATENetwork In; Network Out((Network In)+(Network Out))/3600
NET_IN_BYTE_RATENetwork In(Network In)/3600
NET_OUT_BYTE_RATENetwork Out(Network Out)/3600


Lookup information

Entity typeLookup fields
 StrongWeak
Cloud Service - Azure

CLOUDSERVICE_NAME

Not applicable
Deployment - Azure

DEPLOYMENT_GUID

Not applicable
Role - Azure

PARENT_DEPLOYMENT_GUID&&

ROLE_NAME
Not applicable

Role Instance - Azure

PARENT_DEPLOYMENT_GUID&&

PARENT_ROLE_NAME&&INSTANCE_NAME
Not applicable

Entity relationship

Parent entityChild entityRelationship type 

Cloud Service - Azure

(cs:azu)

Deployment - Azure

(depl:azu)

CS_CONTAINS_DEPL

Deployment - Azure

(depl:azu)

Role - Azure

(role:azu)

DEPL_CONTAINS_ROLE

Role - Azure

(role:azu)

Role Instance - Azure

(gm:azu)

ROLE_CONTAINS_GM


Collecting data for additional instance type configuration metrics

An out-of-the-box JSON file contains the mapping for configuration metrics collected by the extractor. This file is stored on the ETL Engine server. You can configure this file to modify the existing instance type configuration or to add new configuration metrics. For example, you can add the BY_BENCHMARK_VALUE metric in the following format:

{
 	"instanceTypeConfiguration": {
		"standard_a10":{
    		"REQUESTED_INSTANCE_TYPE": "Standard_A10",
        	"CPU_MODEL":"Sparc",         
			"BYBENCHMARK_VALUE":{
             	"SPECINT2006":"2006",    
				"SPECINT2008":"2008"
			}
		},
		"standard_a11":{
			"REQUESTED_INSTANCE_TYPE": "Standard_A11"
		}
	}
}

To upload and then use a JSON file for collecting additional instance type configuration metrics:

  1. Name the JSON file appropriately, such as msazure-metric-conf.json, and save it to your local machine.
  2. Upload the file to a folder on the ETL Engine server.
  3. To use the file, include it when you create the ETL:
    1. In the Add ETL page, after you select Microsoft Azure - Azure extractor as the ETL, click Advanced and navigate to the Microsoft Azure configuration tab.
    2. Type the file path in Instance type definition JSON file path.
    3. Click Save.
      For detailed procedure, see Integration steps.

Creating and using a management certificate

Use Keytool to create and export the management certificate.

 About Keytool

Keytool is a key and certificate management utility that allows users to administer their own public or private key pairs and associated certificates. The certificates are used in self-authentication (where the users authenticate themselves to other users or services) or data integrity and authentication services, using digital signatures.

The keytool.exe file is located in the same folder as the JRE. For example, on a Windows computer, the keytool.exe file is stored in C:\Program Files\Java\jre6\bin.


  1. Open the command console of your operating system and navigate to the directory where keytool.exe is located and run the following command to create a keystore file:

    keytool -genkey -keyalg RSA -alias <alias-name> -keystore <keystore-file-name>.jks 
    -storepass <keystore-file-password> -keysize 2048

    <alias-name> is the alias name for the certificate.
    <keystore-file-name> is the name of the keystore file that you want to create.
    <keystore-file-password> is the password for the keystore file.

    The following example shows the creation of a keystore called MyAzureKeyStore.jks with P@ssword as the password. By default, the keystore file is stored in the directory from where the utility is run. For example, for JRE7, the file is stored in <C:\Program Files\Java\jre7\bin>.

    keytool -genkeypair -alias mydomain -keyalg RSA -keystore MyAzureKeyStore.jks 
    -storepass P@ssword -keysize 2048 
  2. Provide additional information such as your name, about your organization, domain name of the server.
  3. Export the management certificate by running the following command:

    keytool -export -alias <alias-name> -storepass <keystore-file-password> -file <certificate-name> -keystore <ketstore-file-name>

    <certificate-name> is the name of the certificate that you want to export.

    See the following example of an MyAzureSMAPI.cer certificate. This certificate is created in the E:\ directory of your computer.

    keytool -export -file E:\MyAzureSMAPI.cer -keystore MyAzureKeyStore.jks -alias mydomain
  4. Log on to the Microsoft Azure portal , under the SETTINGS tab, click MANAGEMENT CERTIFICATES and upload the management certificate file.

Azure API Extractor - API calls

The extractor makes the following API calls on the cloud services.

TaskAPI call
Get subscription detailsGET https://management.core.windows.net/<subscription-id>
Get list of cloud services for subscriptionGET https://management.core.windows.net/<subscription-id>/services/hostedservices
Get cloud service detailsGET https://management.core.windows.net/<subscription-id>/services/hostedservices/<hosted-service-name>
Get role sizesGET https://management.core.windows.net/<subscription-id>/rolesizes
Get list of performance metrics for specified resource IDGET https://management.core.windows.net/<subscription-id>/services/monitoring/metricdefinitions/query?resourceId=<ms-azure-entity-resource-id>
Get list of performance metric values for specified metric name and resource IDGET https://management.core.windows.net/<subscription-id>/services/monitoring/metricvalues/query?resourceId=<ms-azure-entity-resource-id>&namespace=<namespace>&names=<metric-names>&timeGrain=<PT1H>&startTime=<start-time>&endTime=<end-time>

For more information, see the Microsoft Azure API documentation .