OpenStack - OpenStack API Extractor Service

In TrueSight Capacity Optimization, integration with OpenStack is supported through the OpenStack API Extractor Service that brings in OpenStack entities into TrueSight Capacity Optimization. The ETL can collect data from multiple OpenStack domains.

The OpenStack API Extractor Service supports only VMware and KVM virtualizations.

The TrueSight Capacity Optimization integration with the OpenStack API Extractor Service supports the following OpenStack versions:

  • Mitaka
  • Newton
  • Ocata
  • Pike

 TrueSight Capacity Optimization supports OpenStack v3 API for the Identity service in OpenStack Newton and later versions.

Integration steps

Note

The following ports need to be opened for the OpenStack - OpenStack API Extractor Service:

  • Identity Service URL (default: 35357)
  • Compute Service URL (default: 8774)

To integrate TrueSight Capacity Optimization with the OpenStack - OpenStack API Extractor Service, perform the following steps:

  1. Navigate to Administration > ETL & SYSTEM TASKS > ETL tasks.
  2. In the ETL tasks page, under the Last run tab, click Add > Add ETL.
  3. In the Add ETL page, set values for the following properties under each expandable tab.


    Note

    Basic properties are displayed by default in the Add ETL page. These are the most common properties that you can set for an ETL, and it is acceptable to leave the default selections for each as is.

    Basic properties

    Property Description
    Run configuration
    ETL task name By default, this field is populated based on the selected ETL module. You can specify a different name for the ETL Task. Duplicate names are allowed.
    Run configuration name The default name is already filled out. This field is used to differentiate between various configurations you can specify for the ETL task. You can then run the ETL task based on that.
    Deploy status
    You can select Production or Test to mark the ETL tasks. For example, you can start by marking the task as Test and change it to Production after you have seen that the results are as expected.
    Description (Optional) Enter a brief description for this ETL.
    Log level Select how much detail you want the ETL log to show. The log includes Error, Warning and Info type of log information.
    • 1 - Light: Add bare minimum activity logs to the log file.
    • 5 - Medium: Add medium-detailed activity logs to the log file.
    • 10 - Verbose: Add detailed activity logs to the log file. Info,

    Note

    Log levels 5 and 10 are typically used for debugging or troubleshooting ETL issues. Using a log level of 5 is general practice, however, you may choose level 10 to get a high level of detail while troubleshooting.

    Execute in simulation mode Select Yes if you want to to validate the connectivity between the ETL engine and the target, and to ensure that the ETL does not have any other configuration issues.
    When set to Yes, the ETL will not store actual data into the data warehouse. This option is useful while testing a new ETL task.
    Module selection

    Ensure that the Based on datasource option is selected.

    Note

    If you select Based on Open ETL template, TrueSight Capacity Optimization is integrated with a Generic extractor based on the selected Open ETL template. For more information, see  Generic ETL based on a template.

    ETL module
    (BMC recommends that you make this selection before the previous properties)
    Select OpenStack - OpenStack API ExtractorService.
    Module description This property is a short description of the module, with a link in the user interface that points you to the technical document for this ETL.
    Entity catalog
    Sharing status Set the sharing status by selecting one of the following options:
    • Shared entity catalog: Select this option if, for the same entities, data is coming from multiple sources, for example, BPA ETL.
      • Sharing with Entity Catalog: Available when you select Shared entity catalog. Select an entity catalog from the drop-down list.
    • Private entity catalog: Select this option if, for the same entity, data is coming from a single source.
    Object relationships
    Associate new entities to

    Specify the domain where you want to add the entities created by the ETL. You can select an existing domain or create a new one.

    Note

    By default, a new domain is created for each ETL, with the same name of the extractor module. As the ETL is created, a new hierarchy rule with the same name of the ETL task is automatically created, with status "active," if you update the ETL specifying a different domain, the hierarchy rule is updated automatically.

    Select any one of the following options:

    • New domain: Create a new domain. Specify the following properties:
      • Parent: Select a parent domain for your new domain from the domain selector control.
      • Name: Specify a name for your new domain.
    • Existing domain: Select an existing domain. Make a selection for the following property:
      • Domain: Select an existing domain from the domain selector control.
        If the selected domain is already used by other hierarchy rules, a Domain conflict option is displayed. Select one of the following options:
        • Enrich domain tree: Create a new independent hierarchy rule to add a new set of entities, relations, or both that are not defined by other ETLs (for example this ETL is managing storage while others are managing servers).
        • ETL Migration: This configuration is recommended if a  new ETL manages the same set of entities, relations, or both (already defined in current domain tree). A typical use case is the migration from one or more ETLs to a new ETL instance. It will stop all relations imported by ETL instances and restore only valid relations after the first run; this configuration reuses an existing hierarchy rule to correctly manage relation updates.
    OpenStack configuration
    Identity service URL

    Provide the URL for the OpenStack Identity Service. The Identity Service is used to obtain tokens for accessing cloud services.
    The URL is available on the Keystone server in the OS_AUTH_URL variable of the stack.sh file.

    If the identity service has an HTTPS URL, then the ETL communicates securely over HTTPS.

    Username

    Provide the user name to access the Identity Service. This authenticated user must have an admin or ro_admin (read-only admin) role assigned for the specified tenant. Also, the specified user must have read access to all OpenStack resources and endpoints.

    The name of the role that is assigned to the user must be admin or ro-admin. It cannot be an equivalent role with a different name.

    Password required Select this option if a password is required to access the service.
    Password If a password is required to access the Identity Service, enter the password.
    Tenant name Enter the name of the Keystone tenant.
    Cloud name
    (Optional) Type a name. If you do not enter a name, a default name is created using "Cloud_" as a prefix and the OpenStack identity endpoint (host name, IP address) as the suffix.
    Domain name Specify the OpenStack domains from which you want the ETL to collect data. To specify domains, select Custom, and type the required domain names.
    KVMHypervisor Enable the collection of one of the following KVM hypervisor metrics:
    • Utilization metrics by using the agent installed on a KVM host or other data source. This is the default selection.
    • Metrics that are available from the Nova API calls
    VMware Hypervisor Enable the collection of one of the following vCenter metrics:
    • Utilization metrics by using the vCenter ETL. This is the default selection.
    • Metrics that are available from the Nova API calls
    ETL task properties
    Task group (Optional) Select a task group by which to classify this ETL.
    Running on scheduler Select the scheduler over which you want to run the ETL. The type of schedulers available are:
    • Primary Scheduler: Runs on the Application Server.
    • Generic Scheduler : Runs on a separate machine.
    • Remote: Runs on different remote machines.
    Maximum execution time before warning Indicate the number of hours, minutes or days for which to execute the ETL before generating warnings or alerts, if any.
    Frequency Select the frequency for ETL execution. Available options are:
    • Predefined: Select a Predefined frequency from Each Day, Each Week, or Each Month.
    • Custom: Enter a Custom frequency (time interval) as the number of minutes, hours, days, or weeks to run the ETL in.
    Start timestamp: hour\minute
    (Applies to Predefined frequency)
    Indicate the HH:MM start timestamp to add to the ETL execution running on a Predefined frequency.
    Custom start timestamp Select a yyyy-mm-dd hh:mm timestamp to add to the ETL execution running on a Custom frequency.

    Note

    To view or configure Advanced properties, click Advanced. You do not need to set or modify these properties unless you want to change the way the ETL works. These properties are for advanced users and scenarios only.

    Advanced properties

    Property Description
    Run configuration
    Datasets

    Enables you to select or clear metric groups for which data will be populated.

    The OpenStack connector allows you to choose only from the given list of datasets; you cannot include additional datasets to the run configuration of the ETL.

    1. Click Edit.
    2. From Available datasets, select one (click) or more (shift+click) datasets and click to move them to Selected datasets.
      To clear any dataset from Selected datasets, select one (click) or more (shift+click) datasets and click to move them to Available datasets.
    3. Click Apply.
      The selected datasets are listed.
    Collection level 
    Metric profile selection

    Select any one:

    • Use Global metric profile: Select this option to use an out-of-the-box global profile. By default, all ETL modules use this profile.
    • Select a custom metric profile...: Select from the available metric profiles you may have added in the Add metric profile page (Administration  DATA WAREHOUSE > Metric profiles).

    For more information, see Adding and managing metric profiles .

    Levels up to

    The metric level defines the amount of metric imported into the data warehouse. If you increase the level, additional load is added to the data warehouse, while decreasing the metric level reduces the number of imported metrics.

    Choose the metric level to apply on selected metrics:

    • [1] Essential
    • [2] Basic
    • [3] Standard
    • [4] Extended

    For more information, see Aging Class mapping .

    Metric filter
    Metric list for <selected> datasets Click Edit and select the metrics that will be loaded for each dataset that you selected under Run configuration > Datasets. If no metric is specified, all metrics will be loaded.
    OpenStack configuration
    Role name Specify the role name that is assigned to the user to access the OpenStack resources.
    Additional properties
    List of properties

    Specify additional properties for this ETL that act as user input during execution. You can specify values for these properties either at this time, or from the "You can manually edit ETL properties from this page" link that is displayed for the ETL in view mode.

    1. Click Add.
    2. Add an additional property in the etl.additional.prop.n box.
    3. Click Apply.
      Repeat this task to add more properties.
    Loader configuration
    Empty dataset behavior Choose one of the following actions if the loader encounters an empty dataset:
    • Warn: Warn about loading an empty dataset.
    • Ignore: Ignore the empty dataset and continue parsing.
    ETL log file name Enter the name of the file that contains the ETL execution log; the default value is: %BASE/log/%AYEAR%AMONTH%ADAY%AHOUR%MINUTE%TASKID
    Maximum number of rows for CSV output Enter a number that limits the size of the output files.
    CSV loader output file name Enter the name of the file generated by the CSV loader; the default value is: %BASE/output/%DSNAME%AYEAR%AMONTH%ADAY%AHOUR%ZPROG%DSID%TASKID
    Capacity Optimization loader output file name Enter the name of the file generated by the BMC TrueSight Capacity Optimization loader; the default value is: %BASE/output/%DSNAME%AYEAR%AMONTH%ADAY%AHOUR%ZPROG%DSID%TASKID
    Detail mode Select the level of detail:
    • Standard: Data is stored on the database in different tables at the following time granularities: Detail (configurable, by default: 5 minutes), Hourly, Daily, Monthly.
    • Raw also: Data is stored on the database in different tables at the following time granularities: Raw (as available from the original data source), Detail (configurable, by default: 5 minutes), Hourly, Daily, Monthly.
    • Raw only: Data is stored on the database in a table only at Raw granularity (as available from the original data source).

    For more information, see Accessing data using public views and Sizing and scalability considerations.

    Reduce priority

    Select either Normal or High.

    Remove domain suffix from datasource name (Only for systems) If set to True, the domain name is removed from the data source name. For example, server.domain.com will be saved as server.
    Leave domain suffix to system name (Only for systems) If set to True, the domain name is maintained in the system name. For example: server.domain.com will be saved as such.
    Update grouping object definition If set to True, the ETL will be allowed to update the grouping object definition for a metric loaded by an ETL.
    Skip entity creation (Only for ETL tasks sharing lookup with other tasks) If set to True, this ETL does not create an entity, and discards data from its data source for entities not found in BMC Capacity Optimization. It uses one of the other ETLs that share lookup to create the new entity.
    Scheduling options
    Hour mask Specify a value to execute the task only during particular hours within the day. For example, 0 – 23 or 1,3,5 – 12.
    Day of week mask Select the days so that the task is executed only during the selected days of the week. To avoid setting this filter, do not select any option for this field.
    Day of month mask Specify a value to execute the task only during particular days within a month. For example, 5, 9, 18, 27 – 31.
    Apply mask validation By default this property is set to True. Set it to False if you want to disable the preceding Scheduling options that you specified. Setting it to False is useful if you want to temporarily turn off the mask validation without removing any values.
    Execute after time Specify a value in the hours:minutes format (for example, 05:00 or 16:00) to wait before the task must be executed. This option means that after the task is scheduled, the task execution starts only after the specified time passes.
    Enqueueable Select one of the following options:
    • False(Default): While a particular task is already running, if the next execution command arises, it is ignored.
    • True: While a particular task is already running, if the next execution command arises, it is placed in a queue and is executed as soon as the current execution ends.
  4. Click Save.
    You are returned to the Last run tab under the ETL tasks page.

  5. Validate the results in simulation mode: In the ETL tasks table under ETL tasks > Last run, locate your ETL (ETL task name), click  to run the ETL.
    After you run the ETL, the Last exit column in the ETL tasks table will display one of the following values:
    • OK: The ETL executed without any error in simulation mode.
    • WARNING: The ETL execution returned some warnings in simulation mode. Check the ETL log.
    • ERROR: The ETL execution returned errors and was unsuccessful. Edit the active Run configuration and try again.
  6. Switch the ETL to production mode: Perform the following steps:
    1. In the ETL tasks table under ETL tasks > Last run, click the ETL under the Name column.
    2. In the Run configurations table in the ETL details page, click  to edit the active Run configuration.
    3. In the Edit run configuration page, navigate to the Run configuration expandable tab and set Execute in simulation mode to No.
    4. Click Save.
  7. Locate the ETL in the ETL tasks table and click  to run it, or schedule an ETL run.
    After you run the ETL, or schedule the ETL for a run, it will extract the data from the source and transfer it to the TrueSight Capacity Optimization database.

Lookup details

The OpenStack API Extractor Service defines multiple lookup fields for coordinating with other connectors. The following table lists the sequence of field sets for strong lookup and weak lookup.

Entity type
(Click to navigate to section)
Strong lookup fieldsWeak lookup fields
Cloud - OpenStackOPENSTACK_GUID-
Region - OpenStackOPENSTACK_GUID-
Availability Zone - OpenStackAZ_NAME&&PARENT_OS_GUID-
Host Aggregates - OpenStackNAME&&PARENT_OS_GUID-
Virtual Host - KVMHOSTNAMENAME
Virtual Host - VMware HOSTNAMENAME
Virtual Cluster - VMwareNAME&&VMW_VMREF NAME 
Virtual Machine - KVMUUIDNAME
Virtual Machine - VMwareHOSTNAMENAME
Tenant - OpenStackOPENSTACK_GUID-

Metric details

Nova memory metrics are collected only for the current hour when the  OpenStack - OpenStack API Extractor Service is run for the first time.

Cloud - OpenStack

MetricSource typeSource infoDescription
CLOUD_NAMENANAName of the cloud. If a user-defined name exists, that name is displayed. Otherwise, the default name is displayed which is composed of a prefix "Cloud_" and the OpenStack identity endpoint (host name, ip adders) as suffix.
HOST_NUMAPINANumber of hosts.

Derived metrics (values from host aggregates that belong to the cloud)

Configuration metricsPerformance metrics
  • CPU_NUM
  • TOTAL_REAL_MEM
  • TOTAL_EPHEMERAL_DISK_SIZE
  • GM_NUM
  • GM_ON_NUM
  • VCPU_ON_NUM
  • VCPU_NUM
  • MEM_CONSUMED
  • MEM_UTIL
  • TOTAL_EPHEMERAL_DISK_PROVISIONED

Region - OpenStack

MetricSource typeSource infoDescription
CLOUD_NAME--Name of the cloud. Any name you specify is prefixed with "Cloud_" and the OpenStack identity endpoint (host name, ip adders) is added as suffix.
HOST_NUM API -Number of hosts.

Derived metrics (values from the region that belong to the cloud):

Configuration metricsPerformance metrics
  • CPU_NUM
  • TOTAL_REAL_MEM
  • TOTAL_EPHEMERAL_DISK_SIZE
  • GM_NUM
  • GM_ON_NUM
  • VCPU_NUM
  • VCPU_ON_NUM
  • MEM_CONSUMED
  • MEM_UTIL
  • TOTAL_EPHEMERAL_DISK_PROVISIONED

Availability Zone - OpenStack

MetricSource typeSource infoDescription
CLOUD_NAME - -Name of the cloud. If a user-defined name exists, that name is displayed. Otherwise, the default name is displayed, which is composed of a prefix "Cloud_" and the OpenStack identity endpoint (host name, ip adders) as suffix.
REGION_NAMEAPI -Name of the region read from the OpenStack API.
HOST_NUM API -Number of hosts.

Host Aggregate - OpenStack

MetricSource typeSource infoDescription
CLOUD_NAME--Name of the cloud. If a user-defined name exists, that name is displayed. Otherwise, the default name is displayed, which is composed of a prefix "Cloud_" and the OpenStack identity endpoint (host name, ip adders) as suffix.
REGION_NAMEAPI-Name of the region read from the OpenStack API.
AVAILABILITY_ZONEAPI

/v2.1/<tenant_id>/os-aggregates - aggregates.metadata.availability_zone

Availability Zone read from the OpenStack API.
HOST_NUM API -Number of hosts.

Derived metrics (values from hosts that belong to the host aggregate)

Configuration metricsPerformance metrics
  • CPU_NUM
  • TOTAL_REAL_MEM
  • TOTAL_EPHEMERAL_DISK_SIZE
  • GM_NUM
  • GM_ON_NUM
  • VCPU_NUM
  • VCPU_ON_NUM
  • MEM_CONSUMED
  • MEM_UTIL
  • TOTAL_EPHEMERAL_DISK_PROVISIONED

Virtual Host - KVM, Virtual Host - VMware, Virtual Cluster - VMware

MetricSource typeSource infoDescription
CPU_NUMAPI

/v2.1/<tenant_id>/os-hypervisors/detail -> hypervisors.cpu_info.topology.cores

Hypervisor number of CPU cores configured in OpenStack.
TOTAL_REAL_MEMAPI

/v2.1/<tenant_id>/os-hypervisors/detail -> hypervisors.memory_mb

Hypervisor total real memory configured in OpenStack.
TOTAL_EPHEMERAL_DISK_SIZEAPI

/v2.1/<tenant_id>/os-hypervisors/detail -> hypervisors.local_gb

Hypervisor ephemeral disk size configured in OpenStack.
*HOST_STATEAPI/v2/<tenant_id>/os-hypervisors/detail - hypervisors.stateHypervisor host state configured in OpenStack.
*HOST_STATUSAPI/v2/<tenant_id>/os-hypervisors/detail - hypervisors.statusHypervior host statue configured in OpenStack.
**CLUSTER_STATEAPI/v2/<tenant_id>/os-hypervisors/detail - hypervisors.stateHypervisor cluster state configured in OpenStack.
**CLUSTER_STATUSAPI/v2/<tenant_id>/os-hypervisors/detail - hypervisors.statusHypervisor cluster status configured in OpenStack.
***VCPU_ON_NUMAPI/v2/<tenant_id>/os-hypervisors/detail - hypervisors.vcpusHypervisor number of powered on virtual CPUs configured in OpenStack.
***VCPU_NUMAPI/v2.1/<tenant_id>/os-hypervisors/detail -> hypervisors.vcpus_usedHypervisor number of virtual CPUs configured in OpenStack.
***GM_ON_NUMAPIDerivedHypervisor number of powered on virtual machines configured in OpenStack.
***GM_NUMAPIDerivedHypervisor number of virtual machines configured in OpenStack.
***MEM_CONSUMED (for VMware)API/v2.1/<tenant_id>/os-hypervisors/detail -> hypervisors.memory_mb_useHypervisor total memory consumed in OpenStack for VMware.

***MEM_USED (for KVM)

API/v2.1/<tenant_id>/os-hypervisors/detail -> hypervisors.memory_mb_useHypervisor total memory used in OpenStack (for KVM).
***MEM_FREEAPI/v2.1/<tenant_id>/os-hypervisors/detail -> hypervisors.free_ram_mbHypervisor total memory available in OpenStack.
***MEM_UTILAPIDerivedHypervisor memory utilization configured in OpenStack.
***TOTAL_EPHEMERAL_DISK_PROVISIONEDAPI/v2.1/<tenant_id>/os-hypervisors/detail -> hypervisors.local_gb_usedHypervisor ephemeral disk size provisioned in OpenStack.

* -This metric is available only for Virtual Host - KVM and Virtual Host - VMware entities.

** - This metric is available only for Virtual Cluster - VMware entities.

***- This metric can also be collected by using the Nova API. The default collection method is the Agent. To use Nova API, configure the ETL. For more information, see Configuring the ETL to collect metrics from Nova API.

Virtual Machine - KVM, Virtual Machine - VMware

MetricSource typeSource infoDescription
AVAILABILITY_ZONEAPI

/v2.1/<tenant_id>/servers/<server_id>- server.OS-EXT-AZ:availability_zone

Availability Zone read from OpenStack API.
INSTANCE_NAMEAPI

/v2.1/<tenant_id>/servers/<server_id> - server.name

Name given to the server in OpenStack.
REQUESTED_INSTANCE_TYPEAPI

/v2.1/<tenant_id>/flavors/<server.flavor.id> - flavor.name

Flavor name.
***CPU_NUMAPI

/v2.1/<tenant_id>/flavors/<server.flavor.id> - flavor.vcpus

Requested number of CPU specified on OpenStack flavor associated to the server.
***DISK_SIZEAPI

/v2.1/<tenant_id>/flavors/<server.flavor.id> - flavor.disk

Requested disk size specified on OpenStack flavor associated to the server.
***TOTAL_REAL_MEMAPI

/v2.1/<tenant_id>/flavors/<server.flavor.id> - flavor.ram

Requested memory specified on OpenStack flavor associated to the server.
*** - This metric can also be collected by using the Nova API. The default collection method is the Agent. To use Nova API, configure the ETL. For more information, see Configuring the ETL to collect metrics from Nova API.

Tenant - OpenStack

MetricSource typeSource infoDescription
APP_CLOUD_NAME--Name of the Keystone cloud. The name is available on the Keystone server in the OS_CLOUD_NAME variable of the stack.sh file.
APP_CPU_NUM_QUOTAAPI

/v2.1/<tenant_id>/os-quota-sets/<curr_tenant_id> - quota_class_set.core

Maximum number of CPUs that can be assigned to the Service, Application or Tenant.
APP_INSTANCE_QUOTA   
APP_MEM_QUOTAAPI

/v2.1/<tenant_id>/os-quota-sets/<curr_tenant_id> - quota_class_set.ram

Maximum amount of Memory that can be assigned to the Service, Application or Tenant.
APP_ENVIRONMENT   

OpenStack API Extractor Service - API calls

This section lists all the API calls performed by the OpenStack API Extractor Service on the identity and compute services.

OpenStack Identity Service API

Taskv2.0 APIv3 API
Authenticate and generate a tokenPOST /v2.0/tokens (configured username, password, and tenant name are passed in the request body)POST /v3/tokens (configured username, password, and tenant name are passed in the request body)
Get list of tenants/projectsGET /v2.0/tenantsGET /v3/projects

OpenStack Compute Service API

TaskAPI
List of aggregatesGET /v2.1/{tenant_id}/os-aggregates
List of hypervisorsGET /v2.1/{tenant_id}/os-hypervisors/detail
List of servers for a hypervisorGET /v2.1/{tenant_id}/os-hypervisors/{hypervisor_hostname}/servers
Details for a specific serverGET /v2.1/{tenant_id}/servers/{server_id}
List of details for flavorsGET /v2.1/{tenant_id}/flavors/detail

For more information, refer to the OpenStack API documentation website.

Related topics

Installing capacity views

OpenStack view

Was this page helpful? Yes No Submitting... Thank you

Comments