VMware - vCloud Infrastructure extractor

This topic describes how to integrate BMC Helix Continuous Optimization with VMWare vCloud Director to extract vCloud infrastructure data. The ETL imports infrastructure data from VMware vCloud Director.  

In the vCloud Infrastructure ETL, the following entities are fetched by Provider vDCs and Organization vDCs: 

  • For each Provider vDC, related Organization vDCs, Hosts, Resource pools, Datastores and Datastore Clusters are fetched.
  • For each Organization vDC, related Resource pools are fetched.

The VMWare infrastructure hierarchy is refreshed every 6 hours. Removing or adding a new datastore into a datastore cluster can take upto 6 hours to reflect in BMC Helix Continuous Optimization, and during this period, no data is lost. Any change in configuration values (for example, total space of the datastore cluster) will reflect in BMC Helix Continuous Optimization in 1 hour.

Note

To collect data from 9.x and 10.x versions of VMware Cloud Director, use the VMware - Cloud Director 9.x/10.x Extractor. This ETL collects vCloud infrastructure and services data from the vCloud Director.

For more information, refer to the following sections:

Integration steps

To integrate BMC Helix Continuous Optimization with the vCloud infrastructure extractor:

  1. Navigate to Administration > ETL & SYSTEM TASKS > ETL tasks.
  2. In the ETL tasks page, click Add > Add ETL under the Last run tab.
  3. In the Add ETL page, set values for the following properties under each expandable tab.

    Note

    Basic properties are displayed by default in the Add ETL page. These are the most common properties that you can set for an ETL, and it is acceptable to leave the default selections for each as is.

    Basic properties

    Property Description
    Run configuration
    ETL task name By default, this field is populated based on the selected ETL module. You can specify a different name for the ETL Task. Duplicate names are allowed.
    Run configuration name Default name is already filled out for you. This field is used to differentiate different configurations you can specify for the ETL task. You can then run the ETL task based on it.
    Deploy status You can select Production or Test to mark the ETL tasks. For example, you can start by marking the task as Test and change it to Production after you have seen that you are getting what you wanted.
    Description (Optional) Enter a brief description for this ETL.
    Log level Select how detailed you want the ETL log to be. The log includes Error, Warning and Info type of log information.
    • 1 - Light: Add bare minimum activity logs to the log file.
    • 5 - Medium: Add medium-detailed activity logs to the log file.
    • 10 - Verbose: Add detailed activity logs to the log file. Info,

    Note: Log levels 5 and 10 are typically used for debugging or troubleshooting ETL issues. Using a log level of 5 is general practice, however, you may choose level 10 to get a high level of detail while troubleshooting.

    Execute in simulation mode Select Yes if you want to to validate the connectivity between the ETL engine and the target, and to ensure that the ETL does not have any other configuration issues. This option is useful while testing a new ETL task.
    ETL module Select VMware - vCloud Infrastructure extractor.
    Module description A link in the user interface that points you to this technical document for the ETL.
    Entity catalog
    Sharing status  Select any one:
    • Shared entity catalog: Select this option if, for the same entities, data is coming from multiple sources such as BPA, vCenter Extractor Service, or vCenter History Extractor.
      • Sharing with Entity Catalog: Select an entity catalog from the drop-down list.

        Note

        Ensure that the VMware - vCenter Extractor Service or VMware - vCenter and ESX Server History Extractor ETLs use the same entity catalog that you specify for the VMware - vCloud Infrastructure extractor ETL.

    • Private entity catalog: Select this option if, for the same entity, data is coming from a single source.
    Object relationships
    Associate new entities to

    Specify the domain to which you want to add the entities created by the ETL.

    Select one of the following options:

      • New domain: This option is selected by default. Select a parent domain, and specify a name for your new domain.
      • Existing domain: Select an existing domain from the Domain list.

    By default, a new domain with the same ETL name is created for each ETL. 

    vCloud Director ETL configuration
    vCloud Director server URL Enter the vCloud Director server URL. For example, https://abc.example.com/
    Username

    Type the username for a vCloud Director Administrator user.

    Password required Select either Yes or No.
    Password Enter the password for the credentials provided.
    ETL task properties
    Task group Select a task group to classify this ETL into. it is not necessary to group it into a task group.
    Running on scheduler Select a scheduler for running the ETL. For cloud ETLs, use the scheduler that is preconfigured in Helix. For on-premises ETLs, use the scheduler that runs on the Remote ETL Engine.
    Maximum execution time before warning The number of hours, minutes or days to execute the ETL for before generating warnings or alerts, if any.
    Frequency Select the frequency for ETL execution. Available options are:
    • Predefined: Select a Predefined frequency from Each DayEach Week, or Each Month.
    • Custom: Enter a Custom frequency (time interval) as the number of minutes, hours, days, or weeks to run the ETL in.
    Start timestamp: hour\minute (Applies to Predefined frequency) The HH:MM start timestamp to add to the ETL execution running on a Predefined frequency.
    Custom start timestamp Select a yyyy-mm-dd hh:mm timestamp to add to the ETL execution running on a Custom frequency.

    Note

    To view or configure Advanced properties, click Advanced. You do not need to set or modify these properties unless you want to change the way the ETL works. These properties are for advanced users and scenarios only.

    Advanced properties

    Property Description
    Run configuration
    Datasets

    Enables you to select or deselect metric groups for which data will be populated Available datasets. The OpenStack connector allows you to choose only from the given list of datasets, and you cannot include additional datasets to the run configuration of the ETL.

    1. Click Edit.
    2. Select one (click) or more (shift+click) datasets that you want to exclude from Available datasets and click >> to move them to Selected datasets.
    3. Click Apply.
    Collection level 
    Metric profile selection

    Select any one:

    • Use Global metric profile: Select this to use an out-of-the-box global profile, that is available on the Adding and managing metric profiles page. By default, all ETL modules use this profile.
    • Select a custom metric profile: Any metric profiles you add in the Add metric profile page (Administration > DATAWAREHOUSE > Metric profiles).

    For more information, see Adding and managing metric profiles.

    Levels up to

    The metric level defines the amount of metric imported into Continuous Optimization. If you increase the level, additional load is added to the ingestion, while decreasing the metric level reduces the number of imported metrics.

    Choose the metric level to apply on selected metrics:

    • [1] Essential
    • [2] Basic
    • [3] Standard
    • [4] Extended
    Metric filter
    Metric list <for selected dataset>

    Click Edit and select the metrics that will be loaded for each dataset that you selected under Run configuration > Datasets. If no metric is specified, all metrics will be loaded.

    Additional properties
    List of properties

    Additional properties can be specified for this ETL that act as user inputs during execution. You can specify values for these properties either at this time, or from the "You can manually edit ETL properties from this page" link that is displayed for the ETL in view mode.

    1. Click Add.
    2. Add an additional property in the etl.additional.prop.n box.
    3. Click Apply.
      Repeat this task to add more properties.
    Loader configuration
    Empty dataset behavior Choose one of the following actions if the loader encounters an empty dataset:
    • Warn: Warn about loading an empty dataset.
    • Ignore: Ignore the empty dataset and continue parsing.
    ETL log file name Name of the file that contains the ETL execution log; the default value is:%BASE/log/%AYEAR%AMONTH%ADAY%AHOUR%MINUTE%TASKID
    Maximum number of rows for CSV output A number which limits the size of the output files.
    CSV loader output file name Name of the file generated by the CSV loader; the default value is:%BASE/output/%DSNAME%AYEAR%AMONTH%ADAY%AHOUR%ZPROG%DSID%TASKID

    Continuous Optimization loader output file name

    Name of the file generated by the Continuous Optimization loader; the default value is:%BASE/output/%DSNAME%AYEAR%AMONTH%ADAY%AHOUR%ZPROG%DSID%TASKID

    Remove domain suffix from datasource name (Only for systems) If set to True, the domain name is removed from the data source name. For example, server.domain.com will be saved as server.
    Leave domain suffix to system name (Only for systems) If set to True, the domain name is maintained in the system name. For example: server.domain.com will be saved as such.
    Skip entity creation (Only for ETL tasks sharing lookup with other tasks) If set to True, this ETL does not create an entity, and discards data from its data source for entities not found in the product. It uses one of the other ETLs that share lookup to create the new entity.
    Scheduling options
    Hour mask Specify a value to execute the task only during particular hours within the day. For example, 0 – 23 or 1,3,5 – 12.
    Day of week mask Select the days so that the task can be executed only during the selected days of the week. To avoid setting this filter, do not select any option for this field.
    Day of month mask Specify a value to execute the task only during particular days within a month. For example, 5, 9, 18, 27 – 31.
    Apply mask validation By default this property is set to True. Set it to False if you want to disable the preceding Scheduling options that you specified. Setting it to False is useful if you want to temporarily turn off the mask validation without removing any values.
    Execute after time Specify a value in the hours:minutes format (for example, 05:00 or 16:00) to wait before the task must be executed. This means that once the task is scheduled, the task execution starts only after the specified time passes.
    Enqueueable Select one of the following options:
    • False(Default): While a particular task is already running, if the next execution command arises – it is ignored.
    • True: While a particular task is already running, if the next execution command arises – it is placed in a queue and is executed as soon as the current execution ends.
  4. Click Save.
    You return to the Last run tab under the ETL tasks page.
  5. Validate the results in simulation mode: In the ETL tasks table under ETL tasks > Last run, locate your ETL (ETL task name), click  to run the ETL.
    After you run the ETL, the Last exit column in the ETL tasks table will display one of the following values:
    • OK: The ETL executed without any error in simulation mode.
    • WARNING: The ETL execution returned some warnings in simulation mode. Check the ETL log.
    • ERROR: The ETL execution returned errors and was unsuccessful. Edit the active Run configuration and try again.
  6. Switch the ETL to production mode: To do this, perform the following task:
    1. In the ETL tasks table under ETL tasks > Last run, click the ETL under the Name column.
    2. In the Run configurations table in the ETL details page, click  to edit the active run configuration.
    3. In the Edit run configuration page, navigate to the Run configuration expandable tab and set Execute in simulation mode to No.
    4. Click Save.
  7. Locate the ETL in the ETL tasks table and click  to Run it, or schedule an ETL run.
    After you run the ETL, or schedule the ETL for a run, it will extract the data form the source and transfer it to the BMC Helix Continuous Optimization database.

vCloud infrastructure data metrics

The following BMC Helix Continuous Optimization metrics are populated by the VMware - vCloud Director - Cloud infrastructure extractor ETL.

Viewing assistance

To toggle full-screen mode (recommended) for this topic, type f. To hide the navigation tree on the left, type [.

Legend:
C: Configuration metric
T: Time Series metric

Entity

BMC Helix Continuous Optimization metric

Type*DatasetDescriptionvCloud Metric
vCloud











VCLOUD_NAME
C
SYSVMW
Name of the vCloud system-
CPU_ALLOCATION
T
SYSVIRGLB
Number of Units that have been allocated to all orgvDCs in the vCloud.-
CPU_TOTAL_MHZ
C
SYSCNF
Total CPU available for vCloud.-
CPU_UTILMHZ
T
SYSGLB
Sum of CPU used by all child Provider vDCs-
CPU_UTIL
T
SYSGLB
Percentage (%) utilization, ratio of CPU_UTILMHZ / CPU_TOTAL_MHZ.-
MEM_ALLOCATION
T
SYSVIRGLB
Number of units that have been allocated to consumers that is all orgvDCs in the vCloud.-
TOTAL_REAL_MEM
C
SYSCNF
Total memory available for PvDC-
MEM_USED
T
SYSGLB
Sum of memory used by all child Org vDCs.-
MEM_UTIL
T
SYSGLB
Percentage (%) utilization, ratio of MEM_USED/TOTAL_REAL_MEM.-
DSTORE_ALLOCATION
T
SYSVIRGLB
Number of Units that have been allocated to consumers that is all orgvDCs in the vCloud.-
TOTAL_DSTORE_SIZE
C
SYSVIRDS
Total storage available for vCloud.-
TOTAL_DSTORE_USED
T
SYSVIRDS
Storage used-
TOTAL_DSTORE_UTIL
T
SYSVIRDS
Percentage (%) utilization, ratio of TOTAL_DSTORE_USED / TOTAL_DSTORE_SIZE.-
Provider vDC













VCLOUD_NAME
C
SYSVMW
Name of the vCloud system.-
PROVIDERVDC_NAME
C
SYSVMW
Name of the Provider vDC.QueryResultVMWProviderVdcRecordType.name
CPU_ALLOCATION
T
SYSVIRGLB
Number of Units that have been allocated to all orgVDCs under Provider vDC.QueryResultVMWProviderVdcRecordType.cpuAllocationMhz
CPU_TOTAL_MHZ
C
SYSCNF
Total CPU available for PvDC.QueryResultVMWProviderVdcRecordType.cpuLimitMhz
CPU_UTILMHZ
T
SYSGLB
Sum of CPU reserved by all child Org vDCs.QueryResultVMWProviderVdcRecordType.cpuUsedMhz
CPU_UTIL
T
SYSGLB
Percentage (%) utilization, ratio of CPU_UTILMHZ / CPU_TOTAL_MHZ.QueryResultVMWProviderVdcRecordType.cpuUsedMhz /
QueryResultVMWProviderVdcRecordType.cpuLimitMhz) * 100
MEM_ALLOCATION
T
SYSVIRGLB
Number of Units that have been allocated to all orgVDCs under Provider vDC.QueryResultVMWProviderVdcRecordType.memoryAllocationMB
TOTAL_REAL_MEM
C
SYSCNF
Total memory available for PvDC.QueryResultVMWProviderVdcRecordType.memoryLimitMB
MEM_USED
T
SYSGLB
Sum of memory reserved by all child Org vDCs.QueryResultVMWProviderVdcRecordType.memoryUsedMB
MEM_UTIL
T
SYSGLB
Percentage (%) utilization, ratio of MEM_USED/TOTAL_REAL_MEM.(QueryResultVMWProviderVdcRecordType.memoryUsedMB/
QueryResultVMWProviderVdcRecordType.memoryLimitMB) * 100
DSTORE_ALLOCATION
T
SYSVIRGLB
Number of units that have been allocated to consumers that is all orgvDCs under Provider vDC.QueryResultVMWProviderVdcRecordType.storageAllocationMB
TOTAL_DSTORE_SIZE
C
SYSVIRDS
Total storage available for PvDC.QueryResultVMWProviderVdcRecordType.storageLimitMB
TOTAL_DSTORE_USED
T
SYSVIRDS
Storage UsedQueryResultVMWProviderVdcRecordType.storageUsedMB
TOTAL_DSTORE_UTIL
T
SYSVIRDS
Percentage (%) utilization, ratio of TOTAL_DSTORE_USED / TOTAL_DSTORE_SIZE.(QueryResultVMWProviderVdcRecordType.storageUsedMB/
QueryResultVMWProviderVdcRecordType.storageLimitMB) * 100
HA_STATUS
C
SYSVMW
Read-only indicator. True if compute capacity is highly available.VMWProviderVdcType.RootComputeCapacityType.IsHA
VDC_STATUS
C
SYSVMW
Status of vCloud Virtual Datacenter
QueryResultVMWProviderVdcRecordType.isEnabled
Org vDC
















VCLOUD_NAME
C
SYSVMW
Name of the vCloud system-
PROVIDERVDC_NAME
C
SYSVMW
Name of the Provider vDCQueryResultVMWProviderVdcRecordType.name
ORGVDC_NAME
C
SYSVMW
Organization vDC NameQueryResultAdminVdcRecordType .name
CPU_LIMIT_MHZ
C
SYSVIRGLB
Maximum amount of CPU available to the virtual machines running within this organization vDC (taken from the supporting provider vDC).
  • Applicable in case of Allocation Pool.
  • For Pay-As-You-Go this value is always unlimited.
QueryResultAdminVdcRecordType.cpuLimitMhz
CPU_RESERVED_MHZ
C
SYSVIRGLB
Guaranteed Number of Units that have been allocated to org vDC. This is allocation multiplied by the Resource Guaranteed CPU.QueryResultAdminVdcRecordType.cpuAllocationMhz *
AdminVdcType.ResourceGuaranteedCpu
CPU_UTILMHZ
T
SYSGLB
CPU UsedQueryResultAdminVdcRecordType.cpuUsedMhz
CPU_UTIL
T
SYSGLB
Percentage (%) utilization, ratio of CPU_UTILMHZ / CPU_LIMIT_MHZ(QueryResultAdminVdcRecordType.cpuUsedMhz/
QueryResultAdminVdcRecordType.cpuLimitMhz) * 100
MEM_LIMIT
C
SYSVIRGLB
Maximum amount of memory available to the virtual machines running within this organization vDC (taken from the supporting provider vDC).
  • Applicable in case of Allocation Pool.
  • For Pay-As-You-Go this value is always unlimited.
QueryResultAdminVdcRecordType.memoryLimitMB
MEM_RESERVED
C
SYSVIRGLB
Guaranteed Number of Units that have been allocated to org vDC. This is allocation multiplied by the Resource Guaranteed memory.QueryResultAdminVdcRecordType.memoryAllocationMB * AdminVdcType.ResourceGuaranteedMemory
MEM_USED
T
SYSGLB
Memory UsedQueryResultAdminVdcRecordType.memoryUsedMB
MEM_UTIL
T
SYSGLB
Percentage (%) utilization, ratio of MEM_USED/MEM_LIMIT.(QueryResultAdminVdcRecordType.memoryUsedMB / QueryResultAdminVdcRecordType.memoryLimitMB) * 100
DSTORE_LIMIT
C
SYSVIRDS
The maximum amount of Storage available to the virtual machines running within this organization vDC (taken from the supporting provider vDC).QueryResultAdminVdcRecordType.storageLimitMB
TOTAL_DSTORE_SIZE
C
SYSVIRDS
Maximum amount of storage available to the virtual machines running within this organization vDC (taken from the supporting provider vDC).QueryResultAdminVdcRecordType.storageLimitMB
TOTAL_DSTORE_USED
T
SYSVIRDS
Amount of storage used by vAPPs, VMs, templates, media files under this Org vDC.QueryResultAdminVdcRecordType.storageUsedMB
TOTAL_DSTORE_UTIL
T
SYSVIRDS
Percentage (%) utilization, ratio of TOTAL_DSTORE_USED / TOTAL_DSTORE_SIZE(QueryResultAdminVdcRecordType.storageUsedMB / QueryResultAdminVdcRecordType.storageLimitMB) * 100
ALLOCATION_MODEL
C
SYSVMW
Allocation model used by this vDC. One of:
  • AllocationVApp : Pay as you go. Resources are committed to a vDC only when vAPPs are created in it.
  • AllocationPool: Only a percentage of the resources you allocate are committed to the organization vDC.
AdminVdcType.AllocationModel
CPU_COMPUTED_LIMIT_OPT
C
SYSVIRGLB

Maximum optimistic amount of CPU the virtual machines running within this organization vDC can avail.

  • Applicable only for Pay-As-You-Go model.
  • Calculated by summing up the CPU used by Org vDC and the capacity free at PvDC level.
-
MEM_COMPUTED_LIMIT_OPT
C
SYSVIRGLB

Maximum optimistic amount of memory the virtual machines running within this organization vDC can avail.

  • Applicable only for Pay-As-You-Go model.
  • Calculated by summing up the memory used by Org vDC and the capacity free at PvDC level.
-
DSTORE_COMPUTED_LIMIT_OPT
C
SYSVIRDS

Maximum optimistic amount of storage the virtual machines running within this organization vDC can avail.

  • Applicable only for Pay-As-You-Go model.
  • Calculated by summing up the storage used by Org vDC and the capacity free at PvDC level.
-
VDC_STATUS
C
SYSVMW
Status of vCloud Virtual Datacenter
QueryResultAdminVdcRecordType.isEnabled

vCloud Aggregation Hierarchy Rule metrics

The following BMC Helix Continuous Optimization metrics are populated by the vCloud Aggregation Hierarchy Rule 

Note

 For vCloud Capacity Visibility, the aggregation hierarchy rule executes automatically from the hierarchy manager task.

Entity

BMC Helix Continuous Optimization metric 

DatasetDescription
 vCloud 



HOST_NUM
SYSCNF
Number of hosts attached to vCloud.
GM_NUM
SYSVIRGLB
Total number of guest machines.
GM_ON_NUM
SYSVIRGLB
Total number of guest machines which are turned on.
VCPU_NUM
SYSVIRGLB
Total number of virtual CPUs.
VCPU_ON_NUM
SYSVIRGLB
Total number of virtual CPUs for VMs that are turned on.
Provider vDC



HOST_NUM
SYSCNF
Number of hosts attached to Provider vDC.
GM_NUM
SYSVIRGLB
Total number of guest machines.
GM_ON_NUM
SYSVIRGLB
Total number of guest machines which are turned on.
VCPU_NUM
SYSVIRGLB
Total number of virtual CPUs.
VCPU_ON_NUM
SYSVIRGLB
Total number of virtual CPUs for VMs that are turned on.
 Org vDC


GM_NUM
SYSVIRGLB
Total number of guest machines.
GM_ON_NUM
SYSVIRGLB
Total number of guest machines which are turned on.
VCPU_NUM
SYSVIRGLB
Total number of Virtual CPUs.
VCPU_ON_NUM
SYSVIRGLB
Total number of virtual CPUs for VMs that are turned on.

Lookup details

 The vCD defines multiple lookup fields for coordinating with other connectors. The following table lists the sequence of field sets for strong lookup and weak lookup.

Entity type 
Strong lookup fields 
Weak lookup fields 
vCloud
HOSTNAME
NAME

Provider vDC

GUID
NAME
Org vDC
GUID
NAME
Resource Pool
  • GUID
  • PARENT_VCUUID&&VMWVMREF
  • PARENT_VCNAME&&VMWVMREF
PARENT_VCNAME&&NAME
Host
  • GUID
  • HOSTNAME
  • PARENT_VCUUID&&VMWVMREF
  • PARENT_VCNAME&&VMWVMREF
NAME
Datastore
  • GUID
  • PARENT_VCUUID&&VMWVMREF
  • PARENT_VCNAME&&VMWVMREF
NAME

Related topics

Using ETL datasets

Developing custom ETLs

Dataset reference for ETL tasks

Viewing datasets and metrics by dataset and ETL module

Was this page helpful? Yes No Submitting... Thank you

Comments