VMware - vCloud Services extractor

This topic describes how to integrate BMC Helix Continuous Optimization with VMWare vCloud Director Services ETL to extract vCloud services data. 

Note

To collect data from 9.x and 10.x versions of VMware Cloud Director, use the VMware - Cloud Director 9.x/10.x Extractor. This ETL collects vCloud infrastructure and services data from the vCloud Director.

For more information, refer to the following sections:

Before you begin

Ensure that you have created a VMware - vCloud Infrastructure extractor ETL. To integrate VMware - vCloud Services extractor ETL with VMware - vCloud Infrastructure ETL, set the Entity catalog to SHARED.

Integration steps

To integrate BMC Helix Continuous Optimization with the VMware - vCloud services extractor:

  1. Navigate to Administration > ETL & SYSTEM TASKS > ETL tasks.

  2. In the ETL tasks page, click Add > Add ETL under the Last run tab.

  3. In the Add ETL page, set values for the following properties under each expandable tab.

    Note

    Basic properties are displayed by default in the Add ETL page. These are the most common properties that you can set for an ETL, and it is acceptable to leave the default selections for each as is.

    Basic properties

    Property Description
    Run configuration
    ETL module Select VMware - vCloud Services extractor.
    ETL task name Default name is already filled out for you.
    Run configuration name Default name is already filled out for you.
    Deploy status Select Production.
    Description (Optional) Enter a brief description.
    Log level Select how detailed you want the log to be:
    • 1 - Light: Add bare minimum activity logs to the log file.
    • 5 - Medium: Add medium-detailed activity logs to the log file.
    • 10 - Verbose: Add detailed activity logs to the log file.
    Execute in simulation mode Select Yes if you want to to validate the connectivity between the ETL engine and the target, and to ensure that the ETL does not have any other configuration issues. This option is useful while testing a new ETL task.
    Module description A link that points you to technical documentation for this ETL.
    Entity catalog
    Sharing status Select any one:
    • PRIVATE: Select this option if this is the only ETL that extracts data from the given set of resources and the lookup table is not shared with the specified ETL task.
    • SHARED: Select this option if more than one ETL extracts data from the given set of resources and the lookup table is shared with the specified ETL task.

    For integrating VMware - vCloud Services extractor ETL with VMware - vCloud Infrastructure ETL, set the Entity catalog to SHARED.

    Object relationships
    Associate new entities to

    Specify the domain to which you want to add the entities created by the ETL.

    Select one of the following options:

      • New domain: This option is selected by default. Select a parent domain, and specify a name for your new domain.
      • Existing domain: Select an existing domain from the Domain list.

    By default, a new domain with the same ETL name is created for each ETL. 

    vCloud Director ETL configuration
    vCloud Director server URL

    Enter the vCloud Director server URL.

    Note

    Ensure that you type the details for the server as provided for the vCloud Infrastructure ETL task.

    Username

    Log in as vCloud Director Administrator user.

    Password required Select either Yes or No.
    Password Enter the password to access all functionalities.
    ETL task properties
    Task group Select a task group to classify this ETL into.
    Running on scheduler Select a scheduler for running the ETL. For cloud ETLs, use the scheduler that is preconfigured in Helix. For on-premises ETLs, use the scheduler that runs on the Remote ETL Engine.
    Maximum execution time before warning The number of hours, minutes or days to to execute the ETL for before generating warnings, if any.
    Frequency Select the frequency of ETL execution. Available options are:
    • Predefined: Select a Predefined frequency from Each Day, Each Week or Each Month.
    • Custom: Enter a Custom frequency (time interval) as the number of minutes, hours, days or weeks to run the ETL in.
    Start timestamp: hour\minute (Applies to Predefined frequency) The HH:MM start timestamp to add to the ETL execution running on a Predefined frequency.
    Custom start timestamp Select a YYYY-MM-DD HH:MM timestamp to add to the ETL execution running on a Custom frequency.

    Note

    To view or configure Advanced properties, click Advanced. You do not need to set or modify these properties unless you want to change the way the ETL works. These properties are for advanced users and scenarios only.

    Advanced properties

    Property Description
    Run configuration
    Datasets
    1. Click Edit.
    2. Select one (click) or more (shift+click) datasets that you want to include from Available datasets and click >> to move them to Selected datasets.
    3. Click Apply.
    Collection level 
    Metric profile selection

    Select any one:

    • Use Global metric profile: Select this to use an out-of-the-box global profile, that is available on the Adding and managing metric profiles page. By default, all ETL modules use this profile.
    • Select a custom metric profile: Any metric profiles you add in the Add metric profile page (Administration > DATAWAREHOUSE > Metric profiles).

    For more information, see Adding and managing metric profiles.

    Levels up to

    The metric level defines the amount of metric imported into Continuous Optimization. If you increase the level, additional load is added to the ingestion, while decreasing the metric level reduces the number of imported metrics.

    Choose the metric level to apply on selected metrics:

    • [1] Essential
    • [2] Basic
    • [3] Standard
    • [4] Extended
    Metric filter
    Metric list <for selected dataset>

    Note: This property is activated only after you select Datasets under the Run configuration tab.
    1. Click Edit.
    2. Select one (click) or more (shift+click) metrics that you want to include from Available items and click >> to move them to Selected items.
    3. Click Apply.
    vCloud Director ETL configuration
    vCloud Director server URL

    Enter the vCloud Director server URL.

    Note

    Ensure that you type the details for the server as provided for the vCloud Infrastructure ETL task.

    Extraction parallelism Default value is filled out for you.
    Additional properties
    List of properties
    1. Click Add.
    2. Add an additional property in the etl.additional.prop.n box.
    3. Click Apply.
      Repeat this task to add more properties.
    Loader configuration
    Empty dataset behavior Choose one of the following actions if the loader encounters an empty dataset:
    • Abort: Abort the loader.
    • Ignore: Ignore the empty dataset and continue parsing.
    ETL log file name Name of the file that contains the ETL execution log; the default value is: %BASE/log/%AYEAR%AMONTH%ADAY%AHOUR%MINUTE%TASKID
    Maximum number of rows for CSV output A number which limits the size of the output files.
    CSV loader output file name Name of the file generated by the CSV loader; the default value is: $CPITBASE/output/%AYEAR%AMONTH%ADAY%AHOUR%ZPROG%DSCD%SRCID.

    Continuous Optimization loader output file name

    Name of the file generated by the Continuous Optimization loader; the default value is: $CPITBASE/output/%AYEAR%AMONTH%ADAY%AHOUR%ZPROG%DSCD%SRCID.

    Remove domain suffix from datasource name (Only for systems) If set to True, the domain name is removed from the data source name. For example, server.domain.com will be saved as server.
    Leave domain suffix to system name (Only for systems) If set to True, the domain name is maintained in the system name. For example: server.domain.com will be saved as such.
    Skip entity creation

    (Only for ETL tasks sharing lookup with other tasks) If set to True, this ETL does not create an entity, and discards data from its data source for entities not found in BMC Helix Continuous Optimization. It uses one of the other ETLs that share lookup to create the new entity.

    Scheduling options
    Hour mask Specify a value to execute the task only during particular hours within the day. For example, 0 – 23 or 1,3,5 – 12.
    Day of week mask Select the days so that the task can be executed only during the selected days of the week. To avoid setting this filter, do not select any option for this field.
    Day of month mask Specify a value to execute the task only during particular days within a month. For example, 5, 9, 18, 27 – 31.
    Apply mask validation By default this property is set to True. Set it to False if you want to disable the preceding Scheduling options that you specified. Setting it to False is useful if you want to temporarily turn off the mask validation without removing any values.
    Execute after time Specify a value in the hours:minutes format (for example, 05:00 or 16:00) to wait before the task must be executed. This means that once the task is scheduled, the task execution starts only after the specified time passes.
    Enqueueable Select one of the following options:
    • False (Default): While a particular task is already running, if the next execution command arises – it is ignored.
    • True: While a particular task is already running, if the next execution command arises – it is placed in a queue and is executed as soon as the current execution ends.
  4. Click Save.
    You return to the Last run tab under the ETL tasks page.
  5. Validate the results in simulation mode: In the ETL tasks table under ETL tasks > Last run, locate your ETL (ETL task name), click  to run the ETL.
    After you run the ETL, the Last exit column in the ETL tasks table will display one of the following values:
    • OK: The ETL executed without any error in simulation mode.
    • WARNING: The ETL execution returned some warnings in simulation mode. Check the ETL log.
    • ERROR: The ETL execution returned errors and was unsuccessful. Edit the active Run configuration and try again.
  6. Switch the ETL to production mode: To do this, perform the following task:
    1. In the ETL tasks table under ETL tasks > Last run, click the ETL under the Name column.
    2. In the Run configurations table in the ETL details page, click  to edit the active run configuration.
    3. In the Edit run configuration page, navigate to the Run configuration expandable tab and set Execute in simulation mode to No.
    4. Click Save.
  7. Locate the ETL in the ETL tasks table and click  to Run it, or schedule an ETL run.
    After you run the ETL, or schedule the ETL for a run, it will extract the data form the source and transfer it to the BMC Helix Continuous Optimization database.

Verifying the integration

After you run the ETL task, you can verify if the integration was successful.

  1. Navigate to the Managing and analyzing data from your workspace and select the branch with the name specified in the Object relationships tab while configuring the ETL.
  2. Verify that the hierarchy is populated and data is displayed.

Lookup details

The vCD defines multiple lookup fields for coordinating with other connectors. The following table lists the sequence of field sets for strong lookup and weak lookup.

Entity Type 
Strong lookup fields 
Weak lookup fields 

Organization

GUID
NAME
Org VDC
GUID
NAME
VAPP
GUID
NAME
VM
  • GUID
  • PARENT_VCUUID&&VMWVMREF
  • PARENT_VCNAME&&VMWVMREF
NAME

Note

In vCloud Director Service ETL, the following entities are retrieved:

  • Organizations: Organization vDCs.
  • Organization vDC: vApps.
  • vApp: VMs.

Related topics

VMware - vCloud Infrastructure extractor

Working with ETLs

Using ETL datasets

Developing custom ETLs

Dataset reference for ETL tasks

Horizontal and Vertical datasets

Viewing datasets and metrics by dataset and ETL module

Was this page helpful? Yes No Submitting... Thank you

Comments