Page tree
Skip to end of metadata
Go to start of metadata

This topic describes how to configure the continuous polling and collection of data from a VMware environment by using the VMware - vCenter Extractor Service ETL module.

For more information, see the following sections:

The following video (2:35) illustrates the process of setting up a VMware - vCenter Extractor Service ETL module.

 https://youtu.be/VvA10W4GL2U?list=PLibAMtD70sYEneM-OP7FtO7PrMH-Ce0PL

Before you begin

Before configuring TrueSight Capacity Optimization to continuously poll and collect data from VMware environments:

  • Ensure that you have a user account with read-only role to access the vCenter Servers.
  • For environments with firewalls, ensure that you configure the firewall access between VMware vCenter and TrueSight Capacity Optimization ETL Engine server.
  • The ETL Engine server must have access to the URLs of the web services exposed by the vCenter Servers.
  • The ETL Engine uses JRE, which comes embedded with TrueSight Capacity Optimization, and requires special Security considerations for old SSL keys that may have been created with a  key < 1024 bits. For more information, see User credentials for ETL modules.

The hosts, VMs, clusters, resource pools, and datacenter systems used to build performance counters are cached, so that they can be directly used on scheduler/service restart events.

For information on sizing recommendations, see Sizing the vCenter Extractor Service.

Integration steps

To integrate TrueSight Capacity Optimization with the VMware - vCenter Extractor Service, perform the following task:

  1. Navigate to Administration > ETL & SYSTEM TASKS > ETL tasks.

  2. In the ETL tasks page, click Add > Add ETL under the Last run tab.
  3. In the Add ETL page, set values for the following properties under each expandable tab.

    Note

    Basic properties are displayed by default in the Add ETL page. These are the most common properties that you can set for an ETL, and it is acceptable to leave the default selections for each as is.

    Basic properties

    Property Description
    Run configuration
    ETL task name By default, this field is populated based on the selected ETL module. You can specify a different name for the ETL Task. Duplicate names are allowed.
    Run configuration name Default name is already filled out for you. This field is used to differentiate different configurations you can specify for the ETL task. You can then run the ETL task based on it.
    Deploy status You can select Production or Test to mark the ETL tasks. For example, you can start by marking the task as Test and change it to Production after you have seen that you are getting what you wanted.
    Description (Optional) Enter a brief description for this ETL.
    Log level Select how detailed you want the ETL log to be. The log includes Error, Warning and Info type of log information.
    • 1 - Light: Add bare minimum activity logs to the log file. By default, this option is selected.
    • 5 - Medium: Add medium-detailed activity logs to the log file.
    • 10 - Verbose: Add detailed activity logs to the log file. Info,

    Note: Log levels 5 and 10 are typically used for debugging or troubleshooting ETL issues. Using a log level of 5 is general practice, however, you may choose level 10 to get a high level of detail while troubleshooting.

    Execute in simulation mode Select yes if you want to to validate the connectivity between the ETL engine and the target, and to ensure that the ETL does not have any other configuration issues. When set to yes, the ETL will not store actual data into the data warehouse. This option is useful while testing a new ETL task.
    Module selection

    Ensure that the Based on datasource option is selected.

    Note

    If you select Based on Open ETL template, TrueSight Capacity Optimization is integrated with a Generic extractor based on the selected Open ETL template. For more information, see  Generic ETL based on a template.

    ETL module (BMC recommends you to make this selection first) Select VMware - vCenter Extractor Service.
    Module description A link in the user interface that points you to this technical document for the ETL.
    Data type Select any one:
    • metrics at Cluster, Resource Pool, Host, Datastore and Virtual Machine level: Select this option to collect metrics at all available levels. You should use this option when your purpose is to manage the capacity of a VMware environment, including all of the virtual machines in it. By default, this option is selected.
    • metrics at Cluster, Resource Pool, Host, Datastore level: Select this option when you do not want to collect data for individual virtual machines (for example, when your only purpose is to support Cloud Management Capacity Aware Placement Advice). This option saves disk space, I/O load, and computation, and allows you to manage large cloud environments using a modestly sized implementation of TrueSight Capacity Optimization.
    Entity catalog
    Sharing status Select any one:
    • Shared entity catalog: Select this option if, for the same entities, data is coming from multiple sources. For example, BPA ETL.
      • Sharing with Entity Catalog: Select an entity catalog from the drop-down list.
    • Private entity catalog: Select this option if, for the same entity, data is coming from a single source.
    Object relationships
    Associate new entities to

    Specify the domain where you want to add the entities created by the ETL. You can select an existing domain or create a new one.

    Additional Information

    By default, a new domain is created for each ETL, with the same name of the extractor module. As the ETL is created, a new hierarchy rule with the same name of the ETL task is automatically created, with an active status; if you update the ETL specifying a different domain, the hierarchy rule will be updated automatically. The hierarchy rule will be created with status ACTIVE.

    Select any one of the following options:

    • New domain: Create a new domain. By default, this option is selected. Specify the following properties under New domain:
      • Parent: Select a parent domain for your new domain from the domain selector control.
      • Name: Specify a name for your new domain.
    • Existing domain: Select an existing domain. Make a selection for the following property:
      • Domain: Select an existing domain from the domain selector control.
        If the domain is already used by any other hierarchy rule, a Domain conflict option is displayed. You can select one of the following solutions:
        • Enrich domain tree: create a new independent hierarchy rule to add a new set of entities and/or relations not defined by other ETLs (for example this ETL is managing storage while others are managing servers)
        • ETL Migration: this configuration is recommended if new ETLmanagessame set of entities and/or relations (already defined in current domain tree). Typical use case is the migration from one or more ETLs to a new ETL instance. It will stop all relations imported by ETL instances and restore only valid relationsafterfirst run; this configuration reuses existing hierarchy rule to correctly manage relation updates
          • Select a Hierarchy Rule to be used for current ETL from among the rules that are populating the selected domain.
            Additional information of the ETL using the selected rule is also displayed.
    VMware ETL configuration
    VMware server SDK URL Type the URL of the web service exposed by the vCenter Server in the following format:
    https://<hostAddress>/sdk

    where, <hostAddress> is the IP address of the server hosting the vCenter Server.
    Username Type the name of the vCenter user that VMware vCenter Extractor Service should use to access the vCenter Server.
    Password required  If the vCenter Server requires a password, select yes, and type the Password. Otherwise, select no.
    Password Enter the password.
    Use AS time zone Select any one:
    • yes: Select yes to use the AS time zone.
    • no: Select no if you do not want to use the AS time zone.
    Import cluster failover threshold metric

    Select yes, if you want to import the cluster failover threshold metric using the value of "Admission Control Policy > Host failures the cluster tolerates" cluster configuration.

    Select no, if you do not want to import the cluster failover threshold metric.

    ETL task properties
    Task group Select a task group to classify this ETL into. It is not necessary to group it into a task group.
    Running on scheduler Select the scheduler over which you want to run the ETL. The type of schedulers available are:
    • Primary: Runs on the AS. By default, this option is selected.
    • Generic: Runs on a separate machine.
    • Scheduler: Runs on a dedicated scheduler machine.

    Note

    To view or configure Advanced properties, click Advanced. You do not need to set or modify these properties unless you want to change the way the ETL works. These properties are for advanced users and scenarios only.

    Advanced properties

    Property Description
    Run configuration
    Datasets

    Enables you to select or deselect metric groups for which data will be populated from the Available datasets. The OpenStack connector allows you to choose only from the given list of datasets, and you cannot include additional datasets to the run configuration of the ETL.

    1. Click Edit.
    2. Select one (click) or more (ctrl+click or shift+click) datasets that you want to include from the Available datasets and click >> to move them to Selected datasets.
    3. Click Apply.
    4.x compatibility  Select any one:
    • Do not Import Virtual Machine metrics at Host level, but only at VM level. By default, this option is selected.
    • Import Virtual Machine metrics at Host level [BYGM, BYZN, BYLP] to ensure compatibility of 4.x reports.
    Data type

    Select any one:

    • Metrics at Cluster, Resource Pool, Host, Datastore and Virtual Machine level.
    • Metrics at Cluster, Resource Pool, Host, and Datastore level.
    Saver period Select any one:
    • 1h (By default, this option is selected.)
    • 2h
    Collection level
    Metric profile selection

    Select any one:

    • Use Global metric profile: Select this to use an out-of-the-box global profile, that is available on the Adding and modifying metric profiles page. By default, all ETL modules use this profile.
    • Select a custom metric profile: Any metric profiles you add in the Add metric profile page (Administration > DATAWAREHOUSE > Metric profiles >

    For more information, see Adding and modifying metric profiles.

    Levels up to

    The metric level defines the amount of metric imported into the data warehouse. If you increase the level, additional load is added to the data warehouse while decreasing the metric level reduces the number of imported metrics.

    Choose the metric level to apply on selected metrics:

    • [1] Essential
    • [2] Basic
    • [3] Standard
    • [4] Extended

    For more information, see Aging Class mapping.

    Metric filter

    Metric list <for selected dataset>

    Click Edit and select the metrics that will be loaded for each dataset that you selected under Run configuration > Datasets. If no metric is specified, all metrics will be loaded.

    VMware ETL configuration
    Aggregation period BMC recommends that you leave this at the default value of 1 hour. You can select or define the amount of time between 5 minutes30 minutes, 15 minutes, or 1 hour of data aggregation.
    Pre-aggregated statistics import Select any one:
    • yes: Select yes to import pre-aggregated statistics such as min, max, count for metrics, so that you import data up to hourly granularity.
    • no: Select no if you do not was to import pre-aggregated statistics.
    Hierarchy import period Specify the time interval to sync the object relationship by selecting 6h, 12h, or 24h. By default, 6h is selected.
    Cluster/Pool performance data extraction Select any one of the following aggregation methods to be used for metric aggregation:
    • Aggregate Cluster/Pool performance data starting from Host/VM data (recommended)
    • Extract Cluster/Pool performance data from vCenter
    Compatibility lookup names customization Select any one of the following options that you want to use for sharing the lookup between ETLs:
    • Default/Recommended: UUID for hosts and Virtual Machines.
    • System names for hosts and Virtual Machines.
    • Host name for hosts and Virtual Machines.
    Use Virtual Machine network name as system name Select any one:
    • yes: Select yes to use virtual machine network name as system name.
    • no: Select no, if you do not want to use the virtual machine network name as system name. By default, this option is selected.
    Import cluster failover threshold metric To import the CLUSTER_FAILOVER_THRESHOLD metric, select Yes, using value of "Admission Control Policy > Host failures the cluster tolerates" cluster configuration.
    By default, No is selected.
    Custom detail import

    Select any one:

    • Select yes if you want to import a custom detailed metric set. Specify the following details:
      • Detail resolution: Set the resolution to 5 minutes or 15 minutes. By default, 15 minutes is selected.
      • Detailed metrics for entity type: Select if you want detailed metrics for All supported types or Select entity type.
        • Entity types: Select the entity types such as Host/VM, Pool, and Cluster.
      • Detailed metric set: Select Default or Custom.
        • If you select Custom, the Custom detailed metric set is displayed.

          Click (plus)Add to add metrics to the default metric set.

          Click (minus) to remove individual metrics from the list.

          The default metric set includes the following metics:

          • CPU_UTIL

          • CPU_UTILMHZ

          • CPU_READY

          • MEM_UTIL

          • MEM_ACTIVE

          • MEM_BALLOONING

          • TOTAL_FS_USED

            In addition, you can add BYFS_SIZE, BYFS_FREE, BYFS_USED and BYFS_USED_SPACE_PCT metrics and the minimum Detail resolution for these metrics is 15 minutes.

            Detailed resolution of BYFS metrics is supported only when VMware Tools are installed on the Virtual Machine from which the data is to be imported.

    • Select no, if you do not want to use custom detailed metrics. By default, this option is selected.
    Import filter

    Configure one or both the filtering properties in this section to include or exclude entities while importing.

    Filtering for clusters and top-level hosts

    Specify whether you want to import all or specific clusters and standalone hosts.

    • The default selection is None, which indicates that all clusters and standalone hosts are imported.
    • To import only the specific clusters and standalone hosts, select Whitelist, and specify the names of clusters and standalone hosts to be imported. For example, cl1;cluster24 and host1;esx_host
    • To exclude the specific clusters and standalone hosts while importing, select Blacklist, and specify the names of clusters and standalone hosts to be imported. For example, cl1;cluster45 and host2;esx_host
    Blacklist file path

    Use this property to exclude or include specific entities. While importing entities, you can exclude (blacklist) specific hosts and virtual machines or can include (whitelist) specific virtual machines.

      To exclude specific hosts and virtual machines (blacklist)
    1. In a text file, add the names of virtual machines and hosts that you want to exclude from importing in the following format, and save the file on the ETL Engine Server where the ETL runs:
      SYSTEM_TYPE;ENTITY UUID

      Tip

      To find UUIDs, select the required virtual machine or host in the Workspace, and click View lookup. The Lookup value column in the Lookup Details table shows the UUID values.

      The use of regular expressions is not supported.

    2. In the Use file at path box, specify the path to this text file.
    3. After you run the ETL, verify that the the specified virtual machines and hosts are not displayed in the hierarchy.

    Example:

    To exclude specific hosts and virtual machines from importing, obtain their UUIDs and add them to a text file as follows:

    vh:vmw;44454c4c-4600-1054-8052-cac04f525231
    vh:vmw;44454c4c-4600-1054-8052-cac04f525232
    gm:vmw;4208badb-6a91-23d1-c6b5-061745b2c8d9
    gm:vmw;4208badb-6a91-23d1-c6b5-061745b2c8d7

    Where, vh:vmw is the system type and 44454c4c-4600-1054-8052-cac04f525231 is the UUID of host_1. Similarly, gm:vmw is the system type and 4208badb-6a91-23d1-c6b5-061745b2c8d9 is the UUID of vm_1, and so on.

    After the ETL runs, these specified virtual machines and hosts are not imported.

      To include specific virtual machines (whitelist)
    1. Obtain the DNS names of virtual machines to be imported from the vCenter.
    2. In a text file, add the virtual machine names, and save the file on the ETL Engine Server where the ETL runs:
      1. On the first line, add the following declaration statement: #FORMAT=V2;TYPE=VM_WHITELIST
      2. On the subsequent lines, add the virtual machine names in one of the following ways:
        • Add the DNS names.
        • Add a regular expression. The DNS names of virtual machines that match the pattern in the regular expression are included while importing.
    3. In the Use file at path box, specify the path to this text file.
    4. After you run the ETL, verify that only the specified virtual machines are imported in the hierarchy.

    This filter is applied only on the virtual machines. It does not affect importing of other entities, such as virtual hosts, clusters, and datastores.

    Example 1 (Specifying DNS names)

    To import virtual machines vm_1, vm_2, and vm_3, add their DNS names to the whitelist file as follows:

    #FORMAT=V2;TYPE=VM_WHITELIST
    vm_1.orgname.com
    vm_2.orgname.com
    vm_3.orgname.com

    After you run the ETL, these virtual machines along with other entities are imported.

    Example 2 (Using a regular expression)

    To import the virtual machines for which the name starts with vl, use the following regular expression:

    #FORMAT=V2;TYPE=VM_WHITELIST
    ^vl\-[a-z]*\-[0-9]*\.orgname\.com

    After you run the ETL, all the virtual machines for which the DNS name starts with vl will be imported. For example, vl-pun-023.orgname.com and vl-pun-024.orgname.com

    Tip

    To import virtual machine names in any case, use this regular expression: ^(?i)vl\-[a-z]*\-[0-9]*\.orgname\.com

    Additional properties
    List of properties
    1. Click Add.
    2. Add an additional property in the etl.additional.prop.n box.
    3. Click Apply.
      Repeat this task to add more properties.
    Loader configuration
    Empty dataset behavior Choose one of the following actions if the loader encounters an empty dataset:
    • Warn: Warns about an empty dataset.
    • Ignore: Ignore the empty dataset and continue parsing.
    ETL log file name Name of the file that contains the ETL execution log; the default value is: %BASE/log/%AYEAR%AMONTH%ADAY%AHOUR%MINUTE%TASKID.
    Maximum number of rows for CSV output A number which limits the size of the output files.
    CSV loader output file name Name of the file generated by the CSV loader; the default value is: %BASE/output/%DSNAME%AYEAR%AMONTH%ADAY%AHOUR%ZPROG%DSID%TASKID.
    Capacity Optimization loader output file name Name of the file generated by the TrueSight Capacity Optimization loader; the default value is: %BASE/output/%DSNAME%AYEAR%AMONTH%ADAY%AHOUR%ZPROG%DSID%TASKID.
    Detail mode Select the level of detail:
    • Standard: Data will be stored on the database in different tables at the following time granularities: Detail (configurable, by default: 5 minutes), Hourly, Daily, Monthly.
    • Raw also: Data will be stored on the database in different tables at the following time granularities: Raw (as available from the original data source), Detail (configurable, by default: 5 minutes), Hourly, Daily, Monthly.
    • Raw only: Data will be stored on the database in a table only at Raw granularity (as available from the original data source).
    Reduce priority
    • Normal:
    • High:
    Remove domain suffix from datasource name (Only for systems) If set to True, the domain name is removed from the data source name. For example, server.domain.com will be saved as server.
    Leave domain suffix to system name (Only for systems) If set to True, the domain name is maintained in the system name. For example: server.domain.com will be saved as such.
    Update grouping object definition If set to True, the ETL will be allowed to update the grouping object definition for a metric loaded by an ETL.
    Skip entity creation (Only for ETL tasks sharing lookup with other tasks) If set to True, this ETL does notcreateanentity, and discards data from its data source for entities not found in TrueSight Capacity Optimization. It uses one of the other ETLs that share lookup to create the new entity.
  4. Click Save.

    You return to the Last run tab under the ETL tasks page.

  5. Validate the results in simulation mode: In the ETL tasks table under ETL tasks > Last run, locate your ETL (ETL task name), click  to run the ETL.

    After you run the ETL, the Last exit column in the ETL tasks table will display one of the following values:

    • OK: The ETL executed without any error in simulation mode.
    • WARNING: The ETL execution returned some warnings in simulation mode. Check the ETL log.
    • ERROR: The ETL execution returned errors and was unsuccessful. Edit the active Run configuration and try again.
  6. Switch the ETL to production mode: To do this, perform the following task:
    1. In the ETL tasks table under ETL tasks > Last run, click the ETL under the Name column.
    2. In the Run configurations table in the ETL details page, click  to edit the active run configuration.
    3. In the Edit run configuration page, navigate to the Run configuration expandable tab and set Execute in simulation mode to No.
    4. Click Save.
  7. Locate the ETL in the ETL tasks table and click  to Run it, or schedule an ETL run.

    After you run the ETL, or schedule the ETL for a run, it will extract the data form the source and transfer it to the TrueSight Capacity Optimization databases.

2 Comments

  1.  

    1.