Skip to end of metadata
Go to start of metadata

This topic describes how to configure and use the BMC - TrueSight Capacity Optimization Gateway VIS files parser. This ETL loads daily performance and configuration data from a running Capacity Optimization Gateway Server instance. It can also be used for loading recovery data from a running Capacity Optimization Gateway VIS files parser server instance.

Note

BMC - TrueSight Capacity Optimization Gateway VIS files parser(s) can be automatically created from Gateway Manager. Select the Auto create ETLs to populate data from this gateway server option when you add a Gateway Server. For more information, see Gateway Servers.

This topic contains the following sections:

To integrate BMC TrueSight Capacity Optimization using VIS files parser

Note

If your imported data includes workloads, you can optionally activate the hierarchy rule to create aggregated workloads. For more information, see Activating the CO Aggregated Workloads hierarchy in BMC TrueSight Capacity Optimization.

Integration steps

To integrate BMC TrueSight Capacity Optimization with the Gateway Server File Parser, perform the following task:

  1. Navigate to Administration > ETL & SYSTEM TASKS > ETL tasks.

  2. In the ETL tasks page, click Add > Add ETL under the Last run tab.

  3. In the Add ETL page, set values for the following properties under each expandable tab. The page contains the following tabs:

      Note

      Basic properties are displayed by default in the Add ETL page. These are the most common properties that you can set for an ETL, and it is acceptable to leave the default selections for each as is.

      Property Value
      Run configuration
      ETL task name By default, this field is populated based on the selected ETL module. You can specify a different name for the ETL Task. Duplicate names are allowed.
      Run configuration name Default name is already filled out for you. This field is used to differentiate different configurations you can specify for the ETL task. You can then run the ETL task based on it.
      Environment You can select Production or Test to mark the ETL tasks. For example, you can start by marking the task as Test and change it to Production after you have seen that you are getting what you wanted.
      Description (Optional) Enter a brief description for this ETL.
      Log level Select how detailed you want the ETL log to be. The log includes Error, Warning and Info type of log information.
      • 1 - Light: Add bare minimum activity logs to the log file.
      • 5 - Medium: Add medium-detailed activity logs to the log file.
      • 10 - Verbose: Add detailed activity logs to the log file. Info,

      Note: Log levels 5 and 10 are typically used for debugging or troubleshooting ETL issues. Using a log level of 5 is general practice, however, you may choose level 10 to get a high level of detail while troubleshooting.

      Execute in simulation mode Select Yes if you want to to validate the connectivity between the ETL engine and the target, and to ensure that the ETL does not have any other configuration issues.
      When set to Yes, the ETL will not store actual data into the data warehouse. This option is useful while testing a new ETL task.
      ETL module (BMC recommends you to make this selection first) Select BMC - TrueSight Capacity Optimization Gateway VIS files parser.
      Module description A link in the user interface that points you to technical documentation for this ETL.
      Platform Select any one:
      • Standalone
      • VMware
      • AIX PowerVM/WPAR
      • Solaris LDOM/Zone/DSD
      • HP IVM/nPar/vPar
      • Xen
      • MS Hyper-V
      • KVM
      • Virtual Node
      Data type

      Performance data is selected by default. You cannot change this setting. Based on the Platform selection, you can select the data type options.
      For Xen and MS HyperV platforms, performance data is the only available option.
      For rest of the platforms, select from the following:

      • detailed disk/network data
      • workload data
      • command statistics
      • user statistics
      Import HMC data This option is only available if you select AIX Power VM/WPAR platform. By default, yes is selected. If you do not want to import HMC data, select no.
      Entity catalog
      Sharing status Select any one:
      • Shared entity catalog: Select this option if, for the same entities, data is coming from multiple sources. For example, BPA ETL.
        • Sharing with Entity Catalog: Select an entity catalog from the drop-down list.
      • Private entity catalog: Select this option if, for the same entity, data is coming from a single source.
      Object relationships (displayed if you select a platform)
      Associate new entities to

      Specify the domain where you want to add the entities created by the ETL. You can select an existing domain or create a new one.

      Additional Information

      By default, a new domain is created for each ETL, with the same name of the extractor module. As the ETL is created, a new hierarchy rule with the same name of the ETL task is automatically created, with an active status; if you update the ETL specifying a different domain, the hierarchy rule will be updated automatically. The hierarchy rule will be created with status ACTIVE.

      Select any one of the following options:

      • New domain: Create a new domain. Specify the following properties under New domain:
        • Parent: Select a parent domain for your new domain from the domain selector control.
        • Name: Specify a name for your new domain.
      • Existing domain: Select an existing domain. Make a selection for the following property:
        • Domain: Select an existing domain from the domain selector control.

      If the domain is already used by any other hierarchy rule, a Domain conflict option is displayed. You can select one of the following solutions:


        • Enrich domain tree: Create a new independent hierarchy rule to add a new set of entities and/or relations not defined by other ETLs (for example this ETL is managing storage while others are managing servers)
        • ETL Migration: This configuration is recommended if new ETL manages same set of entities and/or relations (already defined in current domain tree). Typical use case is the migration from one or more ETLs to a new ETL instance. It will stop all relations imported by ETL instances and restore only valid relations after first run; this configuration reuses existing hierarchy rule to correctly manage relation updates
          • Select a Hierarchy Rule to be used for current ETL from among the rules that are populating the selected domain.
            Additional information of the ETL using the selected rule is also displayed.

      Vis parser configuration
      Extractor mode Select any one:
      • via gateway service: Extract the Vis parser through gateway service.
        • Gateway Server: Select any of the Gateway Server server from the drop-down list.
        • Manger run list : Specify the manager run. You can separate the manager run by ";". For example, vcenter.msf; Test.msf; vcenter5.1.msf; Test1.msf.
      • via file
      File location (applicable only for via file option)
       
      File location Select any one of the following methods to retrieve the CSV file:
      • Local directory: Specify a path on your local machine where the CSV file resides.
      • Windows share: Specify the Windows share path where the CSV file resides.
      • FTP: Specify the FTP path where the CSV file resides.
      • SCP: Specify the SCP path where the CSV file resides.
      • SFTP: Specify the SFTP path where the CSV file resides.
      Directory Path of the directory that contains the CSV file.
      File list pattern A regular expression that defines which data files should be read. The default value is (?$<$!done)\$, which tells the ETL to read every file whose name does not end with the string "done". For example, my_file_source.done.
      Recurse into subdirs? Select Yes or No. When set to Yes, BMC TrueSight Capacity Optimization also inspects the subdirectories of the target directories.
      After parse operation Choose what to do after the CSV file has been imported. The available options are:
      • Do nothing: Do nothing after import.
      • Append suffix to parsed file: Append a suffix you add here to the imported CSV file. For example, _done or _impoted, and so on.
      • Archive parsed file in directory: Archive the parsed file in the specified directory.
        • Archive directory (local): Default archive directory path is filled out for you. For example, %BASE/../repository/imprepository
        • Compress archived files: Select Yes or No.
      • Archive bad files in directory: Archive erroneous files in the specified directory.
        • Archive directory (local): Default archive directory path is filled out for you. For example, %BASE/../repository/imprepository
        • Compress archived files: Select Yes or No.

      Note

      You can configure automatic cleaning of parsed files using the Filesystem cleaner task. For more information, see FileSystem Cleaner task.

      Parsed files suffix The suffix that will be appended parsed files; default is .done.
      Remote host (Applies to FTP, SFTP, SCP) Enter the name or address of the remote host to connect to.
      Username (Applies to Windows share, FTP, SFTP, SCP) Enter the username to connect to the file location server.
      Password required (Applies to Windows share, FTP, SFTP, SCP) Select Yes or No.
      Password (Applies to Windows share, FTP, SFTP, SCP) Enter a password to connect to the file location server. Applicable if you selected Yes for Password required.
      ETL task properties
      Task group Select a task group to classify this ETL into. it is not necessary to group it into a task group.
      Running on scheduler Select the scheduler you want to run the ETL on.
      Maximum execution time before warning The number of hours, minutes or days to execute the ETL for before generating warnings or alerts, if any.
      Frequency Select the frequency for ETL execution. Available options are:
      • Predefined: Select a Predefined frequency from Each DayEach Week orEach Month.
      • Custom: Enter a Custom frequency (time interval) as the number ofminutes, hours, days or weeks to run the ETL in.
      Start timestamp: hour\minute (Applies to Predefined frequency) The HH:MM start timestamp to add to the ETL execution running on a Predefined frequency.
      Custom start timestamp Select a YYYY-MM-DD HH:MM timestamp to add to the ETL execution running on a Custom frequency.

      Note

      To view or configure Advanced properties, click Advanced. You do not need to set or modify these properties unless you want to change the way the ETL works. These properties are for advanced users and scenarios only.

      Property Value
      Run configuration
      Datasets

      Enables you to select or deselect metric groups for which data will be populatedAvailable datasets. The OpenStack connector allows you to choose only from the given list of datasets, and you cannot include additional datasets to the run configuration of the ETL.

      1. Click Edit.
      2. Select one (click) or more (shift+click) datasets that you want to exclude fromAvailable datasets and click >> to move them to Selected datasets.
      3. Click Apply.
      4.x compatibility Select any one:
      • Do not Import Virtual Machine metrics at Host level, but only at VM level 
      • Import Virtual Machine metrics at Host level [BYGM, BYZN, BYLP] to ensure compatibility of 4.x reports.
      Enable platform filtering Select any one:
      • Yes: Select Yes to enable the platform filtering.
      • No: Select No to disable the platform filtering
      Default last counter Set the default last counter in dd/mm/yyy hh:mm:ss format.
      Use enterprise grouping Select any one of the following options:
      • Yes: Select any one of the following options:
        • Populate enterprise grouping: Populate the selected grouping object. For more information, see Grouping objects.
        • Use subobject for enterprise grouping: Use subObject to populate the selected grouping object.
      • No: Do not populate any enterprise grouping.
      Metric filter
      Metric list <for selected dataset>

      Click Edit and select the metrics that will be loaded for each dataset that you selected under Run configuration > Datasets. If no metric is specified, all metrics will be loaded.

      Vis file parser configuration
      Lookup names customization

      Select any one:

      • Default/Recommended: UUID for hosts and Virtual Machines
      • System names for hosts and Virtual Machines
      Use network name as system name for VMware virtual machines Select either False or True.
      Data recovery mode Select any one of the following:
      • True
        • Delete recovered files (only for local files): Select True or False.
      • False
      Extractor mode Select any one:

        • Select any one:
          • via gateway service: Extract the Vis parser through gateway service.
            • Gateway Server: Select any of the Gateway Server server from the drop-down list.
            • Manger run list : Specify the manager run. You can separate the manager run by ";". For example, vcenter.msf; Test.msf; vcenter5.1.msf; Test1.msf.
            • Wait configured manager run completion: Select either Yes orNo.
            • Archive parsed file in directory: Specifies the archive parsed file in the directory.
            • Compress archived files
              • Yes: Select Yes to compress the archived files.
              • No: Do not compress the archived files.
          • via file
            • Manager run list: Click Add to add the manager list.
            • Disable node check: Select Yes or No.
      File location (applicable only for via file option)
      Subdirectories to exclude (separated by ';' ) (Local directory) Names of subdirectories to exclude from parsing.
      Input file external validator (Local directory, Windows share, FTP) Select any one of the following options:
      • No external validation: Do not use external validation of the CSV file structure.
      • Use external validation script: Use the following script to validate the CSV file:
        • Script to execute: Specify the validation script to use to validate the input file.
      Additional properties
      List of properties
      1. Click Add.
      2. Add an additional property in the etl.additional.prop.n box.
      3. Click Apply.
        Repeat this task to add more properties.
      Loader configuration
      Empty dataset behavior Choose one of the following actions if the loader encounters an empty dataset:
      • Abort: Abort the loader.
      • Ignore: Ignore the empty dataset and continue parsing.
      ETL log file name Name of the file that contains the ETL execution log; the default value is: %BASE/log/%AYEAR%AMONTH%ADAY%AHOUR%MINUTE%TASKID
      Maximum number of rows for CSV output A number which limits the size of the output files.
      CSV loader output file name Name of the file generated by the CSV loader; the default value is: %BASE/output/%DSNAME%AYEAR%AMONTH%ADAY%AHOUR%ZPROG%DSID%TASKID.
      Capacity Optimization loader output file name Name of the file generated by the BMC TrueSight Capacity Optimization loader; the default value is: %BASE/output/%DSNAME%AYEAR%AMONTH%ADAY%AHOUR%ZPROG%DSID%TASKID.
      Detail mode Select the level of detail:
      • Standard: Data will be stored on the database in different tables at the following time granularities: Detail (configurable, by default: 5 minutes), Hourly, Daily, Monthly.
      • Raw also: Data will be stored on the database in different tables at the following time granularities: Raw (as available from the original data source), Detail (configurable, by default: 5 minutes), Hourly, Daily, Monthly.
      • Raw only: Data will be stored on the database in a table only at Raw granularity (as available from the original data source).

      For more information on granularities, see Accessing data using public views and Sizing and scalability considerations.

      Reduce priority
      • Normal:
      • High:
      Remove domain suffix from datasource name (Only for systems) If set to True, the domain name is removed from the data source name. For example, server.domain.com will be saved as server.
      Leave domain suffix to system name (Only for systems) If set to True, the domain name is maintained in the system name. For example: server.domain.com will be saved as such.
      Update grouping object definition If set to True, the ETL will be allowed to update the grouping object definition for a metric loaded by an ETL.
      Skip entity creation (Only for ETL tasks sharing lookup with other tasks) If set to True, this ETL does not create an entity, and discards data from its data source for entities not found in BMC TrueSight Capacity Optimization. It uses one of the other ETLs that share lookup to create the new entity.
      Scheduling options
      Hour mask Specify a value to execute the task only during particular hours within the day. For example, 0 – 23 or 1,3,5 – 12.
      Day of week mask Select the days so that the task can be executed only during the selected days of the week. To avoid setting this filter, do not select any option for this field.
      Day of month mask Specify a value to execute the task only during particular days within a month. For example, 5, 9, 18, 27 – 31.
      Apply mask validation By default this property is set to True. Set it to False if you want to disable the preceding Scheduling options that you specified. Setting it to False is useful if you want to temporarily turn off the mask validation without removing any values.
      Execute after time Specify a value in the hours:minutes format (for example, 05:00 or 16:00) to wait before the task must be executed. This means that once the task is scheduled, the task execution starts only after the specified time passes.
      Enqueueable Select one of the following options:
      • False (Default): While a particular task is already running, if the next execution command arises – it is ignored.
      • True: While a particular task is already running, if the next execution command arises – it is placed in a queue and is executed as soon as the current execution ends.
    • Click Save.
      You return to the Last run tab under the ETL tasks page.

    • Validate the results in simulation mode: In the ETL tasks table under ETL tasks > Last run, locate your ETL (ETL task name), click  to run the ETL.
      After you run the ETL, the Last exit column in the ETL tasks table will display one of the following values:

    • Switch the ETL to production mode: To do this, perform the following task:

      1. In the ETL tasks table under ETL tasks > Last run, click the ETL under the Name column.
      2. In the Run configurations table in the ETL details page, click  to edit the active run configuration.
      3. In the Edit run configuration page, navigate to the Run configuration expandable tab and set Execute in simulation mode to No.
      4. Click Save.
    • Locate the ETL in the ETL tasks table and click  to Run it, or schedule an ETL run.
      After you run the ETL, or schedule the ETL for a run, it will extract the data form the source and transfer it to the BMC TrueSight Capacity Optimization database.

    Supported platforms

    The following virtualization platforms are supported:

    • Standalone
    • IBM PowerVM/WPAR
    • Oracle Solaris LDOM/Zone/DSD
    • VMware
    • HP IVM/nPar/vPar
    • Xen
    • Microsoft Hyper-V
    • KVM
    • Virtual Node

    Lookup details

    The BMC TrueSight Gateway Server VIS files parser defines multiple lookup fields for . The following table lists the sequence of field sets for strong lookup and weak lookup.

     

    Entity type

    Strong lookup fields

    Weak lookup fields
    Generic system


     

    VMware Host

      
    VMware Cluster  
    VMware Resource Pool  
    VMware Datastore  
    VMware Virtual Machine  
    PowerVM Host  
    PowerVM Pool  
    PowerVM Shared Processor Pool  
    PowerVM Partition  
    WPAR Host (Standalone Host)  
    WPAR Host (LPAR Host)  
    WPAR  
    Solaris LDOM Host  
    Solaris LDOM  
    Solaris DSD Host  
    Solaris DSD  
    Solaris Zone Host (Standalone Host)  
    Solaris Zone Host (Domain Host)  
    Solaris Zone  
    HP nPar/vPar Host  
    HP nPar/vPar Host Partition (for nPar)  
    HP nPar  
    HP vPar  
    HPIVM Host (essentially similar to Standalone Host)   
    HP IVM Partition  
    Hyper-V Host  
    Hyper-V  

     

    Available metrics

    The following topics list all imported metrics and the linked Gateway Server source tables:

    Related topics

    Using datasets

    Developing custom ETLs

    Dataset reference

    Horizontal and Vertical datasets

    Datasets and metrics

    8 Comments

    1.  
    2.  
    3.  
    4.  
    5.  
    6.  

    7.  

    8.