Page tree
    Skip to end of metadata
    Go to start of metadata

    This topic describes how to integrate TrueSight Capacity Optimization with the TrueSight Capacity Optimization schema-based Capacity Management Database (CDB). This ETL extracts and loads historical data from a Gateway Server database. Use this extractor for loading historical data from a Gateway Server database. It can also be used for loading daily performance and configuration data from a Gateway Server database.

    Note

    BMC - TrueSight Capacity Optimization CDB-CDB extractor remains deprecated in version 10.7 of TrueSight Capacity Optimization. This was deprecated in version 10.0. For more details, see Deprecated and dropped features and components.

    This topic contains the following sections:

    Integration steps

    To integrate TrueSight Capacity Optimization with the CDB Extractor, perform the following task:

    1. Navigate to Administration >  ETL & SYSTEM TASKS > ETL tasks.
    2. In the ETL tasks page, click Add > Add ETL under the Last run tab.

    3. In the  Add ETL  page, set values for the following properties under each expandable tab. 

      Note

      Basic properties are displayed by default in the Add ETL page. These are the most common properties that you can set for an ETL, and it is acceptable to leave the default selections for each as is.

      Basic properties

      Property Description
      Run configuration
      ETL task name By default, this field is populated based on the selected ETL module. You can s pecify a different name for the ETL Task. Duplicate names are allowed.
      Run configuration name Default name is already filled out for you. This field is used to differentiate different configurations you can specify for the ETL task. You can then run the ETL task based on it.
      Deploy status You can select  Production or Test  to mark the ETL tasks. For example, you can start by marking the task as  Test  and change it to Production after you have seen that you are getting what you wanted.
      Description (Optional) Enter a brief description for this ETL.
      Log level Select how detailed you want the ETL log to be. The log includes Error, Warning and Info type of log information.
      • 1 - Light: Add bare minimum activity logs to the log file.
      • 5 - Medium: Add medium-detailed activity logs to the log file.
      • 10 - Verbose: Add detailed activity logs to the log file. Info,

      Note: Log levels 5 and 10 are typically used for debugging or troubleshooting ETL issues. Using a log level of 5 is general practice, however, you may choose level 10 to get a high level of detail while troubleshooting.

      Execute in simulation mode Select Yes if you want to to validate the connectivity between the ETL engine and the target, and to ensure that the ETL does not have any other configuration issues.
      When set to  Yes, the ETL will not store actual data into the data warehouse. This option is useful while testing a new ETL task.
      Module selection

      Ensure that the Based on datasource option is selected.

      Note

      If you select Based on Open ETL template, BMC TrueSight Capacity Optimization is integrated with a Generic extractor based on the selected Open ETL template. For more information, see  Generic ETL based on a template.

      ETL module (BMC recommends you to make this selection first) Select BMC TrueSight Capacity Optimization CDB-CDB extractor.
      Module description A link in the user interface that points you to this technical document for the ETL.
      Platform Select any one:
      • Standalone
      • VMware
      • AIX PowerVM/WPAR
      • Solaris LDOM/Zone/DSD
      • HP IVM/nPar/vPar
      • Xen
      • MS Hyper-V
      • KVM
      • Virtual Node

      If you select a platform, the Object relationships tab appears.

      Data type Performance data is selected by default. You cannot change this setting. Based on the Platform selection, you can select the data type options.
      For Xen and MS HyperV platforms, performance data is the only available option.
      For rest of the platforms, select from the following:

      • detailed disk/network data
      • workload data
      • command statistics
      • user statistics 
      Import HMC data This option is only available if you select AIX Power VM/WPAR platform. By default, yes is selected. If you do not want to import HMC data, select no.
      Days to extract Enter the number of days to extract.
      Entity catalog
      Sharing status Select any one:
      • Shared entity catalog: Select this option if, for the same entities, data is coming from multiple sources. For example, Gateway Server ETL.
        • Sharing with Entity Catalog : Select an entity catalog from the drop-down list.
      • Private entity catalog: Select this option if, for the same entity, data is coming from a single source.
      Object relationships (displayed if you select a platform)
      Associate new entities to

      Specify the domain where you want to add the entities created by the ETL. You can select an existing domain or create a new one.

      Additional Information

      By default, a new domain is created for each ETL, with the same name of the extractor module. As the ETL is created, a new hierarchy rule with the same name of the ETL task is automatically created, with an active status; if you update the ETL specifying a different domain, the hierarchy rule will be updated automatically. The hierarchy rule will be created with status ACTIVE.

      Select any one of the following options:

      • New domain: Create a new domain. Specify the following properties under New domain:
        • Parent: Select a parent domain for your new domain from the domain selector control.
        • Name: Specify a name for your new domain.
      • Existing domain: Select an existing domain. Make a selection for the following property:
        • Domain: Select an existing domain from the domain selector control.

      If the domain is already used by any other hierarchy rule, a Domain conflict option is displayed. You can select one of the following solutions:

        • Enrich domain tree: Create a new independent hierarchy rule to add a new set of entities and/or relations not defined by other ETLs (for example this ETL is managing storage while others are managing servers)
        • ETL Migration: This configuration is recommended if new ETL manages same set of entities and/or relations (already defined in current domain tree). Typical use case is the migration from one or more ETLs to a new ETL instance. It will stop all relations imported by ETL instances and restore only valid relations after first run; this configuration reuses existing hierarchy rule to correctly manage relation updates
          • Select a Hierarchy Rule to be used for current ETL from among the rules that are populating the selected domain.
            Additional information of the ETL using the selected rule is also displayed.

      Connection parameters
      Datasource type Select any one:
      • Select existing datasource
        • Datasource: Select a datasource from the drop down list.
      • Custom datasource
        • Database type: Select a database type from the drop down list. 

          Note

          If you choose to connect to multiple instances, multiple CDB instances fields pop up for user name and password. If you enable the Policy Detection feature, it will allow the ETL to wait for manager runs to finish populating all of the data for a particular day before the Gateway Server CDB-CDB ETL executes. For more information on behavior, see Policy Retry Behavior

        • Database user: Enter the database user name.

        • Database password: Enter the database password.  

        • Based on the database type, specify the following details:

          • For Oracle, Database TNS name : Enter the Database TNS name.

          • For Microsoft SQL Server/Sybase Data source name : Enter the data source name.
            Database name Type in the database name .

          • For Other database, Connection URL : Enter the connection URL .

      ETL task properties
      Task group Select a task group to classify this ETL into. It is not necessary to group it into a task group.
      Running on scheduler Select the scheduler over which you want to run the ETL. The type of schedulers available are:
      • Primary: Runs on the AS.
      • Generic: Runs on a separate machine.
      • Scheduler: Runs on a dedicated scheduler machine.
      Maximum execution time before warning The number of hours, minutes, or days for which the ETL should be executed before generating warnings or alerts, if any.
      Frequency Select the frequency for ETL execution. Available options are:
      • Predefined: Specify the following details:
        • Predefined frequency: Select a frequency from Each DayEach Week or Each Month.
        • Start timestamp: Based on the Predefined frequency selection, specify from the following details:
           - hour: Specify hour in HH format.
           - minute: Specify minute in MM format.
           - week day: Select the day of the week when the ETL has to be executed.
           - month day: Select the day of the month when the ETL has to be executed.
      • Custom: Specify the following details:
        • Custom frequency: Enter a time interval in number of minutes, hours, days, or weeks.
        • Custom start timestamp: Specify the timestamp in YYYY-MM-DD HH:MM format or select it using the date picker.

      Note

      To view or configure Advanced properties, click Advanced. You do not need to set or modify these properties unless you want to change the way the ETL works. These properties are for advanced users and scenarios only.

      Advanced properties

      Property Description
      Run configuration
      Datasets

      Enables you to select or deselect metric groups for which data will be populated from the Available datasets. The OpenStack connector allows you to choose only from the given list of datasets, and you cannot include additional datasets to the run configuration of the ETL.

      1. Click Edit.
      2. Select one (click) or more (ctrl+click or shift+click) datasets that you want to include from the Available datasets and click  >> to move them to Selected datasets.
      3. Click Apply.
      4.x compatibility Select any one:
      • Do not Import Virtual Machine metrics at Host level, but only at VM level.
      • Import Virtual Machine metrics at Host level [BYGM, BYZN, BYLP] to ensure compatibility of 4.x reports.
      Default last counter Set the default last counter in YYYY-MM-DD HH:MM:SS format or use the date picker to select the counter.
      Collection level
      Metric profile selection

      Select any one:

      • Use Global metric profile: Select this to use an out-of-the-box global profile, that is available on the Adding and modifying metric profiles page. By default, all new ETL modules use this profile.
      • Select a custom metric profile: Any metric profiles you add in the Add metric profile page (Administration  >  DATAWAREHOUSE  >  Metric profiles).
        • Custom metric profile: Select a custom metric profile from the drop-down list.

       

      Note

      The Compatibility profile uses metrics that are compatible with the releases earlier than the BMC TrueSight Capacity Optimization 10.3. For all ETLs that are upgraded to BMC TrueSight Capacity Optimization 10.3 or 10.5, by default, Compatibility profile is selected.

       

      For more information, see Adding and modifying metric profiles.

      Levels up to

      The metric level defines the amount of metric imported into the data warehouse. If you increase the level, additional load is added to the data warehouse while decreasing the metric level reduces the number of imported metrics.

      Choose the metric level to apply on selected metrics:

      • [1] Essential
      • [2] Basic
      • [3] Standard
      • [4] Extended

      For more information, see Aging Class mapping.

      When you upgrade an ETL from an earlier release to BMC TrueSight Capacity Optimization 10.3, Levels up to will be set to [4] Extended. 

      Metric filter
      Metric list <for selected dataset>

      Click Edit  and select the metrics that will be loaded for each dataset that you selected under  Run configuration > Datasets . If no metric is specified, all metrics will be loaded.

      Visualizer Database extractor configuration
      Use network name as system name for VMware virtual machines Select any one:
      • True: If you want to use network name as system name for VMware virtual machines, select True.
      • False: If you do not want to use network name as system name for VMware virtual machines, select False. By default, this option is selected.
      Enable additional extraction audit

      Select any one:

      • Yes: Enable additional extraction audit.
      • No: Disable additional extraction audit. By default, this option is selected.
      Extraction audit log file Name and path of the file that contains the ETL audit log; the default value is:
      %BASE/output/EXTRACTIONAUDIT_ETL%TASKID_%PLATFORM_%TABLENAME.txt
      Manager run list
      1. Click Add.
      2. Enter the manager run path.

      You can remove an entry using the delete icon.

      Disable node check

      Select any one:

      • Yes: Disable node check.
      • No: Do not disable node check.
      Connection parameters
      Database prefix Type in the database prefix.
      Exclude policy update detection By default, yes is selected. If you do not want to exclude the policy update, select no.
      Policy detection retry number Enter the policy detection retry number. The default value is 5.
      Policy detection retry period Enter the detection retry period. The default period is 30 minutes.
      Policy completion pct Enter the policy completion percentage. The default percentage is 75%.
      Connect to multiple instances?

      Select any one:

      • Yes: Connect to multiple instances. By selecting this option the DB instances list get display.
        • DB Instances list: Add the database instances to the DB instances list:
          1. Click Add.
          2. Enter database instance
          3. Click Apply.
          4. Click delete icon to delete the instance.
      • No: Do not connect to multiple instances.
      Database user<Instance DB1, leave empty to use default values> Type in the database user name for first database instance.
      Database password Enter the database password.
      Database TNS name (Applies to Oracle ) Type in the database TNS name for first database instance.
      Data source name (Applies to  Microsoft SQL Server/Sybase ) Type in the data source name.
      Database name (Applies to  Microsoft SQL Server/Sybase ) Type in the database name
      Connection URL (Applies to  Other database ) Enter the connection URL.
      Database user<Instance DB2, leave empty to use default values> Type in the database user name for second database instance.
      Database password Enter the database password.
      Database TNS name (Applies to Oracle ) Type in the database TNS name for second database instance.
      Data source name (Applies to  Microsoft SQL Server/Sybase )  Type in the data source name.
      Database name (Applies to  Microsoft SQL Server/Sybase )  Type in the database name
      Connection URL (Applies to  Other database ) Enter the connection URL.
      Additional properties
      List of properties

      Additional properties can be specified for this ETL that act as user inputs during execution.

      1. Click Add.
      2. Add an additional property in the etl.additional.prop.n box.
      3. Click Apply.
        Repeat this task to add more properties.
      Loader configuration
      Empty dataset behavior Choose one of the following actions if the loader encounters an empty dataset:
      • Warn: Warn about loading an empty dataset.
      • Ignore: Ignore the empty dataset and continue parsing.
      ETL log file name Name of the file that contains the ETL execution log; the default value is: %BASE/log/%AYEAR%AMONTH%ADAY%AHOUR%MINUTE%TASKID
      Maximum number of rows for CSV output A number which limits the size of the output files.
      CSV loader output file name Name of the file generated by the CSV loader; the default value is: %BASE/output/%DSNAME%AYEAR%AMONTH%ADAY%AHOUR%ZPROG%DSID%TASKID.
      Capacity Optimization loader output file name Name of the file generated by the BMC TrueSight Capacity Optimization loader; the default value is:
      %BASE/output/%DSNAME%AYEAR%AMONTH%ADAY%AHOUR%ZPROG%DSID%TASKID.
      Detail mode Select the level of detail:
      • Standard: Data will be stored on the database in different tables at the following time granularities: Detail (configurable, by default: 5 minutes), Hourly, Daily, Monthly.
      • Raw also: Data will be stored on the database in different tables at the following time granularities: Raw (as available from the original data source), Detail (configurable, by default: 5 minutes), Hourly, Daily, Monthly.
      • Raw only: Data will be stored on the database in a table only at Raw granularity (as available from the original data source).

      For more information on granularities, see Accessing data using public views and Sizing and scalability considerations.

      Reduce priority

      Select either  Normal  or   High .

      Remove domain suffix from datasource name (Only for systems) If set to True, the domain name is removed from the data source name. For example, server.domain.com will be saved as server.
      Leave domain suffix to system name (Only for systems) If set to True, the domain name is maintained in the system name. For example: server.domain.com will be saved as such.
      Update grouping object definition If set to True, the ETL will be allowed to update the grouping object definition for a metric loaded by an ETL.
      Skip entity creation (Only for ETL tasks sharing lookup with other tasks) If set to True, this ETL does not create an entity, and discards data from its data source for entities not found in BMC TrueSight Capacity Optimization. It uses one of the other ETLs that share lookup to create the new entity.
      Scheduling options
      Hour mask Specify a value to execute the task only during particular hours within the day. For example, 0 – 23 or 1,3,5 – 12.
      Day of week mask Select the days so that the task can be executed only during the selected days of the week. To avoid setting this filter, do not select any option for this field.
      Day of month mask Specify a value to execute the task only during particular days within a month. For example, 5, 9, 18, 27 – 31.
      Apply mask validation By default this property is set to True. Set it to False if you want to disable the preceding Scheduling options that you specified. Setting it to False is useful if you want to temporarily turn off the mask validation without removing any values.
      Execute after time Specify a value in the hours:minutes format (for example, 05:00 or 16:00) to wait before the task must be executed. This means that once the task is scheduled, the task execution starts only after the specified time passes.
      Enqueueable Select one of the following options:
      • False (Default): While a particular task is already running, if the next execution command arises – it is ignored.
      • True: While a particular task is already running, if the next execution command arises – it is placed in a queue and is executed as soon as the current execution ends.

        

    4. Click Save.

      You return to the Last run tab under the ETL tasks page.
    5. Validate the results in simulation mode: In the ETL tasks table under ETL tasks > Last run, locate your ETL (ETL task name), click  to run the ETL.
      After you run the ETL, the Last exit column in the ETL tasks table will display one of the following values:
      • OK: The ETL executed without any error in simulation mode.
      • WARNING: The ETL execution returned some warnings in simulation mode. Check the ETL log.
      • ERROR: The ETL execution returned errors and was unsuccessful. Edit the active Run configuration and try again.
    6. Switch the ETL to production mode: To do this, perform the following task:
      1. In the ETL tasks table under ETL tasks > Last run, click the ETL under the Name column.
      2. In the Run configurations table in the ETL details page, click  to edit the active run configuration.
      3. In the Edit run configuration page, navigate to the Run configuration expandable tab and set Execute in simulation mode to No.
      4. Click Save.
    7. Locate the ETL in the ETL tasks table and click  to Run it, or schedule an ETL run.
      After you run the ETL, or schedule the ETL for a run, it will extract the data form the source and transfer it to the BMC TrueSight Capacity Optimization database.

      Notes

      • If your imported data includes workloads, you can optionally activate the hierarchy rule to create aggregated workloads; for more information, see Activating the Aggregated Workload hierarchy.
      • If your imported data are linked to IBM AIX or Oracle Solaris platforms, another hierarchy rule (Gateway Server Aggregated metrics AIX/Solaris 7.5.X) will be created in order to retrieve additional metrics at box/container and pool level.

     

    Policy Retry Behavior

    The Gateway Server Policy Detection feature allows the ETL to wait for Manager runs to finish populating all of the data for a particular day before the Gateway Server CDB-CDB ETL executes.

    You cannot use the Policy Detection feature for data recovery purpose. So, if 99 out of 100 policies are successfully populated on a particular day, the ETL executes and later the last domain is populated into the Gateway Server database after the ETL has finished. The Gateway Server ETL cannot re-extract the data from the missed domains.

    The Policy Detection feature enables the Gateway Server ETL to check whether the policy files have been successfully populated for the current day. So it does not wait until they have been populated or wait for a time out period and then begin population if a defined minimum number of domains have been populated.

    Listed are the Policy Detection behavior shown when integration of BMC TrueSight Capacity Optimization with Gateway Server is preformed.

    1. The ETL checks and updates the policy and last counter values based on the most recent time a policy (a grouping of computers processes together) has been updated in the Gateway Server database.
    2. If 100% of the policy groups populated into the target database schema have current update dates, the Gateway Server ETL begins its execution.
      To calculate the percentage, the Gateway Server ETL will look at the number of policies that successfully transferred on a particular day divided by the number of policies that successfully transferred within the previous "Exclude Policy from minimum completion percentage, if no data for N days". The default value is 5 days.
    3. If less than 100% of the policy groups populated into the target database schema have current update dates, the Gateway Server ETL will wait for bpa.policy.retry.period minutes. The default value is 30 minutes and then repeat the check.
    4. After bpa.policy.retry retries the Gateway Server ETL will check the percentage of the policy files that were successfully updated. By default it retrieves five policies. If more than five bpa.policy.minimum.completion of the policy groups have been updated then the Gateway Server ETL execution begins. If less than bpa.policy.minimum.completion percent have current dates then the execution of the ETL will be aborted.

    Note

    The Visualizer database has five policies in its database and based on the number of checks performed, the policies are populated. For example:

    • The first check populates three policies that is 60%.
    • The second check populates four policies, that is 80%.
    • The third check populates five policies, that is 80% (assuming it a default).

    After the bpa.policy.retry attempts, ETL checks the total percentage of the completed domains (80%) and tests against bpa.policy.minimum.completion (default as 75%). Based on the test. the percentage will became higher and the data processing is started from that point.
    Execute this test for five consecutive days and on the last day the fifth policy fails to populate into the Visualizer database that is it is no longer be included in the total policy count because the last population exceed the "Exclude Policy from minimum completion pct if no data for N days" window.
    So, in this case first check populates three policies (75%) and second check populates four policies (100%). Data processing will begin after the second check.

    Supported platforms and available metrics

    The following table provides downloadable PDFs that contain a list of all imported metrics and linked Gateway Server source tables:

    SystemSupported platformsAvailable metrics (PDF)
    HP PartitionHP IVM/nPar/vParHP-UX_metrics
    IBM PowerVMIBM PowerVM/WPARIBM_PowerVM_metrics
    Linux KVMKVMLinux_KVM_metrics
    Microsoft Hyper-V systemsMicrosoft Hyper-VHyper-V_metrics
    Oracle virtualizationSolaris LDOM/Zone/DSDOracle_virtualization_metrics
    StandaloneStandaloneStandalone_metrics
    VirtualVirtual NodeVirtual_metrics
    VMware vSphereVMwareVMware_metrics
    XenXenXen_metrics

    Note

    Virtual Node provides support for all platforms listed above, except for the Standalone platform.

    Related topics

    Using ETL datasets

    Developing custom ETLs

    Dataset reference for ETL tasks

    Horizontal and Vertical datasets

    Viewing datasets and metrics by dataset and ETL module

    1 Comment


    1. Supported platforms and available metricsThe following table provides downloadable PDFs that contain a list of all imported metrics and linked Gateway Server source tables: