Automatically creating ETLs

You can use the Auto-create ETLs feature to automate the creation of ETLs for populating the Gateway Server data in BMC Helix Capacity Optimization. You can create ETLs to process the Manager runs scheduled on the Gateway Server.

This topic describes following sections:

Using the wizard

Enable the Auto-create ETL feature

To enroll an existing Gateway Server for Auto ETL maintenance:

  1. Select Administration > Gateway Manager > Gateway Servers.
  2. Open an existing Gateway server. 
  3. In the message indicating that the ETL is not enrolled, click here to enroll the ETL. 

Note

The message to enroll the ETL is displayed only if an ETL is not already created for the corresponding gateway server.

For a new Gateway Server, the Auto create ETLs to populate data from this gateway server is selected by default. For details, see Automatically creating ETLs#Auto-create ETLs using wizard when enrollment is enabled.

Auto-create ETLs using wizard when enrollment is enabled

  1. On the WELCOME page, click Next.
  2. On the AUTO-CREATE PARAMETERS page, enter the following details:


    Number of manager runs per ETL: Enter the required manager runs per ETL. Specify the maximum number of scheduled Manager runs each auto-created VIS parser ETL will process. A separate ETL is created for each group of scheduled Manager runs equal to this value. For example, if the value specified for this field is 5 and the Gateway Server has 20 scheduled Manager runs, four Auto-create ETLs are created to process the information from the runs. The runs are assigned to the ETLs irrespective of the processed by the run. The auto-create ETLs are configured to process all platform data.


    Manager runs to exclude: Select the manager runs in the Manager runs to exclude table to be excluded. The selected manager runs are added to the text box. You can also type the name of the manager runs that you want to exclude.

  3. On the ETL parameters page, set the values for the following properties under each tab:


    Property Description
    Entity catalog
    Sharing with Entity Catalog Select any one:
    • Shared entity catalog: Select this option if, for the same entities, data is coming from multiple sources. For example, Gateway Server ETL.
      • Sharing with Entity Catalog: Select an entity catalog from the drop-down list.
    • Private entity catalog: Select this option if, for the same entity, data is coming from a single source.
    Object relationships
    Associate new entities to

    Specify the domain where you want to add the entities created by the ETL. You can select an existing domain or create a new one.

    Additional Information

    By default, a new domain is created for each ETL, with the same name of the extractor module. As the ETL is created, a new hierarchy rule with the same name of the ETL task is automatically created, with an active status; if you update the ETL specifying a different domain, the hierarchy rule will be updated automatically. The hierarchy rule will be created with status ACTIVE.

    Select any one of the following options:

    • New domain: Create a new domain. Specify the following properties under New domain:
      • Parent: Select a parent domain for your new domain.
      • Name: Enter a name for your new domain.
    • Existing domain: Select an existing domain from the Domain list.

    If the domain is already used by any other hierarchy rule, a Domain conflict option is displayed. You can select one of the following solutions:

    • Enrich domain tree: Create a new independent hierarchy rule to add a new set of entities and/or relations not defined by other ETLs (for example this ETL is managing storage while others are managing servers)
    • ETL Migration: This configuration is recommended if new ETL manages same set of entities and/or relations (already defined in current domain tree). Typical use case is the migration from one or more ETLs to a new ETL instance. It will stop all relations imported by ETL instances and restore only valid relations after first run; this configuration reuses existing hierarchy rule to correctly manage relation updates
      • Select a Hierarchy Rule to be used for current ETL from among the rules that are populating the selected domain. Additional information of the ETL using the selected rule is also displayed.


    Property Description
    Collection level
    Metric profile selection
    • Use Global metric profile: Select to use an out-of-the-box global profile, that is available on the Adding and managing metric profiles page. By default, all new ETL modules use this profile.
    • Select a custom metric profile: Any metric profiles you add in the Add metric profile page (Administration > DATAWAREHOUSE > Metric profiles).
      • Custom metric profile: Select a custom metric profile from the drop-down list.

    For more information, see Adding and managing metric profiles.

    Levels up to

    The metric level defines the amount of metric imported into the data warehouse. If you increase the level, additional load is added to the data warehouse while decreasing the metric level reduces the number of imported metrics.

    Choose the metric level to apply on selected metrics:

    • [1] Essential
    • [2] Basic
    • [3] Standard
    • [4] Extended

    For more information, see Aging Class mapping.

    ETL task properties
    Task group Select the required task group from the list.
    Data type
    Data type

    By default, data sets associated with the performance data are included.

    Select additional data types for which you want to collect data:

    Data type Associated datasets

    Detail disk/network data



    [4] SYSDSK - System Disks counters

    [5] SYSNET - System NIC counters
    [110] SYSSEA - System Network Adapter counters

    Workload data

    [100] PROCDET - Process level detailed metrics

    Command statistics

    [3] SYSPROC - System Processes counter

    User statistics

    [101] SYSUSR - User level detailed metrics

    The data types are associated with different data sets. Depending on the selected data type, specific metrics of the associated data set are included. For information about the metrics associated with the data sets, see Entity relationship, metrics and lookup information for BMC - TrueSight Capacity Optimization Gateway VIS files parser.

  4. Click Next.
  5. In the Select values to use in scheduling ETLs page, enter the following details:

    FieldDescription
    Start timestampSelect the timestamp in hours and minutes to schedule the ETL.
    Schedulers for created ETL tasksBy default, all the available schedulers are listed in the Schedulers to use table and are included for the ETL tasks. You can move the schedulers that are not required to the Excluded schedulers table. Ensure the following scheduler configuration:
    • The Schedulers to use table contains the remote scheduler that runs on the host where the Gateway Server and Remote ETL Engine are installed.
    • The other schedulers are moved to the Excluded schedulers table.
  6. Click Next.
  7. In the Summary page, click Finish. The auto-create ETLs and chain controller tasks are created and scheduled for execution.
    Auto-create ETLs are named in the following manner: <GatewayServer_gateway-server-name_scheduler-ID_task-sequence-number>. For example GatewayServer_GWS-austin-05_2_3 would be the third auto-create ETL to process GWS-austin-05 Manager runs on scheduler with ID 2.
    The Auto Create chain controllers are named in the same way except that their task-sequence-number is always 0.

Editing Auto-create ETL configuration

  1. Select Administration > GATEWAY MANAGER > Gateway Servers.
  2. Click the name of the required gateway server.
  3. Click Edit ETL configuration.

    Note

    The Edit ETL configuration option is displayed only if the ETL is not enrolled for the corresponding gateway server.


  4. In the Edit Auto-create ETL configuration page, modify the following details:

    FieldDescription
    Select values to use in scheduling ETLs
    Start timestampEdit the timestamp in hours and minutes to modify the ETL schedule.
    Schedulers for created ETL tasksUse the >> and << options to select the schedulers for the ETL task by moving the schedulers between the Schedulers to use and Excluded schedulers table.
    Configure Auto-create ETL creation parameters
    Manager runs to excludeSelect the required manager runs in the Manager runs to exclude table to be excluded. The selected manager runs are added to the text box. You can also enter the name of the required manager runs in the text box to exclude them.
  5. Click Save.

Auto-create ETL maintenance

The Gateway Server Admin Helper Task, which is configured to run daily, maintains the auto-create ETLs. When Manager runs are scheduled, unscheduled or removed for a Gateway Server that is used in auto-create processing, the Gateway Server Admin Helper Task adjusts the auto-create ETLs as required, during its next execution:

  • If a new run is scheduled, and if there is any auto-create ETL that is processing fewer runs than the value specified for Number of gateway Manager runs per ETL in the Gateway ETL auto-create wizard, then the new run is added to that auto-create ETL. If all existing auto-create ETLs for the Gateway Server are processing the maximum number of runs, then a new auto-create ETL is created to process the new run and it is added to the chain controller executing on the scheduler currently processing the least number of Manager runs.

If a scheduled run is stopped or removed, the Gateway Server Admin Helper task removes the run from the auto-create ETL that was processing the run. If it is the last run being processed by the ETL, the value ##EMPTY is specified for the Manager run list of the ETL, indicating that it will not process any runs.

If a scheduled Manager run which was being processed by another VIS files parser ETL when the auto-create processing was initiated for a Gateway Server ceases to be processed by any ETL, the Gateway Server Admin Helper task adds it to the auto-create processing, in the same way as a new Manager run.

The Gateway Server Admin Helper Task does not override any user modifications or deletions of auto-create ETLs.

For more information on the Gateway Server Admin Helper Task, see Creating and running data feed health reports.

Was this page helpful? Yes No Submitting... Thank you

Comments