Automatically creating ETLs

You can use the Auto-create ETLs to automate the creation of ETLs for populating Gateway Server data. Using the BMC TrueSight Capacity Optimization wizard for Auto-create ETLs you can create ETLs to process the Manager runs scheduled on the Gateway Server.

Info

It is recommended to use Auto-create ETLs to create ETLs for populating Gateway Server data.

This topic describes following sections:

Using the wizard

Before you begin

Ensure that BMC TrueSight Capacity Optimization wizard for Auto-create ETLs is displayed by using either of the following:

  • Select Auto create ETLs to populate data from this Gateway Server in the Add Gateway Server page. For more details, see To Auto-create ETLs using wizard.
  • Select Administration > GATEWAY MANAGER > Gateway Servers > gatewayserverName and in the message indicating that the ETL is not enrolled, click here to enroll the ETL.

    Note

    The message to enroll the ETL is displayed only if an ETL is not already created for the corresponding gateway server.

To Auto-create ETLs using wizard

  1. On the WELCOME page, click Next.
  2. On the AUTO-CREATE PARAMETERS page, enter the following details:


    Number of manager runs per ETL: Type the required manager runs per ETL in the box. Specify the maximum number of scheduled Manager runs each auto-created BMC TrueSight Capacity Optimization VIS parser ETL will process. A separate ETL is created for each group of scheduled Manager runs equal to this value. For example, if the value specified for this field is 5 and the Gateway Server has 20 scheduled Manager runs, four Auto-create ETLs are created to process the information from the runs. The runs are assigned to the ETLs irrespective of the processed by the run. The auto-create ETLs are configured to process all platform data.


    Manager runs to exclude: Select the required manager runs in the Manager runs to exclude table to be excluded and to add them to the text box. You can also type the name of the required manager runs separated by semi-colon in the box to exclude them.

  3. On the ETL PARAMETERS page, set the values for the following properties under each expandable tabs:


    Property Description
    Entity catalog
    Sharing status Select any one:
    • Shared entity catalog: Select this option if, for the same entities, data is coming from multiple sources. For example, Gateway Server ETL.
      • Sharing with Entity Catalog : Select an entity catalog from the drop-down list.
    • Private entity catalog: Select this option if, for the same entity, data is coming from a single source.
    Object relationships
    Associate new entities to

    Specify the domain where you want to add the entities created by the ETL. You can select an existing domain or create a new one.

    Additional Information

    By default, a new domain is created for each ETL, with the same name of the extractor module. As the ETL is created, a new hierarchy rule with the same name of the ETL task is automatically created, with an active status; if you update the ETL specifying a different domain, the hierarchy rule will be updated automatically. The hierarchy rule will be created with status ACTIVE.

    Select any one of the following options:

    • New domain: Create a new domain. Specify the following properties under New domain:
      • Parent: Select a parent domain for your new domain from the domain selector control.
      • Name: Specify a name for your new domain.
    • Existing domain: Select an existing domain. Make a selection for the following property:
      • Domain: Select an existing domain from the domain selector control.

    If the domain is already used by any other hierarchy rule, a Domain conflict option is displayed. You can select one of the following solutions:

    • Enrich domain tree: Create a new independent hierarchy rule to add a new set of entities and/or relations not defined by other ETLs (for example this ETL is managing storage while others are managing servers)
    • ETL Migration: This configuration is recommended if new ETL manages same set of entities and/or relations (already defined in current domain tree). Typical use case is the migration from one or more ETLs to a new ETL instance. It will stop all relations imported by ETL instances and restore only valid relations after first run; this configuration reuses existing hierarchy rule to correctly manage relation updates
      • Select a Hierarchy Rule to be used for current ETL from among the rules that are populating the selected domain. Additional information of the ETL using the selected rule is also displayed.


    Property Description
    Collection level
    Metric profile selection

    Select any one:

    • Use Global metric profile: Select this to use an out-of-the-box global profile, that is available on the Adding and managing metric profiles page. By default, all new ETL modules use this profile.
    • Select a custom metric profile: Any metric profiles you add in the Add metric profile page (Administration  >  DATAWAREHOUSE  >  Metric profiles).
      • Custom metric profile: Select a custom metric profile from the drop-down list.

    Note

    The Compatibility profile uses metrics that are compatible with the releases earlier than the BMC TrueSight Capacity Optimization 10.3. For all ETLs that are upgraded to BMC TrueSight Capacity Optimization 10.3 or 10.5 or 10.7, by default, Compatibility profile is selected.

    For more information, see Adding and managing metric profiles.

    Levels up to

    The metric level defines the amount of metric imported into the data warehouse. If you increase the level, additional load is added to the data warehouse while decreasing the metric level reduces the number of imported metrics.

    Choose the metric level to apply on selected metrics:

    • [1] Essential
    • [2] Basic
    • [3] Standard
    • [4] Extended

    For more information, see Aging Class mapping.

    When you upgrade an ETL from an earlier release to BMC TrueSight Capacity Optimization 10.3, Levels up to will be set to [4] Extended. 
    ETL task properties
    Task group Select the required task group from the drop-down list. <Need to add what it does>
    Data type
    Data type Select from the following:
    • detailed disk/network data
    • workload data
    • command statistics
    • user statistics 

      By default, performance data is selected.


  4. Click Next.
  5. On the SCHEDULE page, enter the following details:

    FieldDescription
    Start timestampSelect the timestamp in hours and minutes to schedule the ETL .
    Schedulers for created ETL tasksBy default, all available schedulers are listed in the Schedulers to use table and are included for the ETL tasks. You can select the required scheduler from the Schedulers to use table, and add them to the Excluded schedulers table using the >> button. You can also select the required scheduler from the Excluded schedulers table and add them to the Schedulers to use table using the << button.
  6. Click Next.
  7. On the Summary page, click Finish. The auto-create ETLs and chain controller tasks are created and scheduled for execution.
    Auto-create ETLs are named in the following manner: <GatewayServer_gateway-server-name_scheduler-ID_task-sequence-number>. For example GatewayServer_GWS-austin-05_2_3 would be the third auto-create ETL to process GWS-austin-05 Manager runs on scheduler with ID 2.
    The Auto Create chain controllers are named in the same way except that their task-sequence-number is always 0.

Auto-create ETL maintenance

The Gateway Server Admin Helper Task, which is configured to run daily, maintains the auto-create ETLs. When Manager runs are scheduled, unscheduled or removed for a Gateway Server that is used in auto-create processing, the Gateway Server Admin Helper Task adjusts the auto-create ETLs as required, during its next execution:

  • If a new run is scheduled, and if there is any auto-create ETL that is processing fewer runs than the value specified for Number of gateway Manager runs per ETL in the Gateway ETL auto-create wizard, then the new run is added to that auto-create ETL. If all existing auto-create ETLs for the Gateway Server are processing the maximum number of runs, then a new auto-create ETL is created to process the new run and it is added to the chain controller executing on the scheduler currently processing the least number of Manager runs.
  • If a scheduled run is stopped or removed, the Gateway Server Admin Helper task removes the run from the auto-create ETL that was processing the run. If it is the last run being processed by the ETL, the value ##EMPTY is specified for the Manager run list of the ETL, indicating that it will not process any runs.
  • If a scheduled Manager run which was being processed by another VIS files parser ETL when the auto-create processing was initiated for a Gateway Server ceases to be processed by any ETL, the Gateway Server Admin Helper task adds it to the auto-create processing, in the same way as a new Manager run.

The Gateway Server Admin Helper Task does not override any user modifications or deletions of auto-create ETLs.

For more information on the Gateway Server Admin Helper Task, see Creating and running data feed health reports.

Editing Auto-create ETL configuration

  1. Select Administration > GATEWAY MANAGER > Gateway Servers.
  2. Click the name of the required gateway server.
    The gatewayserverName page is displayed.
  3. Click Edit ETL configuration.
    The Edit Auto-create ETL configuration page is displayed.

    Note

    The Edit ETL configuration option is displayed only if the ETL is not enrolled for the corresponding gateway server.


  4. In the Edit Auto-create ETL configuration page, modify the following details:

    FieldDescription
    Select values to use in scheduling ETLs
    Start timestampEdit the timestamp in hours and minutes to modify the ETL schedule.
    Schedulers for created ETL tasksYou can modify the Schedulers to use table by using the >> button to add them to the Excluded schedulers table. You can also select the required scheduler from the Excluded schedulers table and add them to the Schedulers to use table using the << button.
    Configure Auto-create ETL creation parameters
    Manager runs to excludeSelect the required manager runs in the Manager runs to exclude table to be excluded and to add them to the text box. You can also type the name of the required manager runs in the text box to exclude them.
  5. Click Save.
Was this page helpful? Yes No Submitting... Thank you

Comments