Automatically creating ETLs

You can use the Auto-create ETLs feature to automate the creation of ETLs for populating the Gateway Server data in BMC Helix Continuous Optimization. You can create ETLs to process the Manager runs scheduled on the Gateway Server.

This topic describes following sections:

• • Using the wizard
  • Editing Auto-create ETL configuration
  • Auto-create ETL maintenance

Using the wizard

Enable the Auto-create ETL feature

To enroll an existing Gateway Server for Auto ETL maintenance:

1. Select Administration > Gateway Manager > Gateway Servers.
2. Open an existing Gateway server. 
3. In the message indicating that the ETL is not enrolled, click here to enroll the ETL. 

For a new Gateway Server, the Auto create ETLs to populate data from this gateway server is selected by default. For details, see Automatically creating ETLs#Auto-create ETLs using wizard when enrollment is enabled.

Auto-create ETLs using wizard when enrollment is enabled

1. On the WELCOME page, click Next.

2. On the AUTO-CREATE PARAMETERS page, enter the following details:

   • Basic
   • Advanced

   
   

   Number of manager runs per ETL: Enter the required manager runs per ETL. Specify the maximum number of scheduled Manager runs each auto-created VIS parser ETL will process. A separate ETL is created for each group of scheduled Manager runs equal to this value. For example, if the value specified for this field is 5 and the Gateway Server has 20 scheduled Manager runs, four Auto-create ETLs are created to process the information from the runs. The runs are assigned to the ETLs irrespective of the processed by the run. The auto-create ETLs are configured to process all platform data.

   
   

   Manager runs to exclude: Select the manager runs in the Manager runs to exclude table to be excluded. The selected manager runs are added to the text box. You can also type the name of the manager runs that you want to exclude.

3. On the ETL parameters page, set the values for the following properties under each tab:

   • Basic
   • Advanced

   
   

   
   
4. Click Next.

5. In the Select values to use in scheduling ETLs page, enter the following details:
   
   

   Field	Description
   Start timestamp	Select the timestamp in hours and minutes to schedule the ETL.
   Schedulers for created ETL tasks	By default, all the available schedulers are listed in the Schedulers to use table and are included for the ETL tasks. You can move the schedulers that are not required to the Excluded schedulers table. Ensure the following scheduler configuration: The Schedulers to use table contains the remote scheduler that runs on the host where the Gateway Server and Remote ETL Engine are installed. The other schedulers are moved to the Excluded schedulers table.

6. Click Next.
7. In the Summary page, click Finish. The auto-create ETLs and chain controller tasks are created and scheduled for execution.
   Auto-create ETLs are named in the following manner: <GatewayServer_gateway-server-name_scheduler-ID_task-sequence-number>. For example GatewayServer_GWS-austin-05_2_3 would be the third auto-create ETL to process GWS-austin-05 Manager runs on scheduler with ID 2.
   The Auto Create chain controllers are named in the same way except that their task-sequence-number is always 0.

Editing Auto-create ETL configuration

1. Select Administration > GATEWAY MANAGER > Gateway Servers.
2. Click the name of the required gateway server.

3. Click Edit ETL configuration.

   Note

   The Edit ETL configuration option is displayed only if the ETL is not enrolled for the corresponding gateway server.

   

4. In the Edit Auto-create ETL configuration page, modify the following details:

   Field	Description
   Select values to use in scheduling ETLs
   Start timestamp	Edit the timestamp in hours and minutes to modify the ETL schedule.
   Schedulers for created ETL tasks	Use the >> and << options to select the schedulers for the ETL task by moving the schedulers between the Schedulers to use and Excluded schedulers table.
   Configure Auto-create ETL creation parameters
   Manager runs to exclude	Select the required manager runs in the Manager runs to exclude table to be excluded. The selected manager runs are added to the text box. You can also enter the name of the required manager runs in the text box to exclude them.

5. Click Save.

Auto-create ETL maintenance

The Gateway Server Admin Helper Task, which is configured to run daily, maintains the auto-create ETLs. When Manager runs are scheduled, unscheduled or removed for a Gateway Server that is used in auto-create processing, the Gateway Server Admin Helper Task adjusts the auto-create ETLs as required, during its next execution:

• If a new run is scheduled, and if there is any auto-create ETL that is processing fewer runs than the value specified for Number of gateway Manager runs per ETL in the Gateway ETL auto-create wizard, then the new run is added to that auto-create ETL. If all existing auto-create ETLs for the Gateway Server are processing the maximum number of runs, then a new auto-create ETL is created to process the new run and it is added to the chain controller executing on the scheduler currently processing the least number of Manager runs.

If a scheduled run is stopped or removed, the Gateway Server Admin Helper task removes the run from the auto-create ETL that was processing the run. If it is the last run being processed by the ETL, the value ##EMPTY is specified for the Manager run list of the ETL, indicating that it will not process any runs.

If a scheduled Manager run which was being processed by another VIS files parser ETL when the auto-create processing was initiated for a Gateway Server ceases to be processed by any ETL, the Gateway Server Admin Helper task adds it to the auto-create processing, in the same way as a new Manager run.

The Gateway Server Admin Helper Task does not override any user modifications or deletions of auto-create ETLs.