ETL life cycle
The development life cycle of a custom ETL consists of the following phases:
The ETL development lifecycle
Each phase comprises of a different set of activities. The following sections discuss these in detail.
In this phase, the work environment is the ETL Development Kit. Activities in this phase include:
- Code editing: Code is written locally on the development workstation.
- Saving and sharing: Code is saved and (optionally) uploaded to the TrueSight Capacity Optimization Repository from where other users can download it for sharing and backup purposes.
- Debugging: New custom ETLs are run for debugging on the development workstation. (In general, it does not deal with real data but with a copy or an extraction of it.)
In this phase the ETL runs in simulation mode on the ETL engine; it gets and manages real data but does not load any data to TrueSight Capacity Optimization. Activities in this phase include:
- Activation: If the ETL is a custom module, it needs to be activated in order to be used in the real environment.
- Configuration: A new ETL instance is created and configured.
- Validation: The ETL instance is run in simulation mode to ascertain how it works with real data.
In this phase the ETL production data runs on the ETL engine. Activities in this phase include:
- Scheduling: The ETL is scheduled to run automatically.
- Running: The ETL extracts and loads data.
- Diagnostics: If errors occur, the ETL generates an alert and the execution log is checked.
- The test and production phases are the same for custom and standard ETLs.
- Activation is not required for standard ETLs (they are active by default).
- The ETL modules you develop are run in a sandbox to ensure that their execution does not harm the production environment.
- Development ETLs are run on development workstations and not on the TrueSight Capacity Optimization ETL engine.
- Data produced by a development ETL cannot be loaded into the TrueSight Capacity Optimization database.