Maintaining ETL tasks
An ETL (Extract, Transform and Load) task is a process that extracts data from a source, such as a database or data file, transforms them in an appropriate format (if necessary), and then loads them into BMC Helix Continuous Optimization, automatically cataloging new entities.
The ETL tasks page enables you to manage the tasks responsible for extracting data from all the available sources and feeding it to BMC Helix Continuous Optimization.
ETL tasks feed BMC Helix Continuous Optimization with data for systems and business drivers. Data is extracted from one or more sources, converted into an internal format, loaded into BMC Helix Continuous Optimization, and then cataloged.
The ETL engine is a core part of BMC Helix Continuous Optimization and its management is restricted to administrators. You can assign individual users the roles required to administer ETL tasks by assigning the admin_section_read and admin_section_edit permissions. You can also use Task Groups to further restrict access to ETL tasks. For more information about users, roles and authorization profiles, see Users and authorization profiles.
ETL task characteristics
Every ETL task has the following characteristics:
- It is modular, so that the whole process is defined by composing modules through configuration.
- It is extensible, so that the collection of a new data format is simple.
- It is source independent, that is all the transformation operations act on data that are represented through a standard format.
- It is sequential, so that only new data is loaded every time an ETL task is run.
ETL task composition
ETL tasks are composed of multiple modules that handle specific operations:
- An extractor module, that connects to the source and extracts new data.
- One or more transformer modules, that apply in-memory transformations to data in order to convert it to the internal format.
- One or more loader modules, which load data into the destination (by default, BMC Helix Continuous Optimization and a CSV file).
Extractor modules
There are various extractor modules bundled with BMC Helix Continuous Optimization, organized in macro-categories:
- ETL tasks for third-party management platforms (for example, HP OpenView).
- ETL tasks that can collect data using OS-native tools and commands (for example, UNIX SAR).
- ETL tasks that can import standard log formats (for example, NCSA logs).
- Open ETL tasks, that can import the specific BMC Helix Continuous Optimization
- Custom ETL tasks, that you can develop using the BMC Helix Continuous Optimization ETL Development API.
All available ETL tasks are displayed in the summary table. For more information, refer to Managing ETL and System tasks.
Comments
Log in or register to comment.