Page tree
Skip to end of metadata
Go to start of metadata

An ETL (Extract, Transform and Load) task is a process that extracts data from a source, such as a database or data file, transforms them in an appropriate format (if necessary), and then loads them into the BMC TrueSight Capacity Optimization Data Warehouse, automatically cataloging new entities.

For more information, refer to the following sections:

Overview

The ETL tasks page lets you manage the tasks responsible for extracting data from all the available sources and feeding it into the BMC TrueSight Capacity Optimization Data Warehouse.

ETL tasks feed BMC TrueSight Capacity Optimization with data for systems and business drivers. Data is extracted from one or more sources, converted into an internal format, loaded into the BMC TrueSight Capacity Optimization Data Warehouse, and then cataloged.

The ETL engine is a core part of BMC TrueSight Capacity Optimization and its management is restricted to administrators. You can assign individual users the roles required to administer ETL tasks by allowing the accounts to perform the WEB_ADMIN and WEB_ADMIN_EDIT activities. You can also use Access Groups and Task Groups to further restrict access to ETL tasks. For more information about users, roles and access groups, refer to Users.

ETL task characteristics

Every ETL task has the following characteristics:

  • It is modular, so that the whole process is defined by composing modules through configuration.
  • It is extensible, so that the collection of a new data format is simple.
  • It is source independent, that is all the transformation operations act on data that are represented through a standard format.
  • It is sequential, so that only new data is loaded every time an ETL task is run.

ETL task composition

ETL tasks are composed of multiple modules that handle specific operations:

  • An extractor module, that connects to the source and extracts new data.
  • One or more transformer modules, that apply in-memory transformations to data in order to convert it to the internal format.
  • One or more loader modules, which load data into the destination (by default, BMC TrueSight Capacity Optimization Data Warehouse and a CVS file).

Extractor modules

There are various extractor modules bundled with BMC TrueSight Capacity Optimization, organized in macro-categories:

  • ETL tasks for third-party management platforms (for example, HP OpenView).
  • ETL tasks that can collect data using OS-native tools and commands (for example, UNIX SAR).
  • ETL tasks that can import standard log formats (for example, NCSA logs).
  • Open ETL tasks, that can import the specific BMC TrueSight Capacity Optimization Open ETL data format defined in Importing data from custom sources through data formats.
  • Custom ETL tasks, that you can develop using the BMC TrueSight Capacity Optimization ETL Development API, as described in Developing.

All available ETL tasks are displayed in the summary table. For more information, refer to Managing ETL and System tasks.

ETL tasks table

For more information about working with ETLs, see the following topics:

Related topics

Developing custom ETLs

Working with ETLs

Out-of-the-box ETL modules

Market Zone Direct ETL modules