Collecting data via ETL modules

An ETL module can be deployed in one or multiple instances (also called ETL tasks) in the same TrueSight Capacity Optimization implementation. For example, to extract data for different instances of the data source or to run multiple processes at the same time (related to different objects) from the same data source. Depending on the way an ETL has been implemented, an ETL task can be either a scheduled ETL, a process executed at scheduled time to run periodically against its data source, or a service ETL, a process that runs continuously.

A TrueSight Capacity Optimization administrator or user having the required privileges can deploy, execute and manage an ETL taskFor more information about how to deploy, execute and manage an ETL task, see ETL tasks. For information on how to develop custom ETL modules, see Developing custom ETLs.

Entity and metric types

Data is imported into TrueSight Capacity Optimization as time-series representing metrics for different types of entities. TrueSight Capacity Optimization provides three types of entities: systems, business drivers and domains.

 For more information on metric types and entities, see Entities and metrics and Entity types.

Entity lookup for ETL modules

In a TrueSight Capacity Optimization implementation, there can be more that one ETL module importing data for the same entities. For example, one ETL module might import performance data about servers, while a configuration management database (CMDB) ETL module might import asset numbers for the same servers. Unless specified, the default option in TrueSight Capacity Optimization considers all entities referred by ETL modules as distinct, therefore TrueSight Capacity Optimization provides a mechanism know as share lookup that allows you to configure two or more ETL modules to consider different entities referred by them as to represent the same entity. For more information, see Handling ETL lookup name.

Types of ETL modules

ETL modules in TrueSight Capacity Optimization extract data such as performance, configuration, and relationship information from a source. They are classified as follows:

  • Supported ETL modules: All the out-of-the-box ETL modules, including those provided by BMC partners, Sentry and Moviri. The ETL modules are supported by BMC, as per BMC support policies. For more information, see Out-of-the-box ETL modulesSentry ETL modulesMoviri ETL modules

  • Custom ETL modules: ETL modules created by customers, BMC Professional Services and BMC deployment partners. These ETL modules do not require an additional license entitlement. TrueSight Capacity Optimization provides a Software Development Kit (SDK) that also includes an Integrated Development Environment (IDE) that can be used to easily create custom ETL modules for any data-source. For more information, see Developing custom ETLs. The TrueSight Capacity Optimization SDK is supported by BMC; However, custom ETL modules are not supported by BMC as noted on the Customization Policy page of the BMC Support website.

  • Community-based ETL modules: These community-based (formerly known as "template") ETL modules are not distributed with the product and do not require an additional license entitlement. They are provided for free on an as-is basis and are not supported by BMC (as noted on the Customization Policy page of the BMC Support website). You can download all these ETL modules from  BMC Marketplace . When you access the  BMC Marketplace , ensure that you select Free as price filtering criteria to visualize the list of Community-based ETL modules. 

    Info

    The BMC Marketplace features project-based ETL modules too. These modules are available only through a professional services engagement and are supported based on the engagement terms.

Depending on how ETL modules extract data from their data sources, they are also classified as Parsers and Extractors.

ETLs can also be classified as service ETLs and batch ETLs:

  • Service ETLs:  Service ETLs continuously extract, accumulate, and load data from the source into TrueSight Capacity Optimization. You can schedule the frequency of data extraction and loading as part of the ETL configuration. By default, service ETLs extract data at every 15th minute of an hour.
  • Batch ETLs: Batch ETLs extract data at scheduled intervals from the source. You can schedule the batch ETLs to run once in an hour, day or week. Batch ETLs extract the data for the entire interval in one run. 

Although batch ETLs extract data less frequently than service ETLs, the granularity of data (data points per hour) that is extracted by both types of ETLs might be the same.

Where to go from here

For information about how TrueSight Capacity Optimization collects data using the ETL modules, see:

Was this page helpful? Yes No Submitting... Thank you

Comments