Overview of ETLs

You can use ETLs that are available with TrueSight Cloud Cost Control to collect the cost and usage data of your public and on-premises infrastructure. The ETLs are configurable modules that are based on the Extract, Transform, and Load (ETL) technology. They are also known as connectors. They connect to the data sources and collect data. The collected data is transferred to the TrueSight Capacity Optimization database. This data is used for analyzing and optimizing the cost of your cloud and on-premises resources.Depending on the operating systems and technologies, different ETLs are available for data collection.

Multiple ETLs can access the same data sources to collect different type of data. For example, the AWS API ETL collects the configuration and performance data of your virtual machines in the AWS cloud, and the AWS Cost and Usage ETL collects the usage and cost data of these virtual machines. TrueSight Capacity Optimization provides a mechanism called as shared lookup to let multiple ETLs reference the same data sources.
For more information about shared lookup, see Understanding entity identification and lookup..

Types of ETLs

Depending on how ETLs collect data from data sources, they are classified as extractors and parsers. Based on the frequency of data collection, ETLs are also classified as service ETLs and batch ETLs. Service ETLs continuously collect data from data sources. You can configure the frequency of loading the collected data into the TrueSight Capacity Optimization database. The VMware vCenter and RHEV ETLs are service ETLs.

Batch ETLs collect data at scheduled intervals from the source. You can configure them to run once in an hour, a day, or a week. These ETLs collect data and load it into the database for the entire interval in one run. All the public cloud provider ETLs are batch ETLs.

Although batch ETLs collect data less frequently than service ETLs, the granularity of data (data points per hour) collected by both types of ETLs might be the same.

ETLs are also classified as follows:

  • Out-of-the-box ETLs: These ETLs are available with the product. You can start using them for data collection after installing the product. For information about out-of-the-box ETLs, see Collecting-data.

  • Custom ETLs: These ETLs are created by customers, BMC Professional Services, and BMC deployment partners. These ETL do not require an additional license entitlement. TrueSight Capacity Optimization provides a Software Development Kit (SDK) that includes an Integrated Development Environment (IDE) for creating custom ETLs. For more information, see Developing custom ETLs. The TrueSight Capacity Optimization SDK is supported by BMC; However, custom ETLs are not supported by BMC as specified on the Customization Policy page of the BMC Support website.

  • Community-based ETLs: These ETLs (formerly known as "template") are not distributed with the product and are not supported by BMC (as mentioned on the Customization Policy page of the BMC Support website). They do not require an additional license entitlement and are provided for free. You can download these ETLs from BMC Marketplace. When you access BMC Marketplace, ensure that you select Free as price filtering criteria to view the list of community-based ETLs.

  • Project-based ETLs: These ETLs are available only through a professional services engagement and are supported based on the engagement terms. You can access these ETLs on the BMC Marketplace.

 

Tip: For faster searching, add an asterisk to the end of your partial query. Example: cert*