Checking the ETL health for optimal data collection

BMC Helix Dashboards provides an example dashboard for BMC Helix Continuous Optimization. Use this dashboard to monitor the overall status of ETLs and data collection. This dashboard is available in the Marketplace content. You can use this example dashboard as a template and customize it. For information about the Marketplace content, see Building a dashboard quickly from marketplace content. 

The dashboard uses the BMC Helix API data source and APIs to display the following information about ETL health:

  • ETL details, such as name, ID, current status, and last execution time
  • Last counter value
  • Entities imported during the last 7 days
  • Types of imported entities
  • Lookup values

For information about using the BMC Helix API data source, see Using the BMC Helix API data source.

For information about the APIs used to fetch data in the dashboard, see Catalog-Proxy-API-endpoints.

Related topics

Using-BMC-Helix-Dashboards

Out-of-the-box-ETLs

Working-with-ETLs

Troubleshooting-ETLs
BMC Helix Dashboards documentation

Scenario

Alan is an administrator who works in Apex Global. He is responsible for managing the infrastructure capacity. Alan wants to know the overall status of ETLs that are running in his environment. He wants to regularly monitor the health of ETLs to ensure accurate and timely data collection, which is important to effectively manage and plan the capacity needs.

Alan can use the new example dashboard that is available in the Marketplace folder of BMC Helix Dashboards. The dashboard provides the following information:

  • ETL status
  • Entities imported by the ETLs
  • Data samples imported by the ETLs daily
  • Trend of the imported data samples

Alan can review this information to understand the overall impact on data collection. He can decide to reconfigure some of the ETLs based on this information. Alan can also create a custom dashboard from this example dashboard and customize it as required.

To check the ETL health in the dashboard

  1. Open BMC Helix Dashboards from BMC Helix Portal.
  2. In the navigation pane, hover your mouse over the Dashboards icon dashboard_icon_dec_push.png.
  3. Click Browse.
  4. From the Filter by tag list, select Continuous Optimization.
  5. Select Marketplace Content > ETL Health.
    See the following VMware ETL details in the ETL Health dashboard:

    etl_health_dashboard.png

  6. From the ETL list, select the ETLs that you want to review.

  7. For each ETL, review the ETL health and other details provided in the following dashboard panels:

    Panel

    Details

    ETL details

    • Name, ID, and type of ETL
    • Current status of the ETL
    • Name of the scheduler used
    • Last time when the ETL was run
    • ETL schedule status
    • Timestamp of the last and next ETL run (if the ETL run is scheduled)

    Last counters

    The last counter value that indicates the date and time when data was last collected by the ETL.

    Loaded samples

    A line chart that shows the trend of data collected by the ETL. Use this trend to identify any data gaps. By default, the data samples for the last 30 days are displayed. You can change this period.

    Loaded entities

    • The total number of active entities that are loaded by the ETL during the last 7 days
    • Bar chart that shows the number of entities according to their type
    • Pie chart that shows the percentage distribution of entities according to their type

    Lookup values

    Lookup information for entities that include, name, ID, type, and values.

To customize the dashboard

You must create a copy of the example dashboard and customize it based on your requirements.

Example

Alan wants to view only specific ETLs in the ETL list. He also wants to modify the data collection period for the loaded entities.

He performs the following steps to achieve these goals:

  1. Click the Settings icon dashboard_settings_icon.pngand click Save as.
  2. Provide a different name to the dashboard.
  3. To view specific ETLs in the ETL list, perform the following actions:
    1. Click the Settings icon dashboard_settings_icon.pngto view the dashboard settings.
    2. Select Variables > Etl.
    3. Under Query options, edit the variable to filter ETLs.
      Example:
      On the Body tab, replace PROD with TEST to display only the ETLs that are used in the test environment.

      etl_list_health_dashboard.png

    4. Save the changes.
  4. To customize the data collection period for the loaded entities, perform the following actions:
    1. Expand Lookup entities (in the last 7 days) and click Edit.
    2. On the Path tab, replace D7 to specify a different period. Use any of the following parameters and specify values:
       (H = hours, D = days, M = months, Y = years)etl_health_ls_period.png
    3. Save the changes.

 

Tip: For faster searching, add an asterisk to the end of your partial query. Example: cert*