Configuring DME - Data Mart Extractor

The Moviri DME - Data Mart Extractor generates standard datasets, including SYSDAT, WKLDAT, APPDAT, EVDAT, TAGDAT and CST (Buffer Table). This ETL is useful to import timeseries, business driver metrics, application data, events and tags, and to generate Custom Structure Tables.

Run Configuration

Entity Catalog

Click the Entity catalog tab, and select one of the following options:

1. Shared Entity Catalog:

   1. From the Sharing with Entity Catalog list, select the entity catalog name that is shared between ETLs.

2. Private Entity Catalog: Select this to generate the hierarchy independent of other ETLs and do not want to include other entities look up from other ETLs.

Object Relationship

Click the Object relationship tab, and select one of the three available options, depending on which domain needs to be populated with the imported data:

• Leave all new entities in 'Newly Discovered': select this option to group the new entities under the Newly Discovered section. These entities will not belong to any domain.
• Move all new entities in a new Domain: creates a new domain with the specified name and creates relationships between the new entities and this domain.
• Move all new entities in an existing Domain: creates relationships between the new entities and the chosen destination domain.

Connection

During the creation of Access Key and Access Secret Key, it is required to assign the Administrators Group and the Administrators and Capacity Administrators Roles.

Proxy settings

This section needs to be configured only if the Remote ETL Engine needs to connect to BHCO through a proxy.

Data mart configuration

How to select the appropriate dataset

When creating an ETL task that uses the Moviri DME – Data Mart Extractor, the first configuration step is associating one or more datasets to the ETL task.

• Select 'WKLDAT' if the task is going to import 'Business Drivers' metrics.
• Select 'SYSDAT' if the task is going to import 'System' metrics.
• Select 'APPDAT' if the task is going to import Application configuration.
• Select ‘EVDAT’ if the task is going to import Event data.
• Select ‘TAGDAT' if the task is going to import Tag data.
• Select ‘CST’ if the task is going to import data into a Buffer Table.

Run the ETL

After configuring the ETL, run it to collect data. You can run the ETL in these modes:

1. Simulation mode: Validates connection to the data source without collecting data. Use this mode when running the ETL for the first time or after changing its configuration.
2. Production mode: Collects data from the data source.

1. Running the ETL in simulation mode

To run the ETL in simulation mode:

1. In the console, navigate to Administration > ETL & System Tasks, and select ETL tasks.
2. On the ETL tasks page, click the ETL to display its details.
3. In the Run configurations table, click the pencil icon to modify the ETL settings.
4. On the Run configuration tab, set Execute in simulation mode to Yes, and click Save.
5. Click Run active configuration. A confirmation message appears.
6. On the ETL tasks page, check the ETL run status in the Last exit column.
   OK means the ETL ran without errors. You can now run the ETL in production mode.

7. If the ETL run status shows Warning, Error, or Failed:

   1. On the ETL tasks page, click the pencil icon in the ETL name row's last column.
   2. Check the log and adjust the ETL configuration if needed.
   3. Run the ETL again.
   4. Repeat these steps until the ETL run status is OK.

2. Running the ETL in production mode

Run the ETL manually when needed or schedule it to run at a set time.

Running the ETL manually

1. On the ETL tasks page, click the ETL to display its details.
2. In the Run configurations table, click the pencil icon to modify the ETL settings. The Edit run configuration page appears.
3. On the Run configuration tab, select No for the Execute in simulation mode option, and click Save.
4. To run the ETL immediately, click Run active configuration. A confirmation message appears.
   The ETL collects data from the source and transfers it to the database.

Scheduling the ETL run

By default, the ETL is scheduled to run daily. You can customize this schedule by changing the frequency and period of running the ETL.

To configure the ETL run schedule:

1. On the ETL tasks page, click the ETL, and click Edit Task. The ETL details are displayed.

2. On the Edit task page, do the following, and click Save:

   1. Specify a unique name and description for the ETL task.
   2. In the Maximum execution time before warning field, specify the duration for which the ETL must run before generating warnings or alerts, if any.
   3. Select a predefined or custom frequency for starting the ETL run. The default selection is Predefined.
   4. Select the task group and the scheduler to which you want to assign the ETL task.
3. Click Schedule. A message confirming the scheduling job submission is displayed.
   When the ETL runs as scheduled, it collects data from the source and transfers it to the database.

Verify data collection

After the ETL completes with status OK, BHCO processes data and populates the Workspace. This may take a few minutes.

Confirm the ETL ran successfully and that the Moviri DME - Data Mart Extractor data is refreshed in the Workspace.

To verify whether the ETL ran successfully:

1. In the console, click Administration > ETL and System Tasks > ETL tasks.
2. In the Last exec time column corresponding to the ETL name, verify that the current date and time are displayed.

To verify that the Moviri DME - Data Mart Extractor data is refreshed:

1. In the console, click Workspace.
2. Expand (Domain name) > Systems.
3. In the left pane, verify that the hierarchy displays the new and updated Moviri DME - Data Mart Extractor instances.
4. Click a Moviri DME - Data Mart Extractor entity, and click the Metrics tab in the right pane.
5. Check if the Last Activity column in the Configuration metrics and Performance metrics tables displays the current date.