Performing the initial ETL run

After you install and configure TrueSight Server Automation - Data Warehouse for the primary TrueSight Server Automation site, you must start the ETL process to transfer data from the TrueSight Server Automation database (sites) to the reports data warehouse. ETL populates the empty reports data warehouse with data from one or more TrueSight Server Automation databases (sites).

Plan the first ETL run carefully because it can be resource intensive and time consuming. The amount of time and database resources needed for the initial data load depends on the data volume in the TrueSight Server Automation databases and how you set the cutoff period while configuring the product.

The amount of data in your TrueSight Server Automation database varies based on how long you have been storing data, whether cleanups have been run, and how much data you store in the database. Data could have been accumulating in the TrueSight Server Automation database for a long time and the database might contain a large amount of data.

When loading data from the TrueSight Server Automation database into the reports data warehouse for the first time, it is important to determine how far back in time you need data for your reports. The decision influences the database sizing process and the amount of time needed for the first ETL run. Consequently, your decision about setting a cutoff period for the transfer of data needs to balance your business needs for reporting historical data against your time and resource requirements. The initial ETL run can take a long time because it is performing a full data load.

To prepare for the first ETL run, perform the reports data warehouse sizing exercise by using sizing tools recommended by BMC. After the sizing exercise is complete, ensure the following requirements:

• The required disk space is allocated for the reports data warehouse.
• CPUs are allocated for the database server.
• Database parameters are set to the correct values (see Recommendations for the Oracle processes parameter and the Microsoft SQL Server documentation for SQL Server parameter recommendations).

Before running ETL for the first time, ensure the following:

1. You have enabled data loading of the Property class type property.
2. You have assigned appropriate roles to the users (for example, a primary site user must have the GlobalReportAdmins role). For more information, see User-permissions.

The instructions in this topic describe how to run ETL from the command line. You can also create a TrueSight Server Automation Network Shell Script Job to run ETL. 

To run ETL for the first time

1. Navigate to the BDS_HOME/shared/odi/bin directory.

2. On the reports server, from a command line, start the Network Shell and enter the following command:

   ./run_etl.nsh -s 1  

   Alternatively, from a command line, enter the following command:

   nsh ./run_etl.nsh -s 1  

   The s switch represents the site ID.

   This command uses standard values for the command arguments. For detailed instructions about running ETL and a description of the command arguments, see Running-ETL-using-an-NSH-script.

   To transfer data from other sites, first add those sites if necessary, and then re-run the run_etl command for each site. For more information, see Managing-sites.

3. Create a job for running ETL on a regular basis. For more information, see Running-ETL-using-TrueSight-Server-Automation-jobs.

Where to go from here

Adding-TrueSight-Server-Automation-Data-Warehouse-as-a-component-in-TrueSight-Smart-Reporting