Viewing the log files for ETL runs
Running the run_etl.nsh or stop_etl.nsh script creates log files that contain details related to the script, scenario execution, and errors. The log files are created in the <TSSA-DW_installation_directory>/logs/etl/timeStamp or agentLogFilesDirectory directory, where timeStamp represents the date and time that the log file was created. A new timeStamp directory is created for each ETL run. If a script fails, you can use these logs to determine the point at which the script failed and how to recover. The administrator can remove the log files from the directory when appropriate.
In the log file names, variables are defined as follows:
- siteID: The site ID
- timeStamp: Date and time that the log file was created; uses the format YYMMDDHHMMSS (year, month, day, hour, minute, and second)
When you run the the run_etl.nsh script, the following log files are created in the <TSSA-DW_installation_directory>/logs/etl/timeStamp directory:
Log file name | Description |
|---|---|
BSA_site_siteID_run_etl.log | Main log file that provides details of the run_etl process. |
BSA_site_siteID_exception_run_etl.err | Log file that contains exceptions. |
When you run the stop_etl.nsh script, the following log files are created in the <TSSA-DW_installation_directory>/logs/etl/timeStamp directory:
Log file name | Description |
|---|---|
BSA_site_siteID_timeStamp_stop_etl.log | Main log file that provides the details of the stop_etl process. |
BSA_site_siteID_exception_stop_etl.err | Log file that contains exceptions. |
To view the agent log files for ETL runs
When you run the run_etl.nsh script using the -f argument with the agentlogs.properties file, a set of log files is created for the agent log.
These log files are located in the following directories: <TSSA-DW_Installation_Directory>/etl and AgentLogProcess
The location of the agentLogFilesDirectory is specified during TrueSight Server Automation - Data Warehouse installation.
To cleanup the log files by ETL run process
- As an administrator connect to the TrueSight Server Automation - Data Warehouse host.
- Navigate to the following path:
- (Windows) <tssadw_installation_directory>\shared\ConfigurationManagement\generic_configuration.properties
- (Linux)<tssadw_installation_directory>/shared/ConfigurationManagement/generic_configuration.properties
- Open generic_configuration.properties file.
- Set the value of the ENABLE_ETL_LOG_FILES_CLEANUP parameter to true.
Set the value for the ETL_LOG_FILES_RETENTION_PERIOD parameter to the number of days for which you have to retain the ETL log files and folders. By default, this configuration parameter is set to 30 days.
Save the changes.
- Run the ETL process.
The system deletes files and folders from the following path after a successful run, that exceed the retention period specified in the parameter:- $BDS_HOME\logs\update_custom_property\etl
- $BDS_HOME\logs\etl
To validate the ETL process
Perform the following steps to verify if the ETL process is successful.
- As a TrueSight Server Automation - Data Warehouse administrator (DWAdmin), launch the TrueSight Server Automation - Data Warehouse console.
- Click ETL Management > ETL Status to check the status.
The latest ETL run log files displays the count and path of the deleted files and folders.For example:
[18 Aug 2023 20:34:28,718] [DEBUG] Cleaning up old log folders and files from ETL logs
[18 Aug 2023 20:34:28,785] [DEBUG] 8 no of folders deleted from path : C:\PROGRA~1\BMCSOF~1\TSSA-DW\logs\etl
[18 Aug 2023 20:34:28,802] [DEBUG] 6 no of folders deleted from path : C:\PROGRA~1\BMCSOF~1\TSSA-DW\logs\update_custom_property\etl