Troubleshooting the Moviri ETLs

The Moviri ETLs might fail to collect data due to incorrect configurations or other issues. Review the ETL logs to investigate and troubleshoot the issues.

Related topics

Moviri-ETLs

Performing-system-maintenance-tasks

Collecting-ETL-logs-at-log-level-10

Resolutions for common issues

The following table explains the common issues for the ETL failures and their resolutions. 

Symptom

Cause

Resolution

The following message is reported in the ETL log:

BCO_TASK_FAIL005: Critical error detected during task XX execution: java.lang.OutOfMemoryError: GC overhead limit exceeded for batch ETLS

The ETL Java heap size is insufficient.

Increase the ETL Java heap size. For details, see the following knowledge articles:

(Support logon ID required)

The Moviri ETL fails to start and the ETL execution is interrupted immediately. The following message is reported in the ETL log:

FAILED;BCO_ETL_FAIL108: Process cannot be started (cannot fork external java/os process) or process fails due to severe errors (no modules in classpath)
java.lang.ClassNotFoundException: com.moviri.etlfactory.MovDateUtil

After upgrading the system or applying a patch, the Moviri class jar files are not available at the following path: $CPITBASE/etl/libmodules/$MODULENAME Example: For the Entuity ETL, EntuityME could be missing.

Confirm that the etlFactoryLibrary-1.X.0.jar is available at the following path:

$CPITBASE/etl/libmodules/$MODULENAME

Redeploy the ETL package or copy the content of libmodules directory from another working system.

The Microsoft SCOM ETL fails with the following error in the ETL log:

FAILED BCO_ETL_ERR011: Detected an abnormal ETL termination. Reason: java.sql.SQLException: Invalid object name 'dbo.vManagedEntity'.

The management pack is missing from the system.

Enable or disable the metric classes in the ETL configuration.

For details, see knowledge article 000269586 (Support logon ID required).

The Splunk ETL fails with the following error in the ETL log:

Warning: [STDERR] - java.sql.SQLRecoverableException: IO Error: Connection reset

An issue occurs with Java during connection to Oracle 11 via the standard JDBC Oracle library. This issue affects the secure communication with the database server.

Do the following steps:

  1. On the Linux system where the ETL Engine is running, navigate to the following directory:
    /opt/bmc/BCO/$CPITBASE
  2. Create a file named customenv.sh.
  3. Add the following line to the file:
    ETL_JAVA_OPTS="$ETL_JAVA_OPTS -Djava.security.egd=file:/dev/./urandom"
    export ETL_JAVA_OPTS
  4. Save the changes.

The Splunk ETL fails with the following error in the ETL log:

FAILED BCO_ETL_ERR011: Detected an abnormal ETL termination. Reason: java.lang.RuntimeException: Received fatal alert: protocol_version

The TLS version that is configured for the ETL is not supported.

Make sure that the ETL is configured to use TLSv1.2. 

For details, see knowledge article 000233511(Support logon ID required).

The HP NNMi ETL fails with the following error in the ETL log:

FAILED BCO_ETL_ERR011: Detected an abnormal ETL termination. Reason: jcifs.smb.SmbAuthException: Logon failure: unknown user name or bad password.

The specified credentials are incorrect or the Windows share path for the ETL to connect to is not reachable.

Verify and update the credentials and permissions on the Windows share directory.

The following message is logged in the Kubernetes ETL log:

WARN [service-###-1]- No batch processed since last save.

The ETL Engine and the Heapster are not communicating over the defined connector port.

To validate the connection, run the following command on the ETL Engine Server where the Kubernetes ETL is scheduled:

sink=doc:xwiki:IT-Operations-Management.Continuous-Optimization.BMC-Helix-Continuous-Optimization.bhco252.Troubleshooting.Troubleshooting-ETLs.Troubleshooting-the-Moviri-ETLs.opentsdb:http://<Helix_Continuous_Optimization
_Server_Hostname>:<Helix_Continuous_Optimization_Kubernetes_
Connector_Port?cluster=<Name_of_the_Kubernetes
_Cluster>

where

Helix_Continuous_Optimization_Server_Hostname is the host name (or IP address) of the  Application Server (or ETL engine) where the Kubernetes connector is deployed.

Helix_Continuous_Optimization_Kubernetes_
Connector_Port is the TCP port on which the Kubernetes connector is listening. This must be set to the same value as the TSCO listening port in the ETL configuration page.

Name_of_the_Kubernetes_Cluster is the string representing the cluster name.

The following message is logged in the Kubernetes ETL log:

BCO_TASK_FAIL005: Critical error detected during task 74 execution: java.lang.OutOfMemoryError: GC overhead limit exceeded

The Kubernetes ETL is not collecting data and the execution is interrupted because the ETL Java heap size is insufficient.

Increase the ETL Java heap size. For details, see the following topics:

(Support logon ID required)

The Kubernetes ETL is not collecting data and the following error message is logged:

Unable to start server. Permission denied

The configuration parameter that the BMC Helix Continuous Optimization Installation Owner can bind to as a LISTEN port must be the same value as TSCO listening port (since only the root user can bind to ports lower than 1024).

On the ETL configuration page, change the value of TSCO listening port from a reserved port (1024 or lower) to an arbitrary port number that is greater than 1024 and is not currently in use on the  ETL Engine.

The following message is logged in the Kubernetes ETL log:

ERROR [service-393-1]- Unable to start serverAddress already in use

The listening port (default is 4242) is already in use by a service or a process or another Kubernetes ETL.

On the ETL Engine Server where the Kubernetes ETL is scheduled, run the following command:

netstat -an | grep <listening_port>

where <listening_port> is the port of the Kubernetes ETL configuration.

Check the command output to verify that the port is already in use. Request a new port from your server administrator.

The following message is logged in the Kubernetes ETL log:

INFO  [service-115-1]- User username cannot list nodes at the cluster scope: User username cannot list all nodes in the cluster

The account added in the ETL configuration does not have access to obtain the necessary data from the Kubernetes API.

To configure the account, follow the steps listed under the Kubernetes API section in the following topic:

Moviri-K8s-Kubernetes-Prometheus-Extractor

The Cloudera ETL is not collecting data. The following message is logged in the Cloudera ETL log for each entity in the environment:

[2017-09-07 00:00:22] INFO Cloudera Extractor: Extracting HOST_PERF data for host hostname.domain.com
[2017-09-07 00:00:22] INFO Cloudera Extractor: Extracting data for [HOST_PERF:hostname.domain.com] since YYYYMM-
DD HH24:MI:SS Z
[2017-09-07 00:00:22] INFO Cloudera Extractor: empty time period, skipping query.

The default last counter value is not correctly set in the ETL configuration.

Confirm and update the last counter value. The value might be in the "YYYY-MM-DD HH24:MI:SS Z" format. It should be set to an actual timestamp value, such as "2017-09-11 00:00:00".

The Cloudera ETL fails with BCO_ETL_ERR011 error or an error similar to the following message:

[2019-11-20 10:08:58] INFO [SQLite Wrapper] Database file /opt/cpit/BCO/etl/temp/237-data/TO1LWWJ1.db
correctly deleted
[2019-11-20 10:08:58] WARNING [SQLite Wrapper] Database file delete failed since there is no such file or directory

This is issue is known with the Cloudera ETL.

Deploy the latest version of the Cloudera ETL.

 

Tip: For faster searching, add an asterisk to the end of your partial query. Example: cert*