Troubleshooting Kubernetes ETL
To collect ETL logs at log level 10
To add detailed activity logs to the log file:
- In the BMC Helix Capacity Optimization console, navigate to Administration > ETL & system tasks > ETL tasks and select the configured Kubernetes ETL.
- In the Run configurations section, click
. - Make a note of the current value of the Log level setting.
- Change the log level to 10 - Verbose and save the run configuration.
- After you run the ETL and the collect the logs, set the log level back to the previous value to avoid filesystem space issues.
To determine the Kubernetes ETL package version
- In the BMC Helix Capacity Optimizationconsole, navigate to the Administration > System > Maintenance.
- In the Additional Packages tab, search for the Kubernetes ETL and make a note of the Execution date, Component version, Installed version, and Status.
This information will help you in troubleshooting issues with Kubernetes ETL.
Resolutions for common issues
Symptom | Cause | Resolution |
|---|---|---|
The following message is reported in the Kubernetes ETL log: WARN [service-###-1]- No batch processed since last save. | There is no communication between the ETL Engine and the Heapster on the defined connector port. | To validate the connection, run the following command on the ETL Engine Server where the Kubernetes ETL is scheduled: sink=opentsdb:http://<Helix_Capacity_Optimization_Server_Hostname>:<Helix_Capacity_Optimization_Kubernetes_Connector_Port?cluster=<Name_of_the_Kubernetes_Cluster> where, Helix_Capacity_Optimization_Server_Hostname is the hostname (or IP address) of the Helix Capacity Optimization application server (or ETL engine) where the Kubernetes connector is deployed. Helix_Capacity_Optimization_Kubernetes_Connector_Port is the TCP port on which the Kubernetes connector is listening. This must be set to the same value as the TSCO listening port in the ETL configuration page. Name_of_the_Kubernetes_Cluster is the string representing the cluster name. |
The following message is reported in the Kubernetes ETL log: BCO_TASK_FAIL005: Critical error detected during task 74 execution: java.lang.OutOfMemoryError: GC overhead limit exceeded | The Kubernetes ETL is not collecting data and the execution is interrupted because the ETL Java heap size is insufficient. | |
The Kubernetes ETL is not collecting data and the execution ends in ERROR. Unable to start server. Permission denied | The configuration parameter that the Helix Capacity Optimization Installation Owner can bind to as a LISTEN port must be the same value as TSCO listening port (since only the root user can bind to ports lower than 1024). | In the ETL configuration page, change the value of TSCO listening port from a reserved port (1024 or lower) to an arbitrary port number that is greater than 1024 and is not currently in use on the Helix Capacity Optimization ETL Engine. |
The following message is reported in the Kubernetes ETL log: ERROR [service-393-1]- Unable to start serverAddress already in use
| There is already a service or process (or another Kubernetes ETL) using the LISTEN port (default is 4242). | On the ETL Engine Server where the Kubernetes ETL is scheduled, run the following command: netstat -an | grep <listening_port> where, <listening_port> is the port of the Kubernetes ETL configuration. This will confirm that the port is already in use. Request your server administrator for a new port. |
The following message is reported in the Kubernetes ETL log: INFO [service-115-1]- User username cannot list nodes at the cluster scope: User username cannot list all nodes in the cluster | The account added in the ETL configuration does not have access to obtain necessary data from the Kubernetes API. | To configure the account, follow the steps listed in the Kubernetes API section in the Moviri-Integrator-for-BMC-Helix-Capacity-Optimization-k8s-Heapster topic. |