Troubleshooting installation and upgrade of the Application Server and the ETL Engine

This section provides information about troubleshooting issues that occur during an installation or upgrade of the Application Server and the ETL Engine.

To access the Knowledge Articles listed in this topic, Support logon ID is required.

You can also create a BMC support case for further assistance.

Verify Remedy SSO and TrueSight Presentation Server status

Before you install or upgrade the Application Server and the ETL Engine, ensure that Remedy Single Sign-On and TrueSight Presentation Server are running successfully.

Remedy Single Sign on (RSSO)

Verify the Remedy SSO status using the Root or installation user.
- (Linux) Navigate to the <RSSO_Installation_directory>/rsso/bin directory, and run the ./rsso server status command.
- (Windows) Navigate to the <RSSO_Installation_directory>\rsso\bin directory, and run rsso.bat server status command.
  A list of RSSO processes are displayed as shown in the following example :
Process   status
=====================================
Database Server      Running (14706)
Tomcat Server        Running (14708)
Verify the access to the Remedy SSO URL using the following format in a browser:
http(s)://<rsso_servername_FQDN>:<tomcat_port>/rsso
(Optional) For any warnings or errors, review the log files located in the following Remedy SSO directories:
- (Linux) <RSSO_Installation_directory>/rsso/logs and <RSSO_Installation_directory>/rsso/tomcat/logs
- (Windows) <RSSO_Installation_directory>\rsso\logs and <RSSO_Installation_directory>\rsso\tomcat\logs

TrueSight Presentation Server (TSPS)

Verify the TrueSight Presentation Server status using the Root or installation user :
- (Linux) Navigate to the <TSPS_installation_directory>/truesightpserver/bin directory, and run the ./tssh server status command.
- (Windows) Navigate to the <TSPS_installation_directory>\truesightpserver\bin directory, and run the tssh server status command.
A list of the Presentation Server processes are displayed as shown in the following example:
Process status
================================
Core Server Running (13176)
Index Server Running (11028)
Database Server Running (6052)
Verify the access to Presentation Server URL using the following format in a browser:
- (Windows) http(s)://<TSPS_FQDN>
- (Linux) http(s)://<TSPS_FQDN>:<tomcat_port>
(Optional) For any warnings or errors, review the log files located in the following Presentation Server directories:
- (Linux) <TSPS_Installation_directory>truesightpserver/logs
- (Windows) <TSPS_Installation_directory>\truesightpserver\log

Collect relevant logs for the Application Server and the ETL Engine

For any warnings or errors encountered while installing or upgrading the Application Server and the ETL Engine, review the following log files:

For issues during the installation, review the bco_install_log.txt file located in the /tmp directory.
For issues during and after the upgrade, review the bco_install_log.txt file. You can also use the Log Grabber utility to collect these logs.

Running Log Grabber using the UI:

Log in to the TrueSight Capacity Optimization console as a user with Administrator privileges.
Navigate to Administration > SYSTEM > Maintenance and select the Maintenance Activities tab.
For Log Grabber, click the Execute Log Grabber icon.

Running Log Grabber using the command line:

Log in to the host computer on which you have installed the Application Server.
At the shell prompt, log in as the user that you specified during installation (example: cpit).
Navigate to the <base_direcory>/tools directory (example: /opt/cpit/tools).
Run the logGrabber.sh script. The.tar.gz archive file is created in the same directory as the script and includes the logs of all Capacity Optimization services installed on the local computer.

For details on logGrabber, see the knowledge article number 000244652.

Common issues

This section describes some common issues that you might encounter with installation or upgrade of the Application Server and the ETL Engine, and provides suggested resolutions for the issues.

Installation issues

Issue	Cause	Resolution
Installer fails with the following error even after the DISPLAY variable is set in the terminal console: java.lang.NoClassDefFoundError: Could not initialize class java.awt.Toolkit at java.awt.Font.<clinit>(Unknown Source) at com.bmc.install.product.base.project.runner.ProjectRunner.getChoosableLocales(ProjectRunner.java:1321) at com.bmc.install.product.base.project.runner.ProjectRunner.initializeProjectUI(ProjectRunner.java:856) at com.bmc.install.product.base.project.runner.ProjectRunner.runProjectRunner(ProjectRunner.java:1085) at com.bmc.install.product.base.project.runner.ProjectRunner.main(ProjectRunner.java:310 .....	This can be caused by one of the following issues: Missing X-Windows server library packages Invalid DISPLAY variable Firewall issues X-windows authentication problems	Install the required X-Windows server library package Set the valid DISPLAY variable Use ssh X-forwarding to send the DISPLAY through a firewall For details, see the knowledge article number 000307856.
Installer fails with the following error: BCO_INST_ERR001 : Failed to connect to Database using User : <user>. Please check Database connection details provided .	This can be caused by one of the following issues: Incorrect database credentials Incorrect port or service name	Test the connection to the database. For details, see the knowledge article number 000239897.

Upgrade issues

Issue	Cause	Resolution
Unable to log into the Capacity Optimization console with the Admin user, after upgrading the Application Server from 10.x to 11.x version. The following error is displayed on the browser login page : BCO-WEB-FAIL105- TSPS server not responding probably the server is unreachable or the configuration is incorrect	Caused due to the partial customization with the Presentation Server configuration file, that is, server.xml.	Add the missing ciphers list in the server.xml file and restart Capacity Optimization and Presentation Server services. For details, see the knowledge article number 000304892.
While upgrading the Application Server from 10.7.x to 11.3.01 or 11.5 versions, following error is displayed on the terminal console: BCO_INST_FAIL142: Error while performing Database alignment], Detail=[com.neptuny.cpit.util.sql.AlignException: Error during the align process of table 'GLOSS_ETLMOD_LOOKUPINFO': Identified 2 bad lines.	The DB alignment task is unable to update the GLOSS_ETLMOD_LOOKUPINFO table due to problematic ETL module. <> ORA-00001: unique constraint (BCO_OWN.LOSS_ETLMOD_LOOKUPINFO_IDX_1) violated ): <>	For details on resolving this issue, see the knowledge article number 000355860.
During the upgrade, setup.sh X-Windows UI fails to start and displays the following error: java.lang.NullPointerException	OS fonts package not installed on the server.	Verify whether the fonts package is installed on the server. For details, see the knowledge article number 000367472.
After the upgrade, unable to add Capacity Optimization component to the TrueSight console if the Presentation Server is deployed in the High Availability mode. The following error is displayed: Error "Failed to add component"	To support TSPS HA functionality every component class should be marshallable. Capacity Optimization component was not marshallable. Error in the Presentation Server logs: InvocationContextInterceptor ISPN000136: Execution error org.infinispan.commons.marshall.NotSerializableException: Object of type class com.bmc.truesight.platform.etl.component.management.types.COComponent expected to be marshallable	For details on resolving this issue, see the knowledge article number 000370549.
After upgrading the Application Server silently, the Component status checker system task displays error in the Dynservices status. The following error is displayed in the dynservices/log/ dynservices-optimizer-zookeeper.log file using the log grabber output: [New I/O worker #1] INFO o.a.zookeeper.ClientCnxnSocketNetty.operationComplete - channel is connected: [id: 0x665e4161, /127.0.0.1:13172 => MyServer/127.0.1.1:8999] [New I/O worker #1] WARN o.a.zookeeper.ClientCnxnSocketNetty.exceptionCaught - Exception caught: [id: 0x665e4161, /127.0.0.1:13172 => MyServer/127.0.1.1:8999] EXCEPTION: javax.net.ssl.SSLHandshakeException: Certificate signature validation failed javax.net.ssl.SSLHandshakeException: Certificate signature validation failed at java.base/sun.security.ssl.Alert.createSSLException(Unknown Source) at java.base/sun.security.ssl.TransportContext.fatal(Unknown Source) at java.base/sun.security.ssl.TransportContext.fatal(Unknown Source) ....	Caused due to incorrect configuration of the options file in the silent upgrade.	Configure the options file of the silent upgrade properly. For details on resolving this issue, see the knowledge article number 000232629.
After the upgrade, the Database instance seems to be down or not reachable. The following error is displayed: BCO_DB_FAIL103: Exception during select all entities query StackTrace: com.neptuny.cpit.querymanager.QueryException: [com.neptuny.cpit.database.DatabaseConnectionManagerException] - Unable to borrow datasource connection: "Cannot create PoolableConnectionFactory (ORA-01033: ORACLE initialization or shutdown in progress )" - [borrowed=0]	Caused due to database instance or host migration.	Reconfigure Capacity Optimization to update environment parameters such as database host or password, Remedy SSO or Presentation Server information. For details, see the knowledge article number 000204799.
After installing a cumulative hotfix, patch, or service pack in the Capacity Optimization console, the following warning is displayed: Warning : TrueSight Capacity Optimization plugin is not updated with the latest version, please download and install the latest version from" Administration> Maintenance> Patches.	Capacity Optimization plugin was not reinstalled.	After installing a cumulative hotfix, patch, or service pack in the Capacity Optimization console, reinstall the Capacity Optimization plugin. For details, see the knowledge article number 000359189.
Dynservices Optimizer tasks are in error status. The following error is displayed: Error : "Dynservices Optimizer cannot be found or reached. Please check the status of this dynservice on the host Component status was last updated on xx/xx/xxxx " when viewing status of Dynservices under TSCO UI >Administration > System > Status > Dynservices	This can be caused by one of the following issues: Value of the ZOOKEEPER_ADDRESS_LIST property was set incorrectly. Ports 8999 and 9002 were not accessible for Dynservices.	For details on resolving this issue, see the knowledge article number 000366752.