Troubleshooting cloned Smart IT environment issues

When you try to access the Smart IT application from the browser, you might see some errors or face issues with logs. Use this information to understand and resolve the problem or to create a BMC Support case.

Symptoms

The Smart IT Universal Client displays a blank page (404 or 500 error).
The following error is seen in the Tomcat catalina.<date>.log file:
Example
<date> SEVERE [localhost-startStop-1] org.apache.catalina.core.StandardContext.startInternal Context [/smartit] startup failed due to previous errors
The Smart IT server does not generate any logs in the Logs folder.
Only a bootstrap.log file is generated in the <installDirectory>\Smart_IT\Logs folder. None of the other expected Smart IT logs are created or updated.
References to incorrect Smart IT hostnames are displayed in the Centralized configuration or in the AR System Configuration Component Setting form.
The following log snippet is displayed in smartit.log upon restarting the Smart IT Tomcat service:
Example
10:13:14.864 | localhost-startStop-1 | RID: none | DEBUG | c.b.a.m.c.i.MessagingClientImpl | Connecting to broker at: null with username Remedy Application Service
10:13:14.911 | localhost-startStop-1 | RID: none | ERROR | c.b.b.m.s.c.CCSPropertiesService | Error while initializing CCSPropertiesService
java.lang.NullPointerException: null

Scope

The new environment is unavailable for all users.

Resolution

When you face issues in a production system, you might want to clone that environment to a separate test system to try and troubleshoot the issues there.

Perform the following tasks to troubleshoot the cloned environment issues:

Task

Action

Steps

Reference

If Smart IT is unavailable and displays a 404 or 500 error, acquire a fresh set of logs for Tomcat and Smart IT.

Instructions:

Stop the Smart IT service and go to the Logs folder. Remove all logs in the following folder:
<InstallDirectory>/SmartIT/logs/
Delete all existing logs from the following Tomcat folder:
<TomcatInstallDirectory>\logs

In environments where Remedy Single Sign-On exists, temporarily disable the integration with Smart IT. This action will affect only the Smart IT application, Remedy Single Sign-On will continue to function in Remedy Mid Tier and other applications.

To disable the integration of Remedy Single Sign-On with Smart IT, run the following queries against the Smart IT database. You might need to adjust the Smart IT system schema name to match that of your system.

Oracle

SQL> update SMARTIT_SYSTEM.TENANT set SAML_AUTHENTICATION=0;

MS SQL

update SmartIT.SmartIT_System.TENANT set SAML_AUTHENTICATION=0;

Integrating BMC Helix Single Sign-On with Smart IT

Enable the Smart IT logs in Debug mode and then restart Smart IT.

To enable the Smart IT logs, see Knowledge Article 000158941.

The following video describes how to enable the logs in the Debug mode in Smart IT..

Knowledge Article 000158941

Review the logs.

After restarting the application, look for errors in the smartit-metadata.log file.

Example:

12:15:56.741 | localhost-startStop-1 | ERROR | c.b.b.m.c.ConfigurationManager | Fail to load list of configurations from AR. Cause - ERROR (90): Cannot establish a network connection to the AR System server; Connection timed out: connect <server>:<port>

Open the catalina.<date>.log file and review any errors that might occur when the application restarts.

This issue occurs due to communication problem with the Remedy AR System server.

Check the AR System server or Load Balancer references in the SMT:Administration Console form.
Review the smartit.xml file and validate if the database references of this cloned environment are correct.

Check the AR Server and load balancer references in Smart IT and the database.

The SMT:Administration Console form is used to specify the AR System server used in the environment. When you clone the environment, this form must be updated.

If you are using...	Then access this form from...
A fresh installation of Smart IT (18.05 and later versions)	the object list or directly via the URL
An upgrade to Smart IT version 18.x and later from Smart IT version 2.0 and earlier	the object list or directly via the URL
Smart IT 2.0 and earlier versions	the Applications menu and the form name is MyIT:Administration Console instead.

Ensure that the Data Store field value on the form has a valid AR System server entry. When cloning environments, you must replace the name of the original AR System server with the cloned AR System server.
For more information, see the following video:
Ensure that the Persistent Data Store section contains the correct AR System server host name and password for the new cloned environments.
In the Persistent Data Store section, ensure that you update old AR System server entries or old AR load balancer information.
If you have a Remedy Single Sign-On integration, ensure that the Data Store points to the AR System load balancer or an AR System server where Remedy Single Sign-On is integrated. The purpose is to have authentication done through the AR System load balancer. All AR System servers that are a part of this load balancer must be integrated with Remedy Single Sign-On in order to successfully authenticate.
Update the values in the CONFIGURATION_PARAMETER table in the Smart IT database as shown in the video.
After making these changes, restart Smart IT.

Excerpt named Using_the_AR_System_Object_List was not found in document xwiki:Service-Management.IT-Service-Management.Remedy-with-Smart-IT.smartit2002._inclusionsLibrary_smartit._smartit_2NW_LinksLibrary.WebHome.
Knowledge Article: 000144726
Changing the AR Reference in Smart IT

Check the AR Server or load balancer references in the Smart IT database.

Verify if the tables in Smart IT database contain the right references for the AR System server or the load balancer.

SmartIT_System.CONFIGURATION_PARAMETER table contains the hostname, port, and Application Service password in the Smart IT SQL database.

In the SmartIT_System.PROVIDER_SETTINGS table, check for host name references in the Smart IT SQL database.

Go to the CONFIGURATION_PARAMETER table within the SmartIT_System schema and run the following query:
SELECT * FROM [SmartIT].[SmartIT_System].[CONFIGURATION_PARAMETER]
Run the following query to update the host name:
UPDATE [SmartIT].[SmartIT_System].[CONFIGURATION_PARAMETER] SET Value = 'Server_Name' Where Name='connect.arsystem.hostName'
If the password has changed, you must encrypt the password by using password-encryption.zip utility located at:
SmartITCustomizationUtil folder folder. Check the readme.txt file more details on the steps to run the utility.
Run the following query to update password information:
UPDATE [SmartIT].[SmartIT_System].[CONFIGURATION_PARAMETER] SET Value = 'encrypted password' Where Name='connect.arsystem.password'
Run the following query to update port information:
UPDATE [SmartIT].[SmartIT_System].[CONFIGURATION_PARAMETER] SET Value = '0' Where Name='connect.arsystem.port'

Note: If you are using an Oracle database, use the COMMIT statement to make the changes permanent.

Knowledge article: 000130901
Knowledge article: 000159701

Review the references in the smartit.xml file.

In localhost.log or catalina.log, you might find errors that indicate failed login attempts to the database. You can troubleshoot this by reviewing the smartit.xml file.

The smartit.xml file located at <SmartIT_Tomcat_Install_Dir>\conf\Catalina\localhost contains references to the JDBC connection to the SmartIT_System and SmartIT_Business schemas.

The following example is from a system connecting to MS SQL Server database: url=jdbc:jtds:sqlserver://mydbserver-011:1433/SmartIT

Notes:

There are two such references to the JDBC connection. The database user name and passwords for this connection are also present in this file.
If you are using Smart IT with SSL, add suitable SSL certificates to a keystore file that should be referenced within the Smart IT Tomcat server.xml file.

Knowledge article: 000137427

Check the new Smart IT cloned server references in the AR System Configuration Component Setting form.

On the AR System Configuration Component Setting form, check record for the current Smart IT server hostname.

If this record belongs to the old environment and not the cloned environment, then this data must be corrected. Restart the AR System server and Smart IT server, and check if the data gets corrected.

Uninstall Smart IT and drop the schemas. Reinstall Smart IT.

Check if invalid AR System servers are listed in the Centralized configuration

If invalid AR System servers are listed in the Centralized configuration, delete the corresponding entries of invalid AR System servers on the AR System Configuration Component form.
In Centralized configuration, delete any references to old Smart IT servers.
If the references are still not valid, uninstall Smart IT from the cloned server, remove the Smart IT specific schemas from the database, and reinstall Smart IT.

Knowledge article: 000097124

Create a BMC Support Case.

When creating a case with BMC Support, collect available Smart IT logs and Tomcat logs.

Provide the following information as part of your case:

Any error messages received or observed.
Attach the zip file to your case.
You can attach a zip file up to 2 GB.
For more information, see SFT - Steps to send logs, files, screenshots to BMC Support for a Remedy Product related case.

After you determine a specific symptom or error message, use the following table to identify the solution:

Symptom	Action	Solution
Smart IT displays one or both these errors: 404 or 500 error Error while initializing CCSPropertiesService in the Bootstrap log, and in catalina or localhost logs	Check the Server-Name parameter in the ar.cfg file. If this parameter is set to <ar_server_name>, change it to <load_balancer_name> in the ar.cfg file on all AR System servers. Delete invalid Smart IT servers present in the Centralized configuration. Delete invalid AR System server entries in the Centralized configuration.	Correct the Server-Name parameter to contain the name of the load balancer. For information about how to clean up this data, see knowledge article 000097124.
The Openfire chat icon is not visible after cloning the environment.	Check the Openfire chat parameters. The connect.properties file in the *<SmartIT_Tomcat_Install_Dir>\external-conf* directory contains references to the MyIT server and Openfire connection details: myit.server.host = myserver-011 chat.server.cm.host = myserver-011 ... ... chat.server.boshUrl = http://myserver-011:7070/http-bind/ chat.server.domain = myserver-011	Update <serverURL> to point to the Openfire instances of the correct SQL database instance for the cloned environment. In Openfire Console >Server Manager >System Properties, check the following parameters and update the values, if required: xmpp.domain: The value in this parameter must match the chat.server.cm.host parameter in the connect.properties file. provider.auth.authResource: Update this value to indicate the new hostname or Load Balancer for the cloned Smart IT instances. admin.authorisedJIDs: Update this value to match the cloned Openfire xmpp.domain. If you are using Openfire with SSL: Add suitable SSL certificates on the TLS/SSL Certificates tab. Add the new Openfire SSL certificates for the cloned system in the cacerts file within the JRE /lib/security directory for the JRE used by the cloned Smart IT instances. If you are using the Hazelcast plug-in for Openfire clustering, update the member entries in the following folder to match the cloned Openfire instances: <Openfire_install_dir>\openfire\plugins\hazelcast\classes\hazelcast-cache-config.xml
The following errors are shown: Smart IT 404 error Catalina or localhost.log snippet Caused by: java.security.UnrecoverableKeyException: Password verification failed ... 27 more java.sql.SQLException: ORA-01017: invalid username/password; logon denied. Jan 11, 2016 11:07:57 AM org.apache.catalina.core. NamingContextListener addResource WARNING: Failed to register in JMX: javax.naming.NamingException: ORA-01017: invalid username/password; logon denied	Collect the Smart IT logs and the catalina or localhost.log. Restart the Tomcat service. Verify if the Smart IT database passwords are correct.	Check if there are any invalid .xml files in this location: C:\Program Files\Apache Software Foundation\Tomcat8.5\conf\Catalina\localhost Only the smartit.xml file must be present in this folder. If there are other files, remove them or if they are needed, ensure that they have a different suffix. Open the smartit.xml file and verify if the database references and password details are correct. Check if you can ping the AR System database from the Smart IT server.

Troubleshooting cloned Smart IT environment issues

Symptoms

Scope

Resolution

On this page