This documentation supports the 19.02 version of Remedy with Smart IT.

To view the latest version, select the version from the Product version menu.

Troubleshooting cloned Smart IT environment issues

When you try to access the Smart IT application from the browser, you might see some errors or face issues with logs. Use this information to understand and resolve the problem or to create a BMC Support case.


Symptoms

  • The Smart IT Universal Client displays a blank page (404 or 500 error).
  • The following error is seen in the Tomcat catalina.<date>.log file:

    Example

    <date> SEVERE [localhost-startStop-1] org.apache.catalina.core.StandardContext.startInternal Context [/smartit] startup failed due to previous errors

  • The Smart IT server does not generate any logs in the Logs folder.
  • Only a bootstrap.log file is generated in the <installDirectory>\Smart_IT\Logs folder. None of the other expected Smart IT logs are created or updated.
  • References to incorrect Smart IT hostnames are displayed in the Centralized configuration or in the AR System Configuration Component Setting form..
  • The following log snippet is displayed in smartit.log upon restarting the Smart IT Tomcat service:

    Example

    10:13:14.864 | localhost-startStop-1 | RID: none | DEBUG | c.b.a.m.c.i.MessagingClientImpl | Connecting to broker at: null with username Remedy Application Service
    10:13:14.911 | localhost-startStop-1 | RID: none | ERROR | c.b.b.m.s.c.CCSPropertiesService | Error while initializing CCSPropertiesService
    java.lang.NullPointerException: null

Scope

  • The new environment is unavailable for all users.

Resolution

When you face issues in a production system, you might want to clone that environment to a separate test system to try and troubleshoot the issues there.

Perform the following tasks to troubleshoot the cloned environment issues:

TaskActionStepsReference
1

If Smart IT is unavailable and displays a 404 or 500 error, acquire a fresh set of logs for Tomcat and Smart IT.

Instructions:

  • Stop the Smart IT service and go to the Logs folder. Remove all logs in the following folder:
    <InstallDirectory>/SmartIT/logs/
  • Delete all existing logs from the following Tomcat folder:
    <TomcatInstallDirectory>\logs
2

In environments where Remedy Single Sign-On exists, temporarily disable the integration with Smart IT. This action will affect only the Smart IT application, Remedy Single Sign-On will continue to function in Remedy Mid Tier and other applications.


To disable the integration of BMC Remedy Single Sign-On with Smart IT, run the following queries against the Smart IT database. You might need to adjust the Smart IT system schema name to match that of your system.

Oracle

SQL> update SMARTIT_SYSTEM.TENANT set SAML_AUTHENTICATION=0;

MS SQL

update SmartIT.SmartIT_System.TENANT set SAML_AUTHENTICATION=0;
3

Enable the Smart IT logs in Debug mode and then restart Smart IT.



To enable the Smart IT logs, see Knowledge Article 000158941.

The following video describes how to enable the logs in the Debug mode in Smart IT..

Knowledge Article 000158941

4

Review the logs.


After restarting the application, look for errors in the smartit-metadata.log file.

Example:

12:15:56.741 | localhost-startStop-1          | ERROR   | c.b.b.m.c.ConfigurationManager | Fail to load list of configurations from AR. Cause - ERROR (90): Cannot establish a network connection to the AR System server; Connection timed out: connect <server>:<port>

Open the catalina.<date>.log file and review any errors that might occur when the application restarts.

This issue occurs due to communication problem with the Remedy AR System server.

  • Check the AR System server or Load Balancer references in the SMT:Administration Console form.
  • Review the smartit.xml file and validate if the database references of this cloned environment are correct.
5

Check the AR Server and load balancer references in Smart IT and the database.

The SMT:Administration Console form is used to specify the AR System server used in the environment. When you clone the environment, this form must be updated.

If you are using...Then access this form from...
A fresh installation of Smart IT (18.05 and later versions)the object list or directly via the URL
An upgrade to Smart IT version 18.x and later from Smart IT version 2.0 and earlierthe object list or directly via the URL
Smart IT 2.0 and earlier versionsthe Applications menu and the form name is MyIT:Administration Console instead.


  1. Ensure that the Data Store field value on the form has a valid AR System server entry. When cloning environments, you must replace the name of the original AR System server with the cloned AR System server.
    For more information, see the following video:

  2. Ensure that the Persistent Data Store section contains the correct AR System server host name and password for the new cloned environments.
  3. In the Persistent Data Store section, ensure that you update old AR System server entries or old AR load balancer information.
  4. If you have a Remedy Single Sign-On integration, ensure that the Data Store points to the AR System load balancer or an AR System server where Remedy Single Sign-On is integrated. The purpose is to have authentication done through the AR System load balancer. All AR System servers that are a part of this load balancer must be integrated with Remedy Single Sign-On in order to successfully authenticate.
  5. Update the values in the CONFIGURATION_PARAMETER table in the Smart IT database as shown in the video.
  6. After making these changes, restart Smart IT.
  • Using the AR System Object List
  • Knowledge Article: 000144726
  • Changing the AR Reference in Smart IT
6

Check the AR Server or load balancer references in the Smart IT database.

Verify if the tables in Smart IT database contain the right references for the AR System server or the load balancer.

SmartIT_System.CONFIGURATION_PARAMETER table contains the hostname, port, and Application Service password in the Smart IT SQL database.

In the SmartIT_System.PROVIDER_SETTINGS table, check for host name references in the Smart IT SQL database.

  1. Go to the CONFIGURATION_PARAMETER table within the SmartIT_System schema and run the following query:
    SELECT * FROM [SmartIT].[SmartIT_System].[CONFIGURATION_PARAMETER]
  2. Run the following query to update the host name:
    UPDATE [SmartIT].[SmartIT_System].[CONFIGURATION_PARAMETER] SET Value = 'Server_Name' Where Name='connect.arsystem.hostName'
  3. If the password has changed, you must encrypt the password by using Smart IT Maintenance Tool located at:
    <SmartIT InstallDirectory>/SmartIT/SmartITMaintTool/SmartIT Maintenance Tool/
  4. Run the following query to update password information:
    UPDATE [SmartIT].[SmartIT_System].[CONFIGURATION_PARAMETER] SET Value = 'encrypted password' Where Name='connect.arsystem.password'
  5. Run the following query to update port information:
    UPDATE [SmartIT].[SmartIT_System].[CONFIGURATION_PARAMETER] SET Value = '0' Where Name='connect.arsystem.port'

Note: If you are using an Oracle database, use the COMMIT statement to make the changes permanent.


7

Review the references in the smartit.xml file.

In localhost.log or catalina.log, you might find errors that indicate failed login attempts to the database. You can troubleshoot this by reviewing the smartit.xml file.

The smartit.xml file located at <SmartIT_Tomcat_Install_Dir>\conf\Catalina\localhost contains references to the JDBC connection to the SmartIT_System and SmartIT_Business schemas.

The following example is from a system connecting to MS SQL Server database: url=jdbc:jtds:sqlserver://mydbserver-011:1433/SmartIT

Notes:

  • There are two such references to the JDBC connection. The database user name and passwords for this connection are also present in this file.
  • If you are using Smart IT with SSL, add suitable SSL certificates to a keystore file that should be referenced within the Smart IT Tomcat server.xml file.

Knowledge article: 000137427


8

Check the new Smart IT cloned server references in the AR System Configuration Component Setting form.

On the AR System Configuration Component Setting form, check record for the current Smart IT server hostname.

If this record belongs to the old environment and not the cloned environment, then this data must be corrected. Restart the AR System server and Smart IT server, and check if the data gets corrected.

Uninstall Smart IT and drop the schemas. Reinstall Smart IT.

9

Check if invalid AR System servers are listed in the Centralized configuration

  • If invalid AR System servers are listed in the Centralized configuration, delete the corresponding entries of invalid AR System servers on the AR System Configuration Component form.
  • In Centralized configuration, delete any references to old Smart IT servers.
  • If you still face issues after deleting incorrect AR System and Smart IT server references, uninstall Smart IT from the cloned server, remove the Smart IT specific schemas from the database, and reinstall Smart IT.

Knowledge article: 000097124


10

Create a BMC Support Case.

When creating a case with BMC Support, collect available Smart IT logs and Tomcat logs.

Provide the following information as part of your case:

  • Any error messages received or observed.
  • Attach the zip file to your case.
  • You can attach a zip file up to 2 GB.

    For more information, see SFT - Steps to send logs, files, screenshots to BMC Support for a Remedy Product related case.

After you determine a specific symptom or error message, use the following table to identify the solution:

SymptomActionSolution

Smart IT displays one or both these errors:

  • 404 or 500 error
  • Error while initializing CCSPropertiesService in the Bootstrap log, and in catalina or localhost logs
  • Check the Server-Name parameter in the ar.cfg file. If this parameter is set to <ar_server_name>, change it to <load_balancer_name> in the ar.cfg file on all AR System servers.
  • Delete invalid Smart IT servers present in the Centralized configuration.
  • Delete invalid AR System server entries in the Centralized configuration.
  • Correct the Server-Name parameter to contain the name of the load balancer.
  • For information about how to clean up this data, see knowledge article 000097124.


The Openfire chat icon is not visible after cloning the environment.

Check the Openfire chat parameters. The connect.properties file in the <SmartIT_Tomcat_Install_Dir>\external-conf directory contains references to the MyIT server and Openfire connection details:

myit.server.host = myserver-011
chat.server.cm.host = myserver-011
...
...
chat.server.boshUrl = http://myserver-011:7070/http-bind/
chat.server.domain = myserver-011

  • Update <serverURL> to point to the Openfire instances of the correct SQL database instance for the cloned environment.
  • In Openfire Console >Server Manager >System Properties, check the following parameters and update the values, if required:
    • xmpp.domain: The value in this parameter must match the chat.server.cm.host parameter in the connect.properties file.

    • provider.auth.authResource: Update this value to indicate the new hostname or Load Balancer for the cloned Smart IT instances.

    • admin.authorisedJIDs: Update this value to match the cloned Openfire xmpp.domain.

  • If you are using Openfire with SSL:
    • Add suitable SSL certificates on the TLS/SSL Certificates tab.
    • Add the new Openfire SSL certificates for the cloned system in the cacerts file within the JRE /lib/security directory for the JRE used by the cloned Smart IT instances.
  • If you are using the Hazelcast plug-in for Openfire clustering, update the member entries in the following folder to match the cloned Openfire instances:
    <Openfire_install_dir>\openfire\plugins\hazelcast\classes\hazelcast-cache-config.xml

The following errors are shown:

  • Smart IT 404 error
  • Catalina or localhost.log snippet

    Caused by: java.security.UnrecoverableKeyException: Password verification failed
    ... 27 more
    java.sql.SQLException: ORA-01017: invalid username/password; logon denied.
    Jan 11, 2016 11:07:57 AM org.apache.catalina.core.
    NamingContextListener addResource

    WARNING: Failed to register in JMX: javax.naming.NamingException: ORA-01017: invalid username/password; logon denied

  • Collect the Smart IT logs and the catalina or localhost.log.
  • Restart the Tomcat service.
  • Verify if the Smart IT database passwords are correct.
  • Check if there are any invalid .xml files in this location:
    C:\Program Files\Apache Software Foundation\Tomcat8.5\conf\Catalina\localhost
    Only the smartit.xml file must be present in this folder. If there are other files, remove them or if they are needed, ensure that they have a different suffix.
  • Open the smartit.xml file and verify if the database references and password details are correct.
  • Check if you can ping the AR System database from the Smart IT server.


Was this page helpful? Yes No Submitting... Thank you

Comments