Troubleshooting Smart IT availability or stability issues

The Smart IT application depends on multiple components for its service to be available. In case of a sudden disruption of the Smart IT service, you must isolate the Smart IT component that is causing the disruption, and then take appropriate action.

Use the information in this topic to identify the problem and restore service as quickly as possible, or to gather relevant information and logs to create a case with BMC Customer Support.


Issue symptoms

The following table lists the error messages or issues that you might encounter:

For this error message or issue...

Perform this task...

Smart IT Universal Client does not load. You might observe the following error messages or issues in the browser:


    • Unable to connect
    • This site can’t be reached - <host.domain.com> refused to connect

    • An error occurred. Please contact your administrator or retry later.

AnError_RetryLater_RSSO_Crop.png


    • The RSSO server is not responding or is not available. Please contact your administrator.

RSSOServerNotResponding.png


    • Fully blank browser window (no spinning wheel)

    • HTTP Status 404 – Not found

1908_HTTP404_ContextDown.png

The Smart IT login screen is displayed but after entering credentials, one of the following issues occurs:


    • An error has occurred please contact your administrator

1908_NoRSSO_AnErrorHasOccurred.png


    • Error message below the Username and Password fields despite entering valid credentials:
      Incorrect username or password

    • Blank browser window with a spinning wheel:

1908_RSSO_SpinningWheelOnly2.png


    • Spinning wheel below Smart IT login screen:

NoRSSO_ARBusyOnLogin_Spinning_Crop.png

The application freezes on the Remedy Single Sign-On login page.

Smart IT Universal Client performance is severely affected, making the application unusable. Sections of the Smart IT interface display spinning wheels as shown in the following Smart IT dashboard screen:

UIThere_SpinningMultiple.png

Smart IT screens display partial or random error messages.

OddErrors_OnJavaHeap_p4_small.png

Issue scope

  • Environments that were running are suddenly affected.
  • All or most users are affected.
  • Problem may be intermittent in case of environments using Smart IT Clusters or AR Server groups with load balancing.

For environments where the Smart IT service is unavailable following a scheduled change (for example, upgraded or cloned environments, change of Remedy Action Request System password, etc.), see Troubleshooting-cloned-Smart-IT-environment-issues.


Diagnosing and reporting an issue

Important

  • For any situation where a Root Cause Analysis may be required, make sure to back up any logs before deleting them. This task provides further details and logs that might be useful when performing this analysis.
  • For server groups or clusters, when you are troubleshooting outage situations, taking additional Smart IT and AR servers temporarily out of the load balancer (either by configuration, or by shutting those servers down) can simplify the troubleshooting process. After the system is available again on a reduced server setup, the other servers can be brought up one by one—either in addition to the now known good server, or by testing the remaining servers one by one.


Perform the following tasks to troubleshoot the issues:

Task

Action

Steps

Reference

1

Make sure that the Smart IT Tomcat service is running and listening on the expected port.

Confirm that there are no network issues or firewalls that could cause connection failure between the client (browser) and Smart IT Server.

Important: In case of a Smart IT environment that is integrated with Remedy Single Sign-On, confirm if the URL that fails to load in the browser is the Smart IT URL. In case the failing URL relates to Remedy Single Sign-On, see Troubleshooting-Remedy-Single-Sign-On-integration-with-Smart-IT.


  • Check if the Tomcat process for Smart IT Server is running.
    • (Windows) Smart IT Tomcat Service must be in the Started or Running state. You can find this service under Windows Services or Tomcat process in the Windows Task Manager.
    • (Linux) Enter the following command at the command line:
      ps -ef | grep tomcat

If Tomcat is not running: Start the Tomcat service and check if Smart IT becomes available. If Tomcat fails to start, confirm if there has been any recent changes for Java, OS patches, or antivirus software on the server.

Back up and clear the following logs:

  • <tomcat>\logs\*.log
  • ..Smart_IT\Logs\bootstrap.log

Enable Smart IT debug logging (if not enabled yet) and then restart the service.

Check all newly created logs for any errors. Also, see this task.
If Tomcat is running, use the following commands from the command line to check if Tomcat is listening on the port as defined in ..<tomcat>\conf\server.xml:

  • Windows: netstat -nao | findstr <tomcat_port>
  • Linux: netstat -lnp | grep <tomcat_port>

  • Check the localhost_access_log.YYYY-MM-DD.txt file in Tomcat's logs-folder to see if you can find the expected incoming requests being captured. KB Article 000375659 provides more details on the format of this log.

  • Check if the Smart IT URL is accessible from the Smart IT Server itself (using a browser on the server, or e.g. the 'curl' command on Linux). If so, next steps may be to confirm:

      • If the host in the Smart IT URL can be pinged from the client machine.
      • If any changes were made to network, firewall or load balancer settings that could be preventing the connection.
      • In case of a load balancer, check if this can be bypassed to directly connect to a specific Smart IT Server from the client machine.

See the Resolution for common issues section in this topic for possible errors and corrective actions.

2

These errors are triggered by the RSSO Agent on the Smart IT Server and are therefore displayed only on environments that have the Remedy SSO integration enabled for Smart IT.

Check the RSSO Agent log file on the Smart IT Server for details on what triggered the error and the next steps to take.

  • Confirm that the Remedy SSO Service is available by either logging in to the Remedy SSO Admin console or in to another application that is integrated with the same Remedy SSO service.
  • If multiple applications were restarted, ensure that all applications were started in the correct order: Remedy SSOServer, AR Server, Mid Tier, and Smart IT.
  • Check the rsso-agent.0.log file in Tomcat logs folder for SEVERE type errors.
  • Validate the setup of the Smart IT Remedy SSO integration and :
    • Check the contents of rsso-agent.properties and sso-sdk.properties in these folders on the Smart IT Servers:
      • <tomcat>\external-conf
      • ..\Smart_IT\smartit\WEB-INF\classes

Important: During the start-up of Smart IT Server, it will sync these .properties files between these two folders. The file with the most recent modified date is present in both folders.


    • Check if the required JAR files are present in the <tomcat>\external-conf\lib folder on the Smart IT Servers. If all files are present, an extra check can be done to compare their file sizes against a working environment that is on the same product version.
      For more information, see Integrating-BMC-Remedy-SSO-with-Smart-IT.
  • Confirm that the SSO Service URL (configured for the sso-service-url parameter in '<tomcat>\external-conf\rsso-agent.properties') can be reached from the Smart IT Servers. Commands that could help with this are ping, telnet, and curl.
    For example, on Linux:
  • curl -vk <sso-service-url>
  • curl -vk http://rssohost.domain:port/rsso

  • To check if Smart IT works without Remedy SSO, temporarily disable the Remedy SSOintegration for Smart IT as described in the Disabling RSSO for Smart IT and Digital Workplace section of KB Article 000246642.
    Important: Restart of Tomcat service is required on the Smart IT Server both to disable and re-enable the integration.

  • If required, debug logging can also be enabled for the RSSO Agent by adding the following Java Option for Smart IT Tomcat Server and then restarting:
    -Drsso.log.level=FINEST

    • (Windows) From Tomcat's bin-folder, double-click the file with a name similar to TomcatServiceNamew.exe, go to the Java tab, and add the option in the Java Options section. Apply and then restart Tomcat.
    • (Linux) Add the following line under the existing JAVA_OPTS variable and then restart Tomcat:
      export JAVA_OPTS="$JAVA_OPTS -Drsso.log.level=FINEST"

3

If you see only a generic error in the browser window, review the browser network logs (F12). Review the response data for the REST calls to see if a detailed error message is captured, which might help isolate the cause of the problem.

  • Check if a detailed error message for the Smart IT REST calls can be found in the network logs of the browser developer tools.
    1. Close all open browser windows, and then open a new private browser window.
    2. Press F12 to open the developer tools and navigate to the Network tab.
      For more information about how to perform this step for common browser types, see Capturing network details by using the browser functionality.
    3. Navigate to the Smart IT URL to reproduce this issue. You should see the network traffic being captured in the developer tools section.
    4. Once the issue is reproduced, check for the HTTP status codes in the network log that indicate a failure:
      1. If Openfire Chat is enabled, ignore any failures that might be present for calls related to chat (which would include 'http-bind' in their URL).
        For more information, see Troubleshooting Openfire Chat issues.
      2. Specifically, look for HTTP Status error codes: 500 and 403
        Important        : HTTP 404 errors on style-smart-it.css can be ignored.
        Examples of REST calls that most often show such HTTP error codes during service disruptions:

/smartit/rest/serverstates

/smartit/rest/users/sessions (for Remedy SSO integrated environments)

/smartit/rest/users/sessions/<loginID> (for non-Remedy SSOintegrated environments)

  • If you see a failing REST call, click the request to check its response details. The following screen is an example of a sessions REST call on a Remedy SSO integrated environment that failed with HTTP 500, and the error message is captured in the response data:

DevTools_HTTP500_details.png

If a more detailed error is present in the response data:


    • Proceed with these steps if the detailMessage section in the response displays the following error on a Remedy SSO integrated environment:
      ERROR (623): Authentication failed; Incorrect username or password
    • For other non-generic errors captured in the response data, see the Resolution for common issues section in this topic.
  • If the network log shows a failure or an incomplete REST call, but the error captured in the response data is either a generic error or does not match a known issue, then note the URL of that REST call and perform a more detailed review of the logs on the Smart IT Server, as described in Review the Tomcat and Smart IT logs.
  • In case the network logs do not show any indications of a possible failure, navigate to the Console tab of the developer tools to see if any error is present there.
  • If no issues are identified in either the Network or Console tab of the F12 logs, continue with these steps.

See the Resolution for common issues section in this topic for possible errors and corrective actions.

4

On a Remedy SSO integrated Smart IT environment, if you see a spinning wheel and ERROR (623) in the browser network logs, the user was successfully authenticated against the identity provider (IdP) configured in Remedy SSO and has received an SSO token. However, authorization fails when Smart IT attempts to connect to AR Server (configured as the Persistent Data Store for Smart IT) with the Login ID and SSO token.

  • The rsso-agent.0.log in Tomcat logs folder on the Smart IT server shows a successful authentication for the user as shown in the following example:

<date time> INFO Thread_471 com.bmc.rsso.agent.RSSOAgent.process(): User 'LoginId' is authenticated, token '_xxxxxxxx-*****-xxxxxxxxxxxx'

  • If multiple applications were restarted, ensure that the applications were started in the correct order: RSSO Server, AR Server, Mid Tier, and Smart IT. Also, make sure that the application has started before restarting the next one.

  • Confirm the host name that is configured as Persistent Data Store for Smart IT (either via Mid Tier as described in step 3 of KB Article 000219036 or via SQL against the AR System Database as described in KB Article 000375711). The AR host configured here must be integrated with Remedy SSO. If this host name is pointing to a Load Balancer, all AR Servers that Smart IT can be directed to via this Load Balancer must be integrated with Remedy SSO.

  • Enable the API and User log category in separate files on these AR Servers to establish which servers trigger the ERROR 623 on the ARVerifyUser API call from Smart IT. Reproduce a failed login attempt and locate the API call in the API logs:

<API > .... <USER: LoginId > .... */ +VER ARVerifyUser -- user LoginId from Palm Pilot (protocol 26) at IP address ....
<API > .... <USER: LoginId > .... */ -VER FAIL -- AR Error(623) Authentication failed : LoginId


    • If you cannot find the API call captured in the API logs of any of the identified AR Servers, reconfirm where the AR host name configured as Smart IT's Persistent Data Store could point to.
    • If you can find the API call in the logs, but the captured Login Id is not a match for the Login Name in AR System's User form (this is case sensitive), see the resources in the next column.
      Important: If the AR Server allows the unregistered Login ID to connect as guest user, you may not always see the API call itself failing with a 623 error in this log. However, in that case, the 623 error will be triggered on Smart IT Server side, as the Smart IT code rejects such guest logins. The AR Server User log can confirm if the user was connected as guest user, and you would see a line similar to this:

/* LOGIN LoginId (guest)(session Id xxxx)


    • If you can find the API call in the API logs, the captured Login ID is an exact match for the Login Name in the User form, and the call is failing with the described 623 error, then proceed with the following steps:

      • Enable debug logging for the default Java Plug-in Server that should run the RSSO Plug-in. For more information, see How to enable DEBUG Java Plugin Server Logging in AR System in the arjavaplugin.log file.
        Reproduce the failed login and check arjavaplugin.log on the AR Servers. With such debug logging enabled, you should see log lines that refer to activity of the ARSYS.AREA.RSSO plug-in for the login attempt on one of the servers. Check these entries for possible errors to find the root cause.
        If no entries related to the ARSYS.AREA.RSSO plug-in are visible with debug logging enabled, restart the Java Plugin Servers. For more information, see How to restart the Java Plugin Server
        After the plug-in restart, check if there are any ERROR messages indicating that the RSSO plug-in failed to load:

*/ An error has occurred while loading a plug-in. Plugin: ARSYS.AREA.RSSO

Or if instances of the RSSO plug-in are successfully created, e.g.:

*/ Thread xx successfully created an instance of ARSYS.AREA.RSSOIf the RSSO Plug-in is loading successfully, try logging in to Smart IT again. If the problem persists, checkarjavaplugin.log for any detailed error messages when the RSSO Plug-in handled the login.



      • Validate the setup of the AR System Server andRemedy SSO integration and restart the applications in the correct order: Remedy SSO Server, AR Server, and Smart IT. Each application should finish its start-up before the next application is started. For more information, see Manually integrating Remedy SSO with BMC applications
      • If the problem persists, remove and re-apply AR Server's integration with Remedy SSO. For more information, see resources in the Reference column.

Use the following steps to review in case the User ID presented to AR Server by Smart IT does not match the Login Name stored in AR System's User form:

  • If @domain.com is appended to the Login ID in the AR logs, and this should not be there, check in the Remedy SSO Admin Console if a tenant is configured for the realm that Smart IT uses for authentication. Such configuration can be seen in environments where BMC Digital Workplace Catalog uses the same realm to authenticate. For more information, see Configuring BMC Remedy Single Sign-On integration for BMC Digital Workplace Catalog.
    In this case, configure the ignore-tenant parameter on the Smart IT Server. For more information, see To configure Smart IT not to require the tenant domain section in Integrating Remedy SSO with Smart IT.
  • If the Login ID captured in the logs is different from what the user entered on login (for example, in a different case), and this is not as specified AR System, check in the Remedy SSO Admin Console if any user ID transformation is configured in the realm that Smart IT uses for authentication. If such transformation is not defined, it might still be required as the user name coming from the identity provider does not match the login ID in AR System. For information about transforming user ID, see Transforming User ID to match Login ID.

See the Resolution for common issues section in this topic for possible errors and corrective actions.

For more information, see the following resources:

5

When all sections of the Smart IT interface show spinning wheels, start by checking the health of the AR Servers configured as Smart IT's persistent data store as well as its related AR System database.

Next, check for any excessive CPU or memory usage on the Smart IT Servers and the Smart IT database.

If no apparent issues are found in these major components, continue with a log review to determine the root cause of the performance issue.



If you observe performance issues on AR Server side, seeTroubleshooting process for performance issues for more information. If the AR Server performance is affected by API calls originating from the Smart IT server, see Resolution for common issues section in this topic for possible errors and corrective actions, or review the information that you can submit to BMC Support for further review.

  • Check for excessive CPU or memory usage on the Smart IT Server and Smart IT Database machines. If this is the case, restart services to see if this action restores Smart IT. If excessive CPU usage on the Smart IT Server is a recurring issue, create Java Thread Dumps for further analysis by BMC Support.
    For the next steps, logs, and information to be provided to BMC Support with the case, see this knowledge article.

See the Resolution for common issues section in this topic for possible errors and corrective actions.

6

Review the main Tomcat and Smart IT logs for errors or exceptions that might be causing service disruption.

If an error is identified, confirm if it is clear from the error message what action to take. If not, see the Resolution for common issues section in this topic for possible errors and corrective actions or create a case with BMC Support for further assistance.

  • Confirm that sufficient free disk space is available on the Smart IT Servers.
  • While the existing logs on the Smart IT Server can be a starting point for the review, creating a fresh set of logs to help identify the initial point of failure is more efficient (especially if the issue started at a time where debug logging was not enabled). To create such a fresh set of logs:

    • Enable Smart IT debug logging (if not enabled yet).

    • Stop the Smart IT Tomcat Service.
    • Back up and delete the log files from these folders (leave any subfolders):

<tomcat>\logs
..\Smart_IT\Logs


    • Start the Tomcat service again.
      • If the problem persists after Tomcat has been restarted, check the newly created Tomcat and Smart IT log files for errors.
      • If the Smart IT Service becomes available following the restart, review the backed-up logs to identify the root cause of the Service Disruption. This task provides more information on details that can help with a Root Cause Analysis when raising a case with BMC Support.
  • Check the main Tomcat and Smart IT log files for errors—use the timestamps in the logs to check for the first error that is recorded (either around the time the issue started, or the first overall error in case of a fresh set of logs).
    Important: Often, multiple errors are shown in a log at (almost) the same timestamp. Consider these error messages together when reviewing the Resolution for common issues section in this topic.
    For both the Tomcat and Smart IT logs, regular expressions are provided that can help during the initial review of logs to locate relevant errors.
  • The following logs are the main Tomcat logs to review, and are present in this location:<tomcat>\logs

catalina.<current_date>.log
localhost.<current_date>.log

Regular expression to help with the initial review of Tomcat logs:

SEVERE|jdbc.*Exception|Exception.*jdbc|OutOfMemoryError|Startup failed|ERROR \([0-9]{0,4}\)
  • The following logs are the main Smart IT logs to review, and are present in this location: ..\Smart_IT\Logs

bootstrap.log
smartit-metadata.log
smartit.log

Regular expression to help with the initial review of Smart IT logs:

\| ERROR   \||errorCode|ERROR \([0-9]{0,4}\)|jdbc.*Exception|Exception.*jdbc|heap space|connection problem

  • The smartit.log file captures details of the handling of the individual REST calls. Each call can be traced with a unique 'RID' number in this log.
    Example:
    <date_time> | <threadId>     | RID: 26 | ERROR  | ....

If you see an error for a specific RID, and DEBUG logging is enabled, the last entries for that RID includes information about the REST call for which the error was reported (and the HTTP status code) to match this with what is seen in the F12 .har logs:

<date_time> | <threadId>     | RID: 26 | DEBUG   | c.b.b.myit.filter.ExpiresFilter | Request '/smartit/rest/users/sessions' with response status '500' content-type ...

  • Tomcat localhost_access_log.<date>.txt log captures information on the requests Tomcat has finished handling. KB Article 000375659 provides details on the format of this log (as well as logging options).
    This log enables you to perform these actions:

    • Check if a specific REST call was handled by this server
    • Search using regular expressions to check for requests with a HTTP Status code in 400-599 (error) range:
      HTTP.*" [4-5][0-9][0-9]

    • If processing time and thread ID logging is enabled for the Access Log, a regular expression could be used to help identify requests.
      For example, the following expression identifies requests that took over 10 seconds to complete:
      [0-9]{5,10} msThe thread ID that handled the request can then also be matched with the thread information captured in the Smart IT logs.
  • Once an error is found in the logs, you perform the following actions:
    • Take corrective action in case the root cause is clear from the error message
    • Check the Resolution for common issues section in this topic for possible errors and corrective actions. (or if the found error can actually be ignored. If so, move on to the next error in the logs).

    • Search for any articles in the BMC Support Knowledge Base that provide more information about the identified error message

    • Log a case with BMC Support for further assistance. Use these steps to provide further information that can be useful when submitting a case.

Possible options to perform searches that use regular expressions:


    • The following screenshot shows how to perform such a search by using an advanced text editor like Notepad++

RegExp_NPP_2.png

Remember to switch the search option back to the Normal mode for plain text searches.


    • Windows PowerShell:

Select-String filename.log -Pattern 'regular_expression'


    • Linux shell - e.g.:

grep -E 'regular_expression' filename.log

See the Resolution for common issues section in this topic for possible errors and corrective actions.

7

Collect data that can help BMC Support analyze the root cause of Smart IT service disruption.

Collect and provide the following information (as applicable) from the environment to help BMC Support analyze the root cause of Smart IT service disruption

  • Date and time when the issue was first observed. If intermittent, state the frequency or any pattern observed.
  • Symptoms observed during outage (such as error messages in user interface, screenshots)
  • The actions taken to restore service
  • High-level overview of environment's architecture
  • Any known recent changes on the environment (such as Java version change, OS patches, antivirus software)

  • F12 Browser network log and Console-tab logs when reproducing the issue. Save the network log in .har format, and the information on the Console tab as a .txt file.
    For information about how to capture logs from different browsers, see Capturing network details by using the browser functionality

  • From the Smart IT Servers in the environment, provide the following information:
    • High CPU or RAM utilization observed for Tomcat process during outage
    • ZIP all log files from the following locations covering the full timeframe of the issue:

'<tomcat>\logs' folder
'..\SmartIT\Logs folder'

Important: Due to rollover of the smartit.log, back-ups of that log that are relevant to the issue might have been moved to a subfolder with name format YYYY-MM-weekX


    • In case the issue relates to Remedy SSO, include the following files:

<tomcat>\external-conf\rsso-agent.properties
<tomcat>\external-conf\sso-sdk.properties

  • From the AR Servers in the environment, provide the following information:
    • High CPU or RAM utilization observed on AR Server machine during outage
    • High (CPU) load or blocking queries observed by the DBA for the AR Server database
    • ZIP the following log and configuration files from all AR Servers, covering the full timeframe of the issue:
      • ar.cfg/ar.conf
      • arerror.log
      • arexception.log
      • armonitor.log
      • arjavaplugin.log
      • API/SQL/Filter logging


Resolutions for common issues

Symptom

Action

Reference

List of common errors that can be ignored while troubleshooting Smart IT outage situations:

File: <tomcat>\logs\catalina.<date>.log

.. SEVERE [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [smartit] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@683d467e]) and a value of type [com.bmc.arsys.apitransport.session.ApiThreadLocalStorageBlock] (value [com.bmc.arsys.apitransport.session.ApiThreadLocalStorageBlock@4cb33167]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

.. SEVERE [http-nio-9000-exec-5] org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse An I/O error has occurred while writing a response message entity to the container output stream.


File: <tomcat>\logs\localhost.<DATE>.log

.. SEVERE [localhost-startStop-1] org.apache.tomcat.util.descriptor.web.SecurityConstraint.findUncoveredHttpMethods For security constraints with URL pattern [/*] only the HTTP methods [TRACE OPTIONS] are covered. All other methods are uncovered.

File: ..\Smart_IT\Logs\smartit.log

.. | ERROR | c.b.b.m.s.r.SmartReportingService | Error generating URL for cross launch
.. | ERROR | c.b.b.m.s.p.a.APNSServiceConfig | Failed iOS certificate verification
.. | ERROR | c.b.b.m.s.n.NotificationService | Notification service could not be initialized for tenant 000000000000001

File: ..\Smart_IT\Logs\smartit-metadata.log

.. | WARN | c.b.b.m.s.ARMetaDataService | Fail to load fields from data store <arServer>. Cause - ERROR (303): Form does not exist on server; SHR:UnionOverview_MasterConsole
.. | WARN | c.b.b.m.s.ARMetaDataService | Fail to load views which are available for schema SHR:UnionOverview_MasterConsole. Cause - ERROR (303): Form does not exist on server; SHR:UnionOverview_MasterConsole
.. | WARN | c.b.b.m.s.ARMetaDataService | Fail to load fields from data store <arServer>. Cause - ERROR (303): Form does not exist on server; SB:ServiceRequestStub
.. | WARN | c.b.b.m.s.ARMetaDataService | Fail to load views which are available for schema SB:ServiceRequestStub. Cause - ERROR (303): Form does not exist on server; SB:ServiceRequestStub
.. | WARN | c.b.b.m.s.ARMetaDataService | Fail to load fields from data store <arServer>. Cause - ERROR (303): Form does not exist on server; CTM:Support Group Assoc LookUp People Join
.. | WARN | c.b.b.m.s.ARMetaDataService | Fail to load views which are available for schema CTM:Support Group Assoc LookUp People Join. Cause - ERROR (303): Form does not exist on server; CTM:Support Group Assoc LookUp People Join


F12 Network log (.har)

HTTP 404 error for the URL: http(s)://<smartITServer:Port>/tenant-custom-res/000000000000001/style-smart-it.css

No action required. The errors listed in this row can be ignored while troubleshooting Smart IT outage situations.


File: <tomcat>\logs\commons-daemon.<date>.log

[date_time] [error] [ 8024] Failed creating Java 'C:\Program Files\Java\jdk-12.0.2\bin\server\jvm.dll'.
[date_time] [error] [ 8024] The system cannot find the path specified.

Specify the correct JVM path for Tomcat.

File: <tomcat>\logs\catalina.<date>.log

.. SEVERE [localhost-startStop-1] org.apache.tomcat.jdbc.pool.ConnectionPool.init Unable to create initial connections of pool.

[Detailed_SQL_Exception_>>_Examples]

.. SEVERE [localhost-startStop-1] org.apache.catalina.core.StandardContext.startInternal One or more listeners failed to start. Full details will be found in the appropriate container log file
.. SEVERE [localhost-startStop-1] org.apache.catalina.core.StandardContext.startInternal Context [/smartit] startup failed due to previous errors

Examples of detailed SQL Exceptions:

com.microsoft.sqlserver.jdbc.SQLServerException: The TCP/IP connection to the host <smartITDBHost>, port <smartITDBPort> has failed.
com.microsoft.sqlserver.jdbc.SQLServerException: Cannot open database "<smartITDBName>" requested by the login. The login failed.
com.microsoft.sqlserver.jdbc.SQLServerException: Login failed for user '<smartITSystemUserName|smartITBusinessUserName>'
java.sql.SQLException: ORA-01017: invalid username/password; logon denied

Files: <tomcat>\logs\localhost.<DATE>.log and ..\Smart_IT\Logs\bootstrap.log

org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'persistenceUnitManager' defined in class path resource [app-context-jpa.xml]
..
Caused by: javax.naming.NameNotFoundException: Name [<resourceName>] is not bound in this Context. Unable to find [jdbc].

This error indicates that the connection to the Smart IT Database for the SmartIT_System and SmartIT_Business resource is failing. The database connection details are stored in the following XML file on the Smart IT Server:

..<tomcat>\conf\Catalina\localhost\smartit.xml

Take corrective steps based on the detailed SQL error message captured in the catalina log.

Parameters of interest in the smartit.xml file (Each parameter is present twice, once for the SmartIT_System resource and once for the SmartIT_Business resource):

  • username (username used to connect to the Smart IT database)
  • password (plain text password, do not encrypt)
  • url (contains Smart IT Database host, port and database details)

Confirm that the Smart IT database can be reached from the Smart IT Server and (have a DBA) check the configured credentials directly against the database.


File: <tomcat>\logs\localhost.<DATE>.log

.. SEVERE [localhost-startStop-1] org.apache.catalina.core.StandardContext.listenerStart Exception sending context initialized event to listener instance of class [org.springframework.web.context.ContextLoaderListener]
..
..
Internal Exception: com.microsoft.sqlserver.jdbc.SQLServerException: Invalid object name 'CONFIGURATION_PARAMETER'.

OR

File: ..\Smart_IT\Logs\smartit.log

.. | ERROR | c.b.b.m.s.s.i.<className> | DB could not be indexed
..
Internal Exception: com.microsoft.sqlserver.jdbc.SQLServerException: Invalid object name 'SUGGESTED'.

These errors suggests Smart IT was able to connect to the database, but an expected table is not found. Check in the Smart IT Database if a table with the indicated object name exists. The diagram below shows the table names and their schema for a default installation.

SmartIT_OOTB_Tables.png

If the table exists, check the connection details in this file on the Smart IT Server:

..<tomcat>\conf\Catalina\localhost\smartit.xml

  • Check if the 'url' parameter is pointing to the expected database.
  • Check if the usernames configured for the SmartIT_Business and SmartIT_System resources are set up with the expected default schema.

For information about pre-creating the Smart IT database for the expected user setup, see Creating-the-database.

File: <tomcat>\logs\rsso-agent.0.log

javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
 sun.security.ssl.Alert.createSSLException(Unknown Source)

Correct the certificate setup.

File: ..\Smart_IT\Logs\bootstrap.log

.. | ERROR | c.b.a.c.c.n.i.NotificationControllerImpl | GSI call failed for message broker 
com.bmc.arsys.api.ARException: ERROR (623): Authentication failed; Remedy Application Service

or

..| ERROR | c.b.b.m.s.c.CCSPropertiesService | Error while checking application upgrade status.
com.bmc.arsys.api.ARException: ERROR (623): Authentication failed; Remedy Application Service

or

.. | DEBUG | c.b.b.m.s.ZDTAppsUpgradeService | Error while fetching the Version from SHARE:Application_Properties
com.bmc.arsys.api.ARException: ERROR (623): Authentication failed; Remedy Application Service

The Remedy Application Service password defined in CONFIGURATION_PARAMETER table (Smart IT System schema) is incorrect. Update the password.


Important: The password must be stored in the table in an encrypted format (plain text will not work). In versions 18.0x and 19.0x, use the Smart IT Maintenance Tool to encrypt the password.

For Smart IT 20.02 and higher, use the Password Encryption utility.

File: ..\Smart_IT\Logs\bootstrap.log

.. | ERROR | c.b.b.m.s.c.CCSPropertiesService | Error while initializing CCSPropertiesService
javax.crypto.IllegalBlockSizeException: Input length must be multiple of 16 when decrypting with padded cipher
 at com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:913)

The Remedy Application Service password defined in the CONFIGURATION_PARAMETER table in the Smart IT System schema is not properly encrypted. Encrypt the password.

File: ..\Smart_IT\Logs\bootstrap.log

.. | DEBUG | c.b.b.m.s.ZDTAppsUpgradeService | Error while fetching the Version from SHARE:Application_Properties
com.bmc.arsys.api.ARException: ERROR (103): The length of the name parameter (or name field in a parameter) is longer than the maximum allowed length.
..
.. | ERROR | c.b.b.m.s.ZDTAppsUpgradeService | Error while checking application upgrade status.
java.lang.NullPointerException: null

The Remedy Application Service password defined in the CONFIGURATION_PARAMETER table in the Smart IT System schema is not properly encrypted. Encrypt the password.

This error can be seen on Smart IT 20.02 when the password is encrypted with the Maintenance Tool of a lower version, while the new 20.02 password encryption utility should be used.

File: ..\Smart_IT\Logs\smartit-metadata.log

.. | ERROR | c.b.b.m.c.ConfigurationManager | Error loading ITSM Version:

com.bmc.arsys.api.ARException: ERROR (623): Authentication failed; Remedy Application Service
 at com.bmc.arsys.apitransport.ApiProxyJRpcBase.arCall(ApiProxyJRpcBase.java:300)

The Remedy Application Service password defined in the CONFIGURATION_PARAMETER table in the Smart IT System schema might be entered in plain text. Encrypt the password and update it in the table.

File: ..\Smart_IT\Logs\bootstrap.log

.. | ERROR | c.b.a.c.c.n.i.NotificationControllerImpl | GSI call failed for message broker 
com.bmc.arsys.api.ARException: ERROR (90): Cannot establish a network connection to the AR System server; Connection refused: connect <arServer>:<port>

or

| ERROR | c.b.b.m.s.c.CCSPropertiesService | Error while checking application upgrade status.
com.bmc.arsys.api.ARException: ERROR (90): Cannot establish a network connection to the AR System server; Connection refused: connect <arServer>:<port>

or

| DEBUG | c.b.b.m.s.ZDTAppsUpgradeService | Error while fetching the Version from SHARE:Application_Properties
com.bmc.arsys.api.ARException: ERROR (90): Cannot establish a network connection to the AR System server; Connection refused: connect <arServer>:<port>

Smart IT is unable to connect to the AR Server with the connection information stored in the CONFIGURATION_PARAMETER table in the Smart IT System schema.

Verify if the AR port defined in this table is correct and that Smart IT can connect to AR Server on the specified port.

File: ..\Smart_IT\Logs\bootstrap.log

org.xml.sax.SAXParseException: schema_reference.4: Failed to read schema document 'http://www.springframework.org/schema/context/spring-context-3.0.xsd', because 1) could not find the document; 2) the document could not be read; 3) the root element of the document is not <xsd:schema>.
 at org.apache.xerces.util.ErrorHandlerWrapper.createSAXParseException(Unknown Source)


File: <tomcat>\logs\localhost.<current_date>.log

.. SEVERE [localhost-startStop-1] org.apache.catalina.core.StandardContext.filterStart Exception starting filter [Agent]
 java.lang.NullPointerException
 at com.bmc.atrium.sso.agents.web.SSOFilter.init(SSOFilter.java:47)

File: ..\Smart_IT\Logs\bootstrap.log

.. | ERROR | c.b.bsm.myit.util.FilesCopyUtil | File sso-sdk.properties from /apps/ars/Smart_IT/Smart_IT/smartit/WEB-INF/classes/sso-sdk.properties not copied to /apps/apache/tomcat8.5/external-conf properly, try to restart application server and verify properties files
java.io.FileNotFoundException: /apps/ars/Smart_IT/Smart_IT/smartit/WEB-INF/classes/sso-sdk.properties (Permission denied)

Add the correct file system permission on the Smart IT Server.

File: <tomcat>\logs\localhost.<current_date>.log

.. SEVERE [main] org.apache.catalina.core.StandardContext.filterStart Exception starting filter [Agent]
java.lang.NoClassDefFoundError: io/jsonwebtoken/ExpiredJwtException

Important: The example error is related to a class from jjwt-api-<version>.jar. The actual class name referenced in the error might be different depending on the .jar file that is affected. General format of the error is:

.. SEVERE [main] org.apache.catalina.core.StandardContext.filterStart Exception starting filter [Agent]
java.lang.NoClassDefFoundError: <classname>

Check the presence (and size) of the required .jar files for the RSSO Agent in the <tomcat>\external-conf\lib folder on the Smart IT Server.

F12 Network log (.har)

{"error":"MOBILITY_ERROR_LOGIN","errorCode":1006,"defaultMessage":"Unable to check user status","additionalMessage":"ServiceException: Unable to check user status","detailedMessage":"Unable to check user status","arconnectionProblem":false}

File: ..\Smart_IT\Logs\smartit.log

.. | ERROR | c.b.b.m.service.UsersServiceImpl | Unable to check user status
com.bmc.bsm.myit.providers.ServiceNotSupportedException: com.bmc.bsm.myit.providers.userdetails.UserDetailsService

Confirm if the User Details Pluggable provider is enabled and the .jar file for this provider (name: itsm-user-details-<version>-BUILD-SNAPSHOT.jar) is present in the ..\Smart_IT\smartit\WEB-INF\classes\provider\itsm-user-details folder.

F12 Network log (.har)

{"error":"MOBILITY_ERROR_LOGIN","errorCode":1006,"defaultMessage":"Login Failed: An authentication error occurred in the data server.","additionalMessage":"Your password has expired. Please, login to AR server and change it","detailMessage":"ERROR (623): Authentication failed; Password has expired","ARConnectionProblem":false}

Open the record of the affected user in AR System User form, and check if the Force Password Change On Login field is set. Have the user change their password to clear this field.

F12 Network log (.har):

{"error":"MOBILITY_ERROR_INTERNAL_SERVER_ERROR","errorCode":1013,"defaultMessage":"Missing People Info.","additionalMessage":"UnauthorizedException: Missing People Info.","detailedMessage":"HTTP 403 Forbidden","arconnectionProblem":false}

File: ..\Smart_IT\Logs\smartit.log

..| ERROR | c.b.b.m.assembler.EntryAssembler | Failed to fill(queryParameters=[{name=loginId, value=<LoginID>}], user:<LoginID>, host: <arServer>)

com.bmc.arsys.api.ARException: ERROR (623): Authentication failed; Remedy Application Service

Validate the connection settings for the MyIT-ITSM Persistent Data Store of Smart IT. Make sure that they are correct and that there are no connectivity issues between Smart IT and AR System.

The specific error here suggests that the Remedy Application Service password stored in the Persistent Data Store of Smart IT is incorrect.

Important: This error can also be seen in the network logs when attempting to log in to Smart IT with a user that has no people record (for instance, if you attempt to log in with Remedy's classic 'Demo' administrator user). Make sure the user ID used to log in has been set up with the correct Smart IT Roles and Permissions.

Long running queries against the SMT:Social_WorklogView form together with high CPU usage on the AR System database.

Check the resources in the Reference column for resolution.


 

Tip: For faster searching, add an asterisk to the end of your partial query. Example: cert*