Space banner This documentation supports the 21.05 version of BMC Digital Workplace Advanced.  To view an earlier version, select the version from the Product Version menu.

Troubleshooting BMC Digital Workplace Tomcat

The topic provides information about how to troubleshoot the most common issues related to BMC Digital Workplace Tomcat.

The most common issues are related to Tomcat startup, Tomcat SSL, and updating AR Server References in BMC Digital Workplace Tomcat when the AR Server password has changed.  This topic also describes how to update Tomcat when you Clone and Restore BMC Digital Workplace environments.

BMC Digital Workplace main configuration files

File

Location

Configuration

server.xml

<tomcat8.5/>conf/

The server. xml file is BMC Digital Workplace Tomcat's main configuration file, and is responsible for specifying Tomcat's initial configuration on startup as well as defining the way and order in which Tomcat boots and builds. The file contains configuration details of Tomcat's Connector Ports, Engine (Catalina), and Host name.

connect-dwp.properties

<tomcat8.5/>external-conf
 

The connect-dwp.properties file contains BMC Digital Workplace Tomcat properties for connections.  Out of the box, this file contains the following connection properties:  SMTP Host, cluster configuration, Service Health Subscription Poller.  You can also add new properties to this file as directed by BMC Digital Workplace Engineering.

rsso-agent.properties

<tomcat8.5/>external-conf
 

The rsso-agent.properties file enables you to define the mapping between the BMC Digital Workplace domain and Remedy Single Sign-On server using the following properties: sso-external-url and sso-service-url.

sso-sdk.properties

<tomcat8.5/>external-conf
 

The sso-sdk.properties enables you to define the tenant setting for RSSO.

logback-dwp.xml

<tomcat8.5/>external-conf
 

The logback-dwp.xml file enables you to define the logging level for BMC Digital Workplace specific troubleshooting.

logback.xml

<DWP/DWP/>data-transfer
 

The logback.xml file enables you to define the logging level for data transfer troubleshooting.

hosts file

/etc/hosts 
C:\Windows\ System32 \drivers\etc\

The hosts file enables you to configure DNS resolution when no DNS server is in place. 
 

web.xml

<tomcat8.5/>conf/

The web.xml file defines the default values for all web applications loaded into this instance of Tomcat.

This file enables you to add more security settings as well as enable session persistence on the BMC Digital Workplace application.

keystore

user defined

The keystore file shows the location of BMC Digital Workplace certificates imported by the administrator.

java cacerts

/usr/java/latest/lib/security/cacerts
C:/java/latest/lib/security/cacerts

The Java cacerts file shows the location of your imported BMC Digital Workplace Catalog, RSSO, ITSM, SmartIT, and related certs.

dwp.xml

<tomcat8.5/>/conf/Catalina/localhost

The dwp.xml file provides the BMC Digital Workplace database connection details.

Tomcat Logs

<tomcat8.5/>/logs

The Tomcat Log files record the details of monitored, processed HTTP traffic (such as the URLs, methods, their status codes and time of execution) in the localhost_access_[date].txt file.

BMC Digital Workplace Logs

<BMC Digital Workplace Installation>/DWP/DWP/logs

The BMC Digital Workplace Log files contain information such as errors and requests made by the application to the server.

Starting with BMC Digital Workplace 19.02, as a MyIT Administrator, you can enable different DEBUG Log Levels from BMC Digital Workplace Admin Console > Configuration > Logging. By default, this Log Level is set to WARN or ERROR. You can set it to DEBUG to capture diagnostic information while troubleshooting BMC Digital Workplace application issues.

Important: After you've finished with troubleshooting, revert the Log Level to WARN or ERROR.

image2021-10-14_13-0-11.png

Diagnosing and reporting an issue

After you identify the symptoms and scope of the issue, use this troubleshooting guide to diagnose and resolve the issue.

Issues/Symptoms

Steps

Reference

Tomcat not starting 

Navigate to <tomcat x>/logs directory, and review the following log files. Look for SEVERE and Warning entries to find the root cause of the issue:

  • localhost.log
  • catalina.out 
  • catalina.%date%.log

Review the BMC Digital Workplace log files in the <BMC Digital Workplace Installation>/DWP/DWP/logs directory and look for ERROR entries.

Most of the time the root cause is likely to be:

  • Applications using the same port as Tomcat
  • Database connectivity issues
  • Locks on the BMC Digital Workplace database
  • Errors in the SSL Configuration

Unable to start BMC Digital Workplace/BMC Digital Workplace Catalog after SSL is enabled, due to the error "PKIX path building failed" under BMC Digital Workplace Enhanced Catalog Section


To fix the error "PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target error", follow these steps:

  1. Log in to BMC Digital Workplace Admin console and go to Configuration > Enhanced Catalog. 
  2. Get the cert from BMC Digital Workplace Catalog by using Firefox:

image2021-6-23_16-10-6.png

3. Ensure that all BMC Digital Workplace Catalog cluster nodes have the cert imported to the servers. Also import the certs into the cluster nodes for BMC Digital Workplace, AR Server, SmartIT, and RSSO servers.

4. Import the certificate to the cacerts file by using the following commands:

Note: This can be Java cacerts or a custom cacerts.


    1. openssl x509 -outform der -in catalogcert.pem -out certificate.der
    2. keytool -import -alias DWPCatalog -trustcacerts cacerts -file certificate.der

5. Restart all application Tomcat (BMC Digital Workplace Catalog, BMC Digital Workplace, SmartIT) and AR Server services.

6. Clear the browser cache.

7. Log into the BMC Digital Workplace Admin Console and go to Configuration > Enhanced Catalog. 

8. Update the BMC Digital Workplace Catalog URL with the correct port and URL values.

Note: Repeat for each BMC Digital Workplace Load Balanced URLs by logging into the BMC Digital Workplace Admin console > Configuration > Enhanced Catalog. Update the BMC Digital Workplace Catalog URL with the correct port and URL information from each node.

9. Do the same (update with correct port and URL values) on the AR server's SB:RemoteApprovalConfiguration and Centralized Configuration's com.bmc.itsm.sbe component.

image2021-6-23_16-13-59.png

10. Update the Setting Value with the correct values.

11. Click Save.

12. Confirm that you are not getting the error anymore.

If you are still getting the error, use SSL Poke to verify connectivity. SSL Poke is a third-party tool that enables you to verify that the cacerts file has the correct certificates added to it to connect securely.

  1. Download SSL Poke from any third-party site, and execute it on the BMC Digital Workplace Server. You will need to repeat these steps from AR Server, RSSO, SmartIT, and BMC Digital Workplace Catalog servers.
  2. On a cmd/terminal Windows, run the following commands to execute SSL Poke:


java SSLPoke DWPCatalogServerFQDN SSLPort
Example:
java SSLPoke dwpcatalogservername.com 8443

If the connection is successful, you will get a successful message such as "Connection was successful".

If the connection failed, you will get more details on the failure, such as:

  • SSL Port is blocked
  • Certificate does not exist in the keystore/truststore
  • Certificate is expired. Check validity

You will need to resolve the issues identified by SSL Poke for the connection to be successful. For example:

  • If the error is "SSL Port is blocked", you will need to open the port to allow the connection in the firewall.
  • If the error is "Certificate does not exist in the keystore/truststore", you can check if the certificate exists by running the following command from cmd/terminal Windows:

    keytool -list -v -keystore keystore.jks  

  • If the error is "certificate chain/Root-intermediate cert is missing", work with your CA Security Team to get this issue is rectified.
  • If the error is "Server alias/dns entry for this catalog does not exist in the certificate", run the following command and replace mydomain with dwpcatalogFQDN

    keytool -list -v -keystore keystore.jks -alias mydomain
 

BMC Digital Workplace Advanced/Basic +SSL

https://docs.bmc.com/docs/digitalworkplaceadvanced/2002/configuring-ssl-for-the-tomcat-server-908222629.html


Additional resources:

  • For user group sync issues not working over SSL
    KA 000133376  BMC Digital Workplace Catalog - User Group sync script doesn't work with SSL
  • For AD  over SSL issues
    KA 000164451  BMC Digital Workplace Catalog - Configuring AD Connector on DWP Catalog. Includes AD over SSL

Authentication Failed
occurs during a restart of the BMC Digital Workplace Catalog server, after SSL is enabled.

BMC Digital Workplace Catalog Services/Banners are not visible after SSL is enabled.

BMC Digital Workplace Catalog loops when service is restarted.

Ensure that you follow these requirements when configuring the BMC Digital Workplace Catalog server files:

  • Use the FQDN where the instruction is to use FQDN (and not the host name) in <installLocation>/sb/rxscripts/bin/setenv.sh
  • use the host name where the instruction is to use host name (and not the FQDN) in <installLocation>/sb/env/set_script_variables.sh
  • Do not use localhost in <installLocation>/sb/rxscripts/bin/setenv.sh or <installLocation>/sb/env/set_script_variables.sh
  • Ensure that you update the section below on <installLocation>/jetty/etc/jetty-http.xml with your correct keystore and truststore information:

<New id="sslContextFactory" class="org.eclipse.jetty.util.ssl.SslContextFactory">
   <Set name="KeyStorePath">/opt/bmc/digitalworkplace/certs/keystore</Set>
   <Set name="KeyManagerPassword">OBF:1v2j1uum1xtv1zej1zer1xtn1uvk1v1v</Set>
   <Set name="KeyStorePassword">OBF:1v2j1uum1xtv1zej1zer1xtn1uvk1v1v</Set>
   <Set name="TrustStorePath">/opt/bmc/digitalworkplace/truststore/cacerts</Set>
   <Set name="TrustStorePassword">OBF:1v2j1uum1xtv1zej1zer1xtn1uvk1v1v</Set>
   <Set name="IncludeCipherSuites"

  • Ensure ar.conf has the following values:

Jetty-Port: 443

Jetty-Protocol: HTTPS

  • Restart dwpcontroller after making any of the changes above.
  • Confirm port 443 is working by running the following command from a terminal:

     netstat -tulpn | grep LISTEN | grep 443;  lsof -i :443

    You should see a response like:

    [root@xxxx work]#  netstat -tulpn | grep LISTEN  | grep 443
    tcp6       0      0 :::443                  :::*                    LISTEN      17353/java

    [root@xxxxxxxx]# lsof -i :443
    COMMAND   PID USER   FD   TYPE  DEVICE SIZE/OFF NODE NAME
    java    17169 root   16u  IPv6 4871373      0t0  TCP localhost:41076->localhost:https (CLOSE_WAIT)
    java    17169 root   17u  IPv6 4871157      0t0  TCP localhost:41070->localhost:https (CLOSE_WAIT)
    java    17169 root   18u  IPv6 4872364      0t0  TCP localhost:41082->localhost:https (CLOSE_WAIT)
    java    17169 root   19u  IPv6 4872436      0t0  TCP localhost:41122->localhost:https (CLOSE_WAIT)
    java    17353 root   32u  IPv6 4855123      0t0  TCP *:https (LISTEN)

    You can also confirm these processes are running by taking the PID (Process ID) and using following commands:

    [root@xxxxx]# ps -aux | grep 17169
    [root@xxxxxxx]# ps -aux | grep 17169;ps -aux | grep 17353

    root     17169  0.2  0.7 6910620 118908 pts/1  Sl   12:42   0:08 /apps/java/bin/java -jar /apps/dwpc/bin/healthcheck-1.0-shaded.jar /apps/dwpc/bin/monitor.properties
    root     27997  0.0  0.0 112716   972 pts/1    S+   13:32   0:00 grep --color=auto 17169
  •  The Jetty <BMC Digital WorkplaceCatalogInstallDir>/sb/jetty/work/jersey.0.0.log should show the following entries:

      cat jersey.0.0.log | grep -i ssl

        at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:426)
        at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:320)
        at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:158)

  • The <BMC Digital WorkplaceCatalogInstallDir>/sb/rxscripts/trace.log will give you additional details:

./sb/rxscripts/trace.log:1:== Info: About to connect() to xxxxxx port 443 (#0)
./sb/rxscripts/trace.log:3:== Info: Connected to xxx (127.0.1.1) port 443 (#0)
./sb/rxscripts/trace.log:8:== Info:     subject: CN=cxxxx
./sb/rxscripts/trace.log:11:== Info:    common name: xxxxx
./sb/rxscripts/trace.log:12:== Info:    issuer: CN=xxxxx
./sb/rxscripts/trace.log:19:00e0: Host: xxxxxx
./sb/rxscripts/trace.log:58:== Info: Added cookie _cacheId="89" for domain xxxx, path /, expire 0


If the issue is still not resolved, you can add additional logging if needed using the following steps:

  1. Go to to /apps/dwpc/bin/arserverd.conf.
  2. Modify cat jersey.0.0.log and add the option:  jvm.option.24=Djavax.net.debug=ssl
  3. Restart dwpcontroller.

The additional logging will be captured in startup_console.log and armonitor.log.

Scenario 1. 623 Authentication Failed after Remedy Application Service password was changed.

Scenario 2. How to Update AR Server References in BMC Digital Workplace when Remedy Application Password has Changed

The following steps apply to BMC Digital Workplace 18.05 and later:

Important: Take a database snapshot prior to modifying anything at the database level.

Follow the steps given below when:

  • You need to change the AR Server and database information used in BMC Digital Workplace
  • Cloning BMC Digital Workplace environments
  • You need to change the Remedy Application System
  • You need to change the Remedy Port number
  • You need to change the Remedy Hostname/Load Balancer
  • All of the above

Encrypting the Password

If the Remedy Application Password has changed, the password needs to be encrypted and updated in the database.

  1. Go the the Smart IT Maintenance Tool location:
    <SmartIT Installdirectory>/SmartIT/SmartITMaintTool/SmartIT Maintenance Tool/
  2. Perform one of the following steps:

    • Windows: Right Click the tool or file, run it as an administrator > click the Encrypt tab > encrypt the password
      image2021-11-2_14-14-36.png
    • Linux: sh SmartITMaintenanceTool.sh

3. Save that encrypted password and use it when updating the AR server password in the database.


Updating AR Server references in the database

1. Edit the following files and point them to the new host. Update SQL/Oracle database details as needed.


    • /opt/apache/tomcat8.5/conf/Catalina/localhost/dwp.xml
    • /opt/apache/tomcat8.5/conf/Catalina/localhost/social.xml  

2. Edit the following file: 

/opt/apache/tomcat8.5/external-conf/connect-dwp.properties

Update the api.token value to the current api.token (if it has changed) and use the same api.token for all your BMC Digital Workplace Servers.     

3. Use the steps given below to verify that the  BMC Digital Workplace database has the new AR Server host name, port, and password.


    • To check for and update the AR server host name:
      1. Check the actual value: 
        select * from DWP_Business.CONFIGURATION_PARAMS where KEY_NAME ='connect.arsystem.hostName' and VALUE = <Primary AR server name> 
      2. Set the new or correct value:
        update DWP_Business.CONFIGURATION_PARAMS SET VALUE= <AR LB Name> where KEY_NAME ='connect.arsystem.hostName' and VALUE = <Primary AR server name> 

    • To check for and update the AR Server Port details:
      1. Check the actual value:
        select * from DWP_Business.CONFIGURATION_PARAMS where KEY_NAME ='connect.arsystem.port'   
      2. Set the new or correct value:
        update DWP_Business.CONFIGURATION_PARAMS SET VALUE = <AR LB Port Number> where KEY_NAME ='connect.arsystem.port' and VALUE = <Primary AR Port>

    • To check for and update the Remedy Application System Password:
      1. Check the actual value:
        select * from DWP_Business.CONFIGURATION_PARAMS where KEY_NAME ='connect.arsystem.password' and VALUE = <password>
      2. Set the new or correct value:
        update DWP_Business.CONFIGURATION_PARAMS SET VALUE = <new password> where KEY_NAME ='connect.arsystem.password' and VALUE = <password> 

 
 4. Update DWP_System.PROVIDER_SETTINGS with correct AR hostname, port number, and password. For reference, use the queries given in the previous step:


    • update DWP_System.PROVIDER_SETTINGS  SET VALUE= 'value'  where KEYNAME ='connect.arsystem.hostName'
    • update DWP_System.PROVIDER_SETTINGS  SET VALUE = 'value'   where KEYNAME ='connect.arsystem.port'
    • update DWP_System.PROVIDER_SETTINGS  SET VALUE = 'value'  where KEYNAME ='connect.arsystem.password'
    • update DWP_System.PROVIDER_SETTINGS  SET VALUE = 'value'   where KEYNAME ='mid_tier.base.url'


Updating AR Server References in BMC Digital Workplace Admin

1. Log in to BMC Digital Workplace Administration and go to Configuration > Providers.

2. Update all Pluggable Providers in /dwp/admin/configuration/features.html with the correct AR Server host name, port number, and credentials where needed.

Updating RSSO configuration

1. If using RSSO, edit the RSSO agent files in <apache install directory>/external-conf:


    • Update the rsso-agent.properties with the new RSSO Server URL
    • Update the "ignore-tenant=true" parameter in the sso-sdk.properties file

2. Restart the Tomcat servers for the changes to take effect.


If you are still seeing the authentication issue, collect the following logs in debug mode to investigate further:

  • BMC Digital Workplace: dwp.log
  • Tomcat: all current logs

 

Tip: For faster searching, add an asterisk to the end of your partial query. Example: cert*