Troubleshooting BMC Helix Digital Workplace Tomcat

Navigate to <tomcat x>/logs directory, and review the following log files. Look for SEVERE and Warning entries to find the root cause of the issue:

• localhost.log
• catalina.out 
• catalina.%date%.log

Review the BMC Helix Digital Workplace log files in the <BMC Helix Digital Workplace Installation>/DWP/DWP/logs directory and look for ERROR entries.

Most of the time the root cause is likely to be:

• Applications using the same port as Tomcat
• Database connectivity issues
• Locks on the BMC Helix Digital Workplace database
• Errors in the SSL Configuration

Configuring BMC Helix Digital Workplace Logs

Configuring-logs-for-BMC-Helix-Digital-Workplace

Unable to start BMC Helix Digital Workplaceor BMC Helix Digital Workplace Catalog after SSL is enabled, due to the error "PKIX path building failed" under BMC Helix Digital Workplace Enhanced Catalog Section

To fix the error "PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target error", follow these steps:

1. Log in to BMC Helix Digital Workplace Admin console and go to Configuration > Enhanced Catalog. 
2. Get the cert from BMC Helix Digital Workplace Catalog by using Firefox:

3. Ensure that all BMC Helix Digital Workplace Catalog cluster nodes have the cert imported to the servers. Also import the certs into the cluster nodes for BMC Helix Digital Workplace, AR System, BMC Helix ITSM: Smart IT, and BMC Helix SSO servers.

4. Import the certificate to the cacerts file by using the following commands:

Note: This can be Java cacerts or a custom cacerts.

1. 
   
   1. openssl x509 -outform der -in catalogcert.pem -out certificate.der
   2. keytool -import -alias DWPCatalog -trustcacerts cacerts -file certificate.der

5. Restart all application Tomcat (BMC Helix Digital Workplace, BMC Helix Digital Workplace Catalog, BMC Helix ITSM: Smart IT) and AR System services.

6. Clear the browser cache.

7. Log into the BMC Helix Digital Workplace Catalog Admin Console and go to Configuration > Enhanced Catalog. 

8. Update the BMC Helix Digital Workplace Catalog URL with the correct port and URL values.

Note: Repeat for each BMC Helix Digital Workplace Catalog Load Balanced URLs by logging into the BMC Helix Digital Workplace Admin console > Configuration > Enhanced Catalog. Update the BMC Helix Digital Workplace Catalog URL with the correct port and URL information from each node.

9. Do the same (update with correct port and URL values) on the AR System's SB:RemoteApprovalConfiguration and Centralized Configuration's com.bmc.itsm.sbe component.

10. Update the Setting Value with the correct values.

11. Click Save.

12. Confirm that you are not getting the error anymore.

If you are still getting the error, use SSL Poke to verify connectivity. SSL Poke is a third-party tool that enables you to verify that the cacerts file has the correct certificates added to it to connect securely.

1. Download SSL Poke from any third-party site, and execute it on the BMC Helix Digital Workplace Catalog Server. You will need to repeat these steps from BMC Helix Digital Workplace, AR System, BMC Helix ITSM: Smart IT, and BMC Helix SSO servers.
2. On a cmd/terminal Windows, run the following commands to execute SSL Poke:

java SSLPoke DWPCatalogServerFQDN SSLPort
Example:
java SSLPoke dwpcatalogservername.com 8443

If the connection is successful, you will get a successful message such as "Connection was successful".

If the connection failed, you will get more details on the failure, such as:

• SSL Port is blocked
• Certificate does not exist in the keystore/truststore
• Certificate is expired. Check validity
  
  

You will need to resolve the issues identified by SSL Poke for the connection to be successful. For example:

• If the error is "SSL Port is blocked", you will need to open the port to allow the connection in the firewall.
• If the error is "Certificate does not exist in the keystore/truststore", you can check if the certificate exists by running the following command from cmd/terminal Windows:

    keytool -list -v -keystore keystore.jks  

• If the error is "certificate chain/Root-intermediate cert is missing", work with your CA Security Team to get this issue is rectified.
• If the error is "Server alias/dns entry for this catalog does not exist in the certificate", run the following command and replace mydomain with dwpcatalogFQDN

    keytool -list -v -keystore keystore.jks -alias mydomain
 

BMC Helix Digital Workplace Catalog Advanced/Basic +SSL

Additional resources:

• For user group sync issues not working over SSL
  KA 000133376  BMC Helix Digital Workplace Catalog - User Group sync script doesn't work with SSL
• For AD  over SSL issues
  KA 000164451  BMC Helix Digital Workplace Catalog- Configuring AD Connector on DWP Catalog. Includes AD over SSL

Authentication Failed
occurs during a restart of the BMC Helix Digital Workplace Catalog server, after SSL is enabled.

BMC Helix Digital Workplace Catalog Services/Banners are not visible after SSL is enabled.

BMC Helix Digital Workplace Catalog loops when service is restarted.

Ensure that you follow these requirements when configuring the BMC Helix Digital Workplace Catalog server files:

• Use the FQDN where the instruction is to use FQDN (and not the host name) in <installLocation>/sb/rxscripts/bin/setenv.sh
• use the host name where the instruction is to use host name (and not the FQDN) in <installLocation>/sb/env/set_script_variables.sh
• Do not use localhost in <installLocation>/sb/rxscripts/bin/setenv.sh or <installLocation>/sb/env/set_script_variables.sh
• Ensure that you update the section below on <installLocation>/jetty/etc/jetty-http.xml with your correct keystore and truststore information:

• Ensure ar.conf has the following values:

Jetty-Port: 443

Jetty-Protocol: HTTPS

• Restart dwpcontroller after making any of the changes above.
• Confirm port 443 is working by running the following command from a terminal:
  
   netstat -tulpn | grep LISTEN | grep 443;  lsof -i :443
  
  You should see a response like:
  
  [root@xxxx work]#  netstat -tulpn | grep LISTEN  | grep 443
  tcp6       0      0 :::443                  :::*                    LISTEN      17353/java
  
  [root@xxxxxxxx]# lsof -i :443
  COMMAND   PID USER   FD   TYPE  DEVICE SIZE/OFF NODE NAME
  java    17169 root   16u  IPv6 4871373      0t0  TCP localhost:41076->localhost:https (CLOSE_WAIT)
  java    17169 root   17u  IPv6 4871157      0t0  TCP localhost:41070->localhost:https (CLOSE_WAIT)
  java    17169 root   18u  IPv6 4872364      0t0  TCP localhost:41082->localhost:https (CLOSE_WAIT)
  java    17169 root   19u  IPv6 4872436      0t0  TCP localhost:41122->localhost:https (CLOSE_WAIT)
  java    17353 root   32u  IPv6 4855123      0t0  TCP *:https (LISTEN)
  
  You can also confirm these processes are running by taking the PID (Process ID) and using the following commands:
  
  [root@xxxxx]# ps -aux | grep 17169
  [root@xxxxxxx]# ps -aux | grep 17169;ps -aux | grep 17353
  
  root     17169  0.2  0.7 6910620 118908 pts/1  Sl   12:42   0:08 /apps/java/bin/java -jar /apps/dwpc/bin/healthcheck-1.0-shaded.jar /apps/dwpc/bin/monitor.properties
  root     27997  0.0  0.0 112716   972 pts/1    S+   13:32   0:00 grep --color=auto 17169
•  The Jetty <BMC Digital WorkplaceCatalogInstallDir>/sb/jetty/work/jersey.0.0.log should show the following entries:

      cat jersey.0.0.log | grep -i ssl

        at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:426)
        at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:320)
        at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:158)

• The <

  BMC Helix Digital Workplace Catalog

  InstallDir>/sb/rxscripts/trace.log will give you additional details:

./sb/rxscripts/trace.log:1:== Info: About to connect() to xxxxxx port 443 (#0)
./sb/rxscripts/trace.log:3:== Info: Connected to xxx (127.0.1.1) port 443 (#0)
./sb/rxscripts/trace.log:8:== Info:     subject: CN=cxxxx
./sb/rxscripts/trace.log:11:== Info:    common name: xxxxx
./sb/rxscripts/trace.log:12:== Info:    issuer: CN=xxxxx
./sb/rxscripts/trace.log:19:00e0: Host: xxxxxx
./sb/rxscripts/trace.log:58:== Info: Added cookie _cacheId="89" for domain xxxx, path /, expire 0

If the issue is still not resolved, you can add additional logging if needed using the following steps:

1. Go to to /apps/dwpc/bin/arserverd.conf.
2. Modify cat jersey.0.0.log and add the option:  jvm.option.24=Djavax.net.debug=ssl
3. Restart dwpcontroller.

The additional logging will be captured in startup_console.log and armonitor.log.

BMC Helix Digital Workplace Catalog + SSL

Scenario 1. 623 Authentication Failed after Helix Application System password was changed.

Scenario 2. How to Update AR System References in BMC Helix Digital Workplace when Helix Application Password has Changed

Important: Take a database snapshot prior to modifying anything at the database level.

Follow the steps given below when:

• You need to change the AR System and database information used in BMC Helix Digital Workplace
• Cloning BMC Helix Digital Workplace environments
• You need to change the Helix Application System
• You need to change the Helix Port number
• You need to change the Helix Hostname/Load Balancer
• All of the above
  
  

Encrypting the Password

If the Helix Application Password has changed, the password needs to be encrypted and updated in the database.

1. Go the the BMC Helix ITSM: Smart IT Maintenance Tool location:
   <BMC Helix ITSM: Smart ITInstalldirectory>/SmartIT/SmartITMaintTool/BMC Helix ITSM: Smart IT Maintenance Tool/
2. Perform one of the following steps:

• 
  
  • Windows: Right Click the tool or file, run it as an administrator > click the Encrypt tab > encrypt the password
    
  • Linux: sh SmartITMaintenanceTool.sh

3. Save that encrypted password and use it when updating the AR System password in the database.

Updating AR System References in BMC Helix Digital Workplace Admin

1. Log in to BMC Helix Digital Workplace Administration and go to Configuration > Providers.

2. Update all Pluggable Providers in /dwp/admin/configuration/features.html with the correct AR System host name, port number, and credentials where needed.

Updating BMC Helix SSO configuration

1. If using BMC Helix SSO, edit the BMC Helix SSO agent files in <apache install directory>/external-conf:

• 
  

  • Update the rsso-agent.properties with the new 

    BMC Helix SSO

     Server URL

  • Update the "ignore-tenant=true" parameter in the sso-sdk.properties file

2. Restart the Tomcat servers for the changes to take effect.


If you are still seeing the authentication issue, collect the following logs in debug mode to investigate further:

• BMC Helix Digital Workplace: dwp.log
• Tomcat: all current logs

See also http://youtube.com/watch?v=YpoTU30vOBY