Troubleshooting authentication issues

A non-existing domain is mapped to a realm

When a user request is redirected to the BMC Helix SSO login URL, the following message is displayed:

Unexpected error happened. Not possible to define a realm.

This error happens when the administrator deletes the default realm "*'" and then adds another realm, but does not configure a domain of the new realm.

In most cases, the issue is related to the incorrect redirect URI in the /authorize request. The following steps are required:

1. Compare the redirect_uri query parameter in the end-user request query with the redirect URI specified for the particular OAuth client in the BMC Helix SSO Admin Console.
   The client_id can be found in the same end-user request query. Make sure that the redirect URI you are looking for in the BMC Helix SSO Admin Console corresponds to the proper client_id.
2. Add the missing redirect URI to the OAuth client in the BMC Helix SSO Admin Console.
3. Reinitiate the login flow.

After configuring the BMC Helix SSO agent for using the OpenID scope, the BMC Helix SSO server fails to generate id_token in the following scenarios, and an exception is logged in the BMC Helix SSO agent logs:

• If you do not generate the JSON web key (JWK), the BMC Helix SSO server does not find the private key to sign and cannot generate the issue id_token. 
• You do not provide the URL of the server issuing the id_token.

For more information about the exception, see the BMC Helix SSO agent logs.

After configuring the BMC Helix SSO agent for using the OpenID scope, make sure that you:

• Provide the URL of the server issuing the id_token in the OpenID Issuer URL field on the OAuth2 > Settings tab.
• Generate the JWK from the OAuth2 > OpenID tab.

At the time of registering a client as an OAuth2 client, if you do not select the openid (Scope used for OpenID connect) check box, the client cannot use the OpenID scope. The BMC Helix SSO server logs a message mentioning that the specified OAuth2 client is not allowed to use the OpenID scope.

Synchronize the time on the BMC Helix SSO server and BMC Helix SSO agent machines.

The cross launch link is not displayed.

• In Chrome, no text is displayed in the iframe. The following message is displayed in the developer console (F12):

  Refused to display 
  '<originating_app_host>:<port>/rsso/cross-sso?
  goto=<target_app_host>#jwt=<jwt_value>' 
  in a frame because an ancestor violates the following
  Content Security Policy directive: "frame-ancestors 'self'".
  
  

• The following message is displayed in Firefox:

  Blocked by Content Security Policy.
  
  

• The following message is displayed in Microsoft Internet Explorer:

  This content cannot be displayed in an frame.
  
  

• The following message is displayed in Microsoft Edge:

  This content can’t be shown in an frame.

The target BMC Helix SSO server is not configured correctly.

On the target BMC Helix SSO server, for the Preauth authentication method, make sure that you enter the name of the originating BMC Helix SSO server in the ALLOW-FROM Domain(s) field.

For information about how to configure this field, see Configuring preauthentication.

The following error message is displayed in an iframe:

Unexpected error happened. Failed to login. Please contact the Administrator.

A wrong certificate is configured for a realm with the Preauth authentication method on the target BMC Helix SSO server.

1. Enable the DEBUG log level on the target BMC Helix SSO server and reproduce the issue.

2. Check the logs. The following statements might be displayed:

   Authentication failed. Reason: 'Could not parse certificate:
   java.io.IOException: java.lang.IllegalArgumentException: 
   Input byte array has wrong 4-byte ending unit'
   Authentication failed. 
   Reason: 'JWT signature does not match locally computed signature. 
   JWT validity cannot be asserted and should not be trusted.'
   
   

3. Configure the correct certificate for the Preauth authentication method on the target BMC Helix SSO server.

AREA plugin error after AR user credentials were submitted

The user completes a BMC Helix SSO login (on the BMC Helix SSO login page for AR authentication or on the IdP login page) and is redirected back to BMC application URL.

Then, the application might display an authentication error.

The AR AREA plugin log file (ARSystemInstallFolder/Arserver/Db/arjavaplugin.log) might contain the following or similar error logs:

2015-09-13 17:04:21,324 ERROR [pool-4-thread-10] com.bmc.arsys.pluginsvr.plugins.ARPluginContext (?:?) 
- <ARSYS.AREA.RSSO> Could not validate userId with Service Provider. 
Could not retrieve user from authentication string.

2015-09-13 17:04:21,324 ERROR [pool-4-thread-10] com.bmc.arsys.pluginsvr.plugins.ARPluginContext (?:?) 
- <ARSYS.AREA.RSSO> Return Code:2

1. Check the contents of the ARSystemInstallFolder/Conf/rsso.cfg file, and verify that the value of the sso-service-url property corresponds to the BMC Helix SSO server URL.
2. Check that the sso-service-url is accessible from the AR server node.
   You can use curl to test network connectivity from the AR server node to the BMC Helix SSO server URL.
3. Check if any proxy is used on the AR System plug-in server for the outgoing network connection.
   • The proxy option must be defined in the Java options in armonitor.cfg. The startup string for the Java plugin server would have Java parameters for enabling proxy.
   • If the proxy server is defined in the Java plugin server, force a bypass of the proxy for the BMC Helix SSO server host name.

Remedy AR System is integrated with BMC Helix SSO, and authenticated users have no groups after login.

Various issues can cause this error. Use the steps in the workaround to identify and fix any issues that you find. This troubleshooting can be applied only if BMC Helix SSO is integrated with Remedy AR System. For all other cases, see Troubleshooting in the Remedy AR System documentation.

IdP error on SAML request if SAML IdP login URL contains a query parameter

If the NameID in a SAML response returned by IdP exceeds 255 characters, increase the size of the NameId column in the IssuesTokens table of the BMC Helix SSO database. For example, if the NameID length is 300 characters, set the NameId column to at least VARCHAR(300).

The issue is caused by the absence of a time sync between BMC Helix SSO and the IdP server. In the BMC Helix SSO Admin Console, configure the Assertion Time Skew attribute; see Importing configuration from an identity provider and configuring SAML.

Fail to generate and view SP metadata in BMC Helix SSO

When you generate or view SP metadata in BMC Helix SSO, the following error is displayed:

Metadata is incomplete. Following attributes are missed or empty: [SP certificate]

SP certification configuration is missing or incorrect. Follow the steps:

1. Configure an SP certificate as described in Creating and updating the SP signing certificate for SAML authentication.
2. Run the keytool command (keytool -list -keystore <path to the keystore file> -storepass <keystore password>) or a certificate tool such as Keystore Explorer to identify the key alias. 
   The key alias configured in the BMC Helix SSO SP certificate configuration must match the key alias in the keystore.

An incorrect client application URL (relayState) might be provided. These URLs are stored in the BMC Helix SSO database table "RelayState" and should be the same as the requested ones. You can also verify that the browser shortcut to the client application URL is correct.

An IdP certificate configured in the BMC Helix SSO Admin Console might be invalid or expired. Obtain an updated IdP metadata URL from the SAML team of the organization and import it into the BMC Helix SSO configuration. For more information, see Importing configuration from an identity provider and configuring SAML.

Log in to the BMC application through BMC Helix SSO fails with the configured SAML authentication

In the BMC Helix SSO Admin Console, remove the saml:AuthNContextClassRef value from the SAML configuration. For more information, see Troubleshoot sign-on to SAML-based SSO apps in Microsoft .

Users are not authenticated. The following records are available in the BMC Helix SSO server logs:

javax.naming.CommunicationException: simple bind failed:

<LDAP SERVER NAME>:PORT

[Root exception is javax.net.ssl.SSLHandshakeException:

server certificate change is restricted during renegotiation]

Add the Java properties for the JVM that Tomcat uses.

-Djdk.tls.allowUnsafeServerCertChange=true 
-Dsun.security.ssl.allowUnsafeRenegotiation=true

JAVA_OPTS="-Djdk.tls.allowUnsafeServerCertChange=true"
JAVA_OPTS="$JAVA_OPTS - D sun.security.ssl.allowUnsafeRenegotiation=true"
export JAVA_OPTS 

Microsoft Windows example:

• 1. Use the tomcatXw.exe GUI.
  2. Create a setenv.bat file with the content similar to the Linux example.

If the LDAP server uses a self-signed certificate, the JVM that Tomcat uses on the BMC Helix SSO server does not trust this certificate.

To use TLS/SSL connection to the LDAP server, import the LDAP server certificates (cacerts) to the truststore (JavaHome \jre\lib\security) of the Apache Tomcat used by the Remedy SSO server. Import the certificates by using third-party utilities such as KeyStore Explorer .

When the BMC Helix SSO server and the integrated application both use self-signed TLS/SSL certificates for the HTTPS connection, the certificate confirmation dialog box breaks the flow, and you cannot log in by using Microsoft Edge and Safari browsers.

LDAP test connection fails

In the BMC Helix SSO Admin Console, when you configure LDAP and click Test, the following message is displayed: Connection to the LDAP server failed or the host name or port number is incorrect.

The test connection failure can be caused by the incorrect attributes in the LDAP authentication window.

First, in the BMC Helix SSO Admin Console, check the following attributes:

• Server Host(s)—The host name of your directory server, for example, ad.bmc.com, ldap.bmc.com, opends.bmc.com.
• Server Port—The port that your directory listens on, for example, 389, 10389, 636 (for example, the SSL port that your directory listens on).
• Use TLS connection—This check box should be selected if the connection to the directory server is an SSL connection.
  To use this setting, you also need to configure an SSL certificate.
• Bind DN—The distinguished name of the user that the application leverages when connecting to the directory server; for example, cn=administrator,cn=users,dc=ad,dc=bmc,dc=com; cn=user,dc=domain,dc=com; user@domain.com.
  To connect to LDAP, the user must have the "Bind" and "Read" privileges.
• Bind Password—The password of the user specified in the Bind DN field.
  The password cannot be one-way hashed. It must be recoverable in the context of the application.

If the test connection still fails, use LDAP administration tools such as LDAP Admin for Windows or JXplorer for Linux to identify whether the issue is caused by BMC Helix SSO or your LDAP connection.

LDAP Admin example:

1. Click Start > Connect > Create New Connection.
2. In the Connection name field, specify a descriptive connection name.
3. Click Admin, and configure the following attributes:
   

• Host—Specify the name of the LDAP server.
• Port—Specify the default LDAP port is 389.

4. (Optional) If the LDAP server uses an SSL certificate, the port number changes to 636. In this case, you must select the SSL check box.

5. Clear the Anonymous connection check box.

6. In the Username field, specify a username or an LDAP bind user format name.

7. In the Password field, specify the password for the user.

8. Click Test connection.

If the test connection is successful, click Fetch DNs and see a list of DNs on the LDAP server.

If the test connection fails with the following messages, contact the LDAP Administrator:

• Exception Security Context—The username or password is incorrect.

• LDAP Server Down—The hostname is incorrect or you cannot reach the LDAP server from your location.

To enable LDAPS:

1. If the LDAP team does not have such a certificate, install a certificate that meets the documented requirements; see the Requirements for an LDAPS certificate paragraph in How to enable LDAP over SSL with a third-party certification authority .
2. Import the LDAP certificate into the Java keystore file of the BMC Helix SSO server by running the following command: keytool -importcert -v -keystore "<path-to-jre>\lib\security\cacerts" -file "<path-to-ldap-cert-file>"
   
3. Enter the password.
   This Java keystore file should be the one that runs the Tomcat service for BMC Helix SSO. To find out which Java keystore file serves the BMC Helix SSO Tomcat on Windows, run tomcat.exe depending on which Tomcat version you use, and under the Java tab, see the path. For Linux, run ps ef | grep tomcat.

4. (Windows) In the Java Options field, specify the Java truststore parameters, for example:

-Djavax.net.ssl.trustStore=<path to jre cacerts> 
-Djavax.net.ssl.trustStorePassword=changeit 

5. (Linux) Go to the sh Tomcat startup file and specify the Java truststore parameters, for example:

set JAVA_OPTS="-Djavax.net.ssl.trustStore=<path to jre_keystore" "-Djavax.net.ssl.trustStorePassword=changeit"  

6. Save the configuration.

7. Restart the Tomcat service.

8. In the BMC Helix SSO Admin Console, test the LDAP connection by using the server port 686 and selecting the Use TLS connection check box.

Login to BMC Mid Tier, BMC Helix ITSM: Smart IT and BMC Helix Digital Workplace fails.

Users cannot log in to BMC Mid Tier, BMC Helix ITSM: Smart IT and BMC Helix Digital Workplace.

There are multiple factors that can cause this problem. On the BMC Helix SSO side, you can use the two paths:

• Check whether the LDAP users are already present in the CTM:People form.
  In the BMC Helix SSO logs, you can see the following message:
  ERROR: [LDAP: error code 49 - 80090308: LdapErr: DSID-0C09042F, comment: AcceptSecurityContext error, data 52e, v2580
  This error indicates an issue with the user account that is used to connect to the LDAP server. The account can be locked out, requires a password change, or the password is incorrect.
• Check one of the BMC Helix SSO server logs: "rsso.log" or "rsso.0.log". The UserID must not be "None" and LDAP must send the correct user name (the same as the Login name in the CTM:People form of AR user forms).

If you log in to one of these BMC products and see the error "You are not authorized to use this system", the user does not belong to any group that has correct login permissions.

To check it on the BMC Helix SSO side, in the BMC Helix SSO Admin Console, click the Session tab.

If the userID is listed, but disappears in 10 seconds. BMC Helix SSO has successfully authenticated the user with LDAP, but BMC TrueSight Operations Management has not given the user a group permission.

You must check whether the users and groups belong to the Authorization Profile in BMC TrueSight Presentation Server:

1. Log in to BMC TrueSight Presentation Server as a default admin user.
2. Go to Administration > Authorization Profiles.
3. Create a new authorization profile by clicking the vertical ellipsis next to the Authorization Profiles field.
4. In the Authorization Profile Name field, specify the authorization profile name.
5. Click User Groups > Add.

BMC Helix SSO connects to LDAP to fetch the groups from the LDAP server and the local BMC Helix SSO User Management Groups (Roles).

After the group has been added to the profile, you must assign roles to this group:

1. Click Roles > Add.
2. In the Search for Roles window, select the roles that you want to add for the profile.
3. Click OK.
   Depending on the roles you have given to the authorization profiles, you might also need to add objects.

If you log in to BMC TrueSight Operations Management as an LDAP user, BMC Helix SSO authenticates you and then BMC TrueSight Operations Management authorizes you. Any other LDAP users that belong to this group can also log in to BMC TrueSight Operations Management.

In the BMC Helix SSO Admin Console, check that the attributes in the LDAP authentication window are correct and the Enable Group Retrieval check box is selected.

The external LDAP attributes are mapped to the BMC Helix SSO fields:

• User Attributes are mapped to the User Authentication tab in BMC Helix SSO.
• Group Attributes are mapped to the Group Support tab in BMC Helix SSO.
• Connection Settings are mapped to the LDAP connection settings in the BMC Helix SSO Admin Console (Host, Port, Bind DN, Bind Password, Users Base DN, Identity Attribute).

If the issue persists, use the filters in the LDAP administration tool such as LDAP Admin to see how many results you have. If the list is lengthy, narrow down the search to see the groups.

The LDAP server returns more than one entry for one user with an error in the BMC Helix SSO logs:

04-03-20 10:23:57.069 [3-exec-249] WARN  com.bmc.rsso.data.idps.IdPLDAP.getUserGroups()  : LDAP error: 'LDAP server returned more than one entry for user: userxxxx'
com.bmc.rsso.core.ldap.LDAPConfigurationException: LDAP server returned more than one entry for user: userxxxx
    at com.bmc.rsso.core.ldap.LDAPHelper.searchUserEntry(LDAPHelper.java:103)
    at com.bmc.rsso.core.ldap.LDAPHelper.getGroupsByUser(LDAPHelper.java:173)
    at com.bmc.rsso.data.idps.IdPLDAP.getUserGroups(IdPLDAP.java:298)
    at com.bmc.rsso.servlet.rest.admin.GroupsForUserResource$1.doWithNode(GroupsForUserResource.java:63)
    at com.bmc.rsso.servlet.rest.admin.IdpsToJson.convert(IdpsToJson.java:35)
    at com.bmc.rsso.servlet.rest.admin.GroupsForUserResource.getUserGroupsForUser(GroupsForUserResource.java:65)
    at sun.reflect.GeneratedMethodAccessor123.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    at java.lang.reflect.Method.invoke(Unknown Source)

It is a LDAP configuration issue. In the BMC Helix SSO Admin Console, in the User Authentication tab for LDAP, compare the default filter parameters against their values and correct any that do not match. The (sAMAccountName=$USER$) parameter must be specified.