Converting to FIPS-140 mode

BMC recommends that you monitor the FIPS-140 mode conversion. See Monitoring-FIPS-140-and-normal-mode-conversions.

Perform the following steps for converting the FIPS-140 mode.

1. Converting from BMC Atrium Single Sign-On to FIPS-140 mode
2. Installing the unlimited strength policy files
3. Installing the cryptography library
4. Enabling FIPS-140 mode
5. To deploy the web agent for FIPS-140

To convert from BMC Atrium Single Sign-On to FIPS-140 mode

When operating in FIPS-140 mode, BMC Atrium Single Sign-On blocks contact with products which are not also operating in a FIPS-140 compliant mode.

Before performing the switch to FIPS-140 mode

• Perform a system backup before switching to (or from) FIPS-140 mode. An unexpected hardware or software failure during the conversion can corrupt the server configuration.
• Verify that the integrated BMC products are capable of operating in a FIPS-140 compliant mode and are capable of making the reconfiguration that is required to continue operating with BMC Atrium Single Sign-On.
• If you plan to integrate additional products with BMC Atrium Single Sign-On after the switch to FIPS-140 mode is complete, be sure that these products can be integrated with the server. See the BMC Atrium Single Sign-On Product Availability Compatibility on the support website.
• Ensure that your Internet browser is capable of supporting 256-bit Advanced Encryption Standard (AES) encryption. See #Browser cipher capabilities.
• Obtain the RSA CryptoJ FIPS cryptography module. See #RSA CryptoJ FIPS cryptography module.
  
  Contact Customer Support for access to the RSA CryptoJ FIPS cryptography module. This library file must be installed into the server's Java Virtual Machine (JVM), replacing the current version which is not certified.
   Obtain unlimited strength Java policy files.
  
  BMC Atrium Single Sign-On uses Oracle JVM 1.7.0. The unlimited policy files for this JVM are available for download from the following URL: http://java.sun.com/javase/downloads/index.jsp.

Browser cipher capabilities

When operating in FIPS-140 mode with default networking ciphers, the Internet browser must be capable of supporting 256-bit Advanced Encryption Standard (AES) encryption. Otherwise, the browser cannot connect with BMC Atrium Single Sign-On for administrator or user authentication purposes. FireFox 3+ is able to operate at this level. Internet Explorer might not be able to support 256-bit AES depending on the version.

You can check your browser cipher capabilities at the following URL: http://www.fortify.net/sslcheck.html. This web site provides the encryption status of your browser.

RSA CryptoJ FIPS cryptography module

The FIPS-approved cryptography module used by BMC Atrium Single Sign-On for FIPS-140 compliance is the RSA CryptoJ library version 6.1.

The following table shows the algorithms used in normal mode and FIPS-140 mode.

To install the unlimited strength policy files

BMC Atrium Single Sign-On uses Oracle JVM version 1.7.0. By default, this JVM is installed with strong encryption policy files allowing for limited strength settings for encryption algorithms. These limitations prevent BMC Atrium Single Sign-On from running in FIPS-140 mode. To overcome this limitation, the Unlimited Strength Jurisdiction Policy Files must be downloaded from Oracle and installed into the BMC Atrium Single Sign-On JVM.

1. Shut down all BMC Atrium Single Sign-On integrated products.
2. Stop BMC Atrium Single Sign-On.
3. If you have not done so already, download the archive that contains the unlimited strength policy files from the following URL: http://java.sun.com/javase/downloads/index.jsp.
4. Extract the contents of the files.
5. Make a backup copy of the currently installed strong strength policy files.
6. Copy the unlimited strength policy files into the BMC Atrium Single Sign-On JVM.

JVM location

The JVM is located in the following default location:

• (Microsoft Windows) C:\Program Files\BMC Software\AtriumSSO\jre\lib\security
• (UNIX) /opt/bmc/AtriumSSO/jre/lib/security

If BMC Atrium Single Sign-On has been installed in a non-default location, the location of the JVM can be determined by using the following pattern:

• (Windows) <installationDirectory>\AtriumSSO\jre\lib\security
• (UNIX) <installationDirectory>/AtriumSSO/jre/lib/security

In this case, installationDirectory is the base directory selected during the server installation.

For BMC Atrium Single Sign-On servers using an external Tomcat server, the location of the JVM was determined by the administrator that configured the Tomcat server. Regardless of the JVM location, the following templates indicate the correct location:

• (Windows) <jdkDirectory>\jre\lib\security
• (UNIX) <jdkDirectory>/jre/lib/security

In this case, jdkDirectory is the base directory of the JDK used to run BMC Atrium Single Sign-On.

To install the cryptography library

For cryptographic functions in normal mode, BMC Atrium Single Sign-On uses the JVM and a version of the RSA CryptoJ library that is not certified for FIPS-140 operation. However, when placed into FIPS-140 mode, the server reconfigures the JVM to use the RSA CryptoJ provider as the primary provider. In addition, the cryptography needs of the server exclusively uses this provider.

For the server to start in FIPS-140 mode successfully, the FIPS-140 certified version of the RSA CryptoJ library must be installed into the JVM, replacing the uncertified version. The versions of the library can be externally identified by the names of the libraries. Normal mode library is cryptoj.jar and the FIPS-140 mode libraries are cryptojce.jar, cryptojcommon.jar, and jcmFIPS.jar.

1. Make a backup copy of the cryptoj.jar file. You might need to restore BMC Atrium Single Sign-On to normal encryption mode.
2. Copy the FIPS-140 mode libraries are cryptojce.jar, cryptojcommon.jar, and jcmFIPS.jar files onto the file system of the computer hosting BMC Atrium Single Sign-On.
3. Copy the FIPS-140 mode libraries are cryptojce.jar, cryptojcommon.jar, and jcmFIPS.jar files to the server's JVM library directory.

4. Remove the cryptoj.jarfile.

   Note

   This is an important step to prevent a collision of the two libraries.

JVM library file location

The JVM library is located in the following default location:

• (Microsoft Windows) C:\Program Files\BMC Software\AtriumSSO\jre\lib\ext
• (UNIX) /opt/bmc/AtriumSSO/jre/lib/ext

If BMC Atrium Single Sign-On server has been installed in a non-default location, determine the location of the JVM library using the following pattern:

• (Windows) <installationDirectory>\AtriumSSO\jre\lib\ext
• (UNIX) <installationDirectory>/AtriumSSO/jre/lib/ext

In this case, installationDirectory is the base directory selected during the server installation.

For BMC Atrium Single Sign-On servers utilizing an external Tomcat server, the location of the JVM was determined by the administrator that configured the Tomcat server. Regardless of the JVM location, the following templates indicate the correct location:

• (Windows) jdkDirectory\jre\lib\ext
• (UNIX) jdkDirectory/jre/lib/ext

To enable FIPS-140 mode

After restarting BMC Atrium Single Sign-On with the required JVM modifications in place, the server's configuration can be updated to trigger the change of cryptography. Before performing this next step, be sure that the following JVM modifications have been performed:

• Unlimited strength policy files are installed.
• The library cryptojce.jar, cryptojcommon.jar, and jcmFIPS.jar files are installed in library directory.
• The library cryptoj.jar file has been removed from the library directory.

1. (Optional) Update your network ciphers if desired. See Changing-FIPS-140-network-ciphers.
2. Restart BMC Atrium Single Sign-On.
3. Log on to BMC Atrium Single Sign-On administrator console.
4. Click Edit Server Configuration.
5. Select Enable FIPS-140

6. Click Save.

   Warning

   After the configuration has been successfully saved, the conversion process starts. This process cannot be interrupted. Do not stop BMC Atrium Single Sign-On, log on with another Administrator console, log off the current Administrator console, or initiate any other interactions with the server.

   This process takes approximately 10 to 20 seconds, depending upon the computer hardware. Be sure that the background task validation process posts a successful conversion message before proceeding to the next step.

7. Monitor the log files for the completion of the cryptography conversion. For more information on how to monitor the conversion, see Monitoring-FIPS-140-and-normal-mode-conversions.
8. After the conversion process completes, stop and start the server.
9. Verify that the server is properly operating in FIPS-140 mode by viewing the BMC Atrium Single Sign-On log file (for example, atsso.0.log)

10. Reconfigure all integrated products to operate in FIPS-140 mode.

    Note

    All products which were configured with BMC Atrium Single Sign-On prior to conversion to FIPS-140 mode must be reconfigured to operate in FIPS-140 compliant mode. These integrated products cannot use BMC Atrium Single Sign-On for authentication until they are synchronized with BMC Atrium Single Sign-On.

To deploy the web agent for FIPS-140

The following steps are used to integrate a new BMC application with an existing BMC Atrium Single Sign-On server which is already running in FIPS-140 mode. Follow these steps to modify the JVM that will be used to execute the deployer utility for the integration task.

1. Install the unlimited strength policy files and cryptography libraries as mentioned in the earlier steps.

2. Replace the existing list of security providers in the java.security file stored at the JAVA_HOME_FOR_AGENT>\jre\lib\security\ location with the following list of providers from the java.security.fips file.

   # List of providers and their preference orders (see above):
   #
   security.provider.1=com.rsa.jsafe.provider.JsafeJCE
   security.provider.2=sun.security.provider.Sun
   security.provider.3=sun.security.rsa.SunRsaSign
   security.provider.4=sun.security.ec.SunEC
   security.provider.5=com.sun.net.ssl.internal.ssl.Provider
   security.provider.6=com.sun.crypto.provider.SunJCE
   security.provider.7=sun.security.jgss.SunProvider
   security.provider.8=com.sun.security.sasl.Provider
   security.provider.9=org.jcp.xml.dsig.internal.dom.XMLDSigRI
   security.provider.10=sun.security.smartcardio.SunPCSC
   security.provider.11=sun.security.mscapi.SunMSCAPI
   
   com.rsa.cryptoj.jce.kat.strategy=on.load
   com.rsa.cryptoj.jce.fips140initialmode=FIPS140_SSL_MODE

3. Run the deployer.jar.

   Note

   If you running the deployer on JDK or JRE version 1.7.0_51 or later, you must use the JVM parameter, -Dhttps.protocols=TLSv1.2.

   "C:\Program Files\Java\jdk1.7.0_51\bin\java.exe" -Dhttps.protocols=TLSv1.2 -jar deployer.jar --install --container-type tomcatv6 --atrium-sso-url https://sso-server.bmc.com:8943/atriumsso --web-app-url https://sso-server.bmc.com:8943/test/index.html --container-base-dir "C:/tomcat6" --admin-name amadmin --admin-pwd admin123  --truststore "C:/tomcat6/conf/cacerts.p12" --truststore-type PKCS12 --truststore-password changeit --jvm-truststore "C:/Program Files/Java/jdk1.7.0_51/jre/lib/security/cacerts" --jvm-truststore-password changeit
   
4. After the agent deployment to the FIPS enabled server, open the cfg.properties file stored at <Tomcat>\atssoAgents\test\cfg.properties location and verify that fipsMode parameter's value is set to true.
5. Modify the network ciphers. For information about the network ciphers, see Changing-FIPS-140-network-ciphers.
6. After saving the changes, restart the container server.
   The web agent updates the encryption level and integration with the BMC Atrium Single Sign-On server.