Mitigation for the Apache Log4j vulnerabilities CVE-2021-44228 and CVE-2021-45046

Issues

A zero-day exploit for the following vulnerabilities was publicly released: 

• CVE-2021-44228 (code named Log4Shell) on December 9, 2021
• CVE-2021-45046 on December 14, 2021

A detailed description of the vulnerabilities can be found here: Apache Log4j Security Vulnerabilities. Please follow the BMC Security Advisory Note on BMC Community for continuous updates and details about this issue.

We recommend that you immediately apply the fix as described in this topic.

Resolution

Download the hotfix required for your platform from the Patches tab of the following EPD website page and apply the hotfix. You must provide your BMC Support credentials to access the EPD website. You might also be prompted to complete the Export Compliance form.

Applying the hotfix

Apply the hotfix to various components in the following sequence:

1. Application Server
2. Console (RCP client)
3. (Optional) PXE 
4. (Optional) Offline Patch Downloader Utility
5. (Optional) Live Reporting
6. (Optional) Smart Hub Gateway

Step 1: Applying the hotfix to the Application Server

Before you begin

Before you start applying the hotfix, do the following for each of the Application Servers:

1. If present, back up and remove the following directories from the <TSSA_INSTALL_DIR>\NSH\br\dbm-rcp\configuration directory:
   • org.eclipse.core.runtime
   • org.eclipse.e4.ui.css.swt.theme
   • org.eclipse.equinox.app
   • org.eclipse.equinox.launcher
   • org.eclipse.osgi
   • org.eclipse.update

2. (Windows Application Servers only) Disable the NSH Proxy, if configured, by running the following command on the Application Server:

   secadmin -m default -p 5 -appserver_protocol clear -T encryption_only -e tls
3. (Windows Application Servers only) Remove the nouser entry (if present) from the rsc\users file.
4. (Windows Application Servers only) Ensure that the user mapping is done correctly:
   1. Launch NSH on the Application Server.
   2. Run the following command against the local host name: agentinfo <AppServerhostName>
   3. Ensure that the User Permissions line of the command output includes an administrative user. For example,
      BladeLogicRSCD@appserver5->Administrator@ appserver5:PrivilegeMapped (Identity via trust)
   4. If the User Permissions line does not include an administrative user, resolve the issue before proceeding further. 

5. Depending on your environment, download and extract the hotfix files to a temporary location (for example, c:\temp):

   Environment	Files to download
   Linux Application Servers and Windows consoles (RCP client)	TSSA_LOG4J_LIN_21-3_HF_v1.zip TSSA_LOG4J_WIN_21-3_HF_v1.zip
   Windows Application Servers and Windows consoles (RCP client)	TSSA_LOG4J_WIN_21-3_HF_v1.zip

To apply the hotfix

1. Perform the following tasks on each of the Application Servers:
   1. Run the RollingUpdate script.
   2. Replace the uninstaller.jar file (if present) .
2. Import the Configuration Objects (COs) into any console (RCP client).

1 (a). Run the RollingUpdate script

Do the following for each of the Application Servers:

1. Log in to the Application Server with a user having administrator privileges.

2. From the c:\temp\TSSA_LOG4J_<Platform>_<version>_HF_v1\Appserver\RollingUpdate directory, open an nsh terminal and execute the following script:

   nsh rollingUpdateInstaller.nsh

   The following message is displayed when the script execution completes successfully:

   DB logging completed successfully.
   #### Rolling Updatation Completed Successfully ####
   ##### Rolling Update Version File modified Successfully #####
   ** Please look for the post installtion steps mentioned in readme file.
   As those need to be done in order to complete the update process.
   ** Please find the log file with name - "rollingUpdateInstaller.<serverName>.log" at location - /opt/bmc/bladelogic/NSH/br. Do attach this log for better support.
   |Contact BMC Bladelogic Support: |
   |Toll-Free:  (800) 537 1813      |
   |EMail: customer_support@bmc.com |

 1 (b): Replace the Uninstaller.jar file (if present)

Do the following for each of the Application Servers:

1. Navigate to the <TSSA_INSTALL_DIR>\UninstallBMCBSAOneClickInstall directory and back up the unistaller.jar file (if present) outside the <TSSA_INSTALL_DIR> directory.
2. Copy the uninstaller.jar file from the c:\temp\TSSA_LOG4J_<Platform>_<version>_HF_v1\Appserver\Uninstaller directory and replace it in the <TSSA_INSTALL_DIR>\UninstallBMCBSAOneClickInstall directory.

 2. Import the COs

Do the following in any of the consoles (RCP clients):

1. Go to Configuration Object Dictionary:
   1. Open the console.
   2. Click Configuration > Config Object Dictionary, and then click the + icon.
      A new window opens.
2. Select Server Object and click Next.
3. Browse to the jpavmware.zip file in the c:\temp\TSSA_LOG4J_<Platform>_<version>_HF_v1\Appserver\CO directory, click Next and then click Finish.
   It takes a while to import.
4. Once CO import is complete, run the Distribute Configuration Objects Job against the target servers where this CO was distributed previously.
5. Repeat steps 1 to 4 with the rhev.zip file (present in the c:\temp\TSSA_LOG4J_<Platform>_<version>_HF_v1\Appserver\CO directory).

Do the following after applying the hotfix to the Application Server

1. Verify that the  environment is running successfully.
2. Remove the following directories and files:
   1. Delete the TSSA_LOG4J_<Platform>_<version>_HF_v1.zip file and its extracted files.
   2. Remove the BSA<version>RU2Backup_<Date>_<Time> backup directory (for example, BSA21.3.00.38RU2Backup_17.12.2021_04.28.58), which gets created when installing the hotfix. Typically, the backup directory is located in the <INSTALL_DIR>\BladeLogic directory.
   3. On the file server, search for the log4j-api-2.13.1.jar and log4j-core-2.13.1.jar files and delete them.
      Typically, these .jar files are located in the following directory: \storage\blassetclasses\<uuid>\implementations\all. For example,
      \storage\blassetclasses\09abfd1b-6c86-4e31-9455-266dd4dd2e2f\implementations\all\log4j-api-2.13.1.jar or log4j-core-2.13.1.jar
   4. After you delete the files, move to their parent <uuid> directory and delete the rhev.zip or jpavmware.zip file (whichever present) from that directory.
3. Delete the unistaller.jar file that you backed up in step 1(b) .

4. (Windows Application Servers only) Enable the NSH Proxy, if required, by running the following command on the Application Server:

   secadmin -m default -p 5 -appserver_protocol ssoproxy -T encryption_only -e tls
5. (Windows Application Servers only) Add the nouser entry to the rsc\users file, if required.

Step 2: Applying the hotfix to the console (RCP client)

Do the following on any of the computers where the console is installed:

1. Close the console and NSH processes.
2. Download and extract the TSSA_LOG4J_Win_<version>_HF_v1.zip file to a temporary location on the computer where the console is installed (for example, c:\temp).
3. Download the rcp_rollingUpdateInstaller_v1.nsh script and copy it to the c:\temp\TSSA_LOG4J_<Platform>_<version>_HF_v1\RCP directory.

4. From the c:\temp\TSSA_LOG4J_<Platform>_<version>_HF_v1\RCP directory, open an nsh terminal and execute the following script:

   nsh rcp_rollingUpdateInstaller_v1.nsh
5. You are prompted to confirm the following details:
   1. Path where the console is installed. Review the path and enter Y to confirm.
   2. Path to the backup directory: Accept the default value (c:\temp\TSSA_LOG4J_<Platform>_<version>_HF_v1\RCP) and press Y. Or, enter the path where you want to create the backup directory and then press Enter. The backup directory is created with the following naming convention: BSA<version>RCP_RU2Backup_<Date>_<Time> (for example, BSA21.3.00.38RU2Backup_17.12.2021_04.28.58).     

6. Press Y if you want to launch the console now, or press N if you want to launch the console later manually.
   The following message is displayed when the script execution completes successfully:
   

   #### RCP Rolling Updatation Completed Successfully ####
   ##### Rolling Update Version File modified Successfully #####
   ** Please look for the post installtion steps mentioned in readme file.
   As those need to be done in order to complete the update process.
   ** Please find the log file with name - "rollingUpdateInstaller.<serverName>.log" at location - /cygdrive/C/Program Files/BMC Software/BladeLogic/NSH/br/ Do attach this log for better support.
   |Contact BMC Bladelogic Support: |
   |Toll-Free:  (800) 537 1813      |
   |EMail: customer_support@bmc.com |
7. (Applicable only if you have multiple instances of the console installed on the same server) By default, the rcp_rollingUpdateInstaller_v1.nsh script applies the hotfix to the console version for which you have downloaded the TSSA_LOG4J_Win_<version>_HF_v1.zip file. If you have multiple instances of the console installed on the same server, do the following:
   1. Download the TSSA_LOG4J_Win_<version>_HF_v1.zip file for the console version to which you want to apply the hotfix.

   2. Open an nsh terminal and run the following command to execute the script for each instance at a time:
      

      nsh rcp_rollingUpdateInstaller_v1.nsh "RCP_INSTALL_DIR"

      For example:

      nsh rcp_rollingUpdateInstaller_v1.nsh "/cygdrive/c/Program Files/BMC Software/BladeLogic 2/NSH"
   3. Confirm details for the script as described in Step 5.

Do the following after applying the hotfix to the console (RCP client)

1. Verify that the  environment is running successfully.
2. Delete the TSSA_LOG4J_<Platform>_<version>_HF_v1.zip file and its extracted files.
3. Delete the backup directory (BSA<version>RCP_RU2Backup_<Date>_<Time>) that is created in Step 5. 

 (Optional) Step 3: Applying the hotfix to PXE

1. Stop the PXE and TFTP services.
2. Back up the following files outside the <PXE_INSTALL_DIR> directory and then delete the files:
   • <PXE_INSTALL_DIR>\br\stdlib\log4j-1.2-api-2.13.1.jar
   • <PXE_INSTALL_DIR>\br\stdlib\log4j-api-2.13.1.jar
   • <PXE_INSTALL_DIR>\br\stdlib\log4j-core-2.13.1.jar
   • <PXE_INSTALL_DIR>\br\dbutility\DBConnectionValidator.bat
   • <PXE_INSTALL_DIR>\br\dbutility\DBConnectionValidator.sh

3. Copy the following files from the c:\temp\TSSA_LOG4J_<Platform>_<version>_HF_v1\ PXE directory to the directories mentioned in the following table:

   File	Directory
   log4j-1.2-api-2.16.0.jar	<PXE_INSTALL_DIR>\br\stdlib
   log4j-api-2.16.0.jar	<PXE_INSTALL_DIR>\br\stdlib
   log4j-core-2.16.0.jar	<PXE_INSTALL_DIR>\br\stdlib
   DBConnectionValidator.bat	<PXE_INSTALL_DIR>\br\dbutility
   DBConnectionValidator.sh	<PXE_INSTALL_DIR>\br\dbutility

4. Navigate to the br\deployments directory, search for the rest.war and autoupgrade-rest.war files and back up the files outside the <PXE_INSTALL_DIR> directory, and delete the files.
   Search example:
   <PXE_INSTALL_DIR>\br\deployments\<default (or)_postmig (or)_pxe (or) custom_deployment_name>\tomcat\webapps\rest.war
   <PXE_INSTALL_DIR>\br\deployments\<default (or)_postmig (or)_pxe (or) custom_deployment_name>\tomcat\webapps\autoupgrade-rest.war
5. Navigate to the br\deployments directory, search for the rest and autoupgrade-rest directories, back up the directories outside the <PXE_INSTALL_DIR> directory, and then delete them from the br\deployments directory. These directories are re-created after the Application Server restarts, or when a REST call is triggered.
   Search example:
   <PXE_INSTALL_DIR>\br\deployments\<default (or)_postmig (or) _pxe (or) custom_deployment_name>\tomcat\webapps\rest
   <PXE_INSTALL_DIR>\br\deployments\<default (or)_postmig (or) _pxe (or) custom_deployment_name>\tomcat\webapps\autoupgrade-rest
6. Copy the rest.war and autoupgrade-rest.war files from the c:\temp\TSSA_LOG4J_<Platform>_<version>_HF_v1 \PXE directory to directories that you found in step 4 results.
7. (Windows) Update the classpath in the Registry with the exact version number of log4j by using one of following methods:
   • (Using a script)
     1. Navigate to the c:\temp\TSSA_LOG4J_<Platform>_<version>_HF_v1 \PXE\registryupdatescript directory.

     2. Run the following command to update the classpath in the Windows Registry.

        PXE64and32ModifyRegClassPath.bat
   • (Manually)
     1. Open the Windows Registry.
     2. Export the following key:
        • (64-bit PXE) [HKEY_LOCAL_MACHINE\SOFTWARE\BladeLogic\PXE\PXE Server\option2]
        • (32-bit PXE) [HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\BladeLogic\PXE\PXE Server\option2]
     3. Open the exported Registry file with a text editor, replace the string 2.13.1 with 2.16.0 , and save the Registry file.
     4. Import the edited Registry file back into the Windows Registry.
8. (Linux) Update the classpath in the required files with the exact version number of log4j by using one of following methods:
   • (Using a script)
     1. Navigate to the /tmp/ TSSA_LOG4J_<Platform>_<version>_HF_v1/ PXE/ registryupdatescript directory.
     2. Update PXE64and32ModifyRegClassPath.sh with <PXE_INSTALL_DIR>.

     3. Run the following command to update the classpath in the required files.

        sh PXE64and32ModifyRegClassPath.sh
   • (Manually) 
     1. Open the following files with a text editor and replace the string 2.13.1 with 2.16.0 and save the files.
        • <PXE_INSTALL_DIR>/pxe/files/files.reg
        • <PXE_INSTALL_DIR>/pxe/br/dbdiagnostics
        • <PXE_INSTALL_DIR>/pxe/br/bljconsole
        • <PXE_INSTALL_DIR>/pxe/br/blasadmin
        • <PXE_INSTALL_DIR>/pxe/br/bltftp
        • <PXE_INSTALL_DIR>/pxe/br/bljconsole-launcher
        • <PXE_INSTALL_DIR>/pxe/br/blciviewer
        • <PXE_INSTALL_DIR>/pxe/br/bljython
        • <PXE_INSTALL_DIR>/pxe/br/postmigration
        • <PXE_INSTALL_DIR>/pxe/br/blcli
        • <PXE_INSTALL_DIR>/pxe/br/blpxe
        • <PXE_INSTALL_DIR>/pxe/br/blcred
        • <PXE_INSTALL_DIR>/pxe/br/blpxeconf
        • <PXE_INSTALL_DIR>/pxe/br/bl_gen_blcli_user_info
        • <PXE_INSTALL_DIR>/pxe/br/blcli-browse
        • <PXE_INSTALL_DIR>/pxe/br/jmxcli
        • <PXE_INSTALL_DIR>/pxe/br/blmkcert
        • <PXE_INSTALL_DIR>/pxe/br/blcli-generate-html
9. Start the PXE and TFTP services.

Do the following after applying the hotfix to PXE

1. Verify that the  environment is running successfully.
2. Delete the TSSA_LOG4J_<Platform>_<version>_HF_v1.zip file and its extracted files.
3. Delete the files that you backed up in Step 2.
4. Delete the rest.war and autoupgrade-rest.war files that you backed up in Step 4.
5. Delete the rest and autoupgrade-rest directories that you backed up in Step 5 .

 (Optional) Step 4: Set up the Offline Patch Downloader utility

Replace the existing Offline Patch Downloader utility with the utility bundled in the hotfix.

Before you begin

Back up the configuration file that you had prepared for the existing utility. 

To replace the existing utility

1. Extract the LOG4J_TSSA_<Platform>_<version>_HF.zip file to a temporary directory. The extracted file contains the following files:
   • All-OS-Patch-Downloaders-aix-build-<version>.tar
   • All-OS-Patch-Downloaders-linux-build-<version>.tar.gz  
   • All-OS-Patch-Downloaders-windows-build-<version>.zip
2. Depending on the platform, extract the compressed files:
   • (Windows) Extract the ZIP files by using a file compression utility.
   • (Linux) Run the following command: tar  -xvf All-OS-Patch-Downloaders-<platform>-<build>-<version>.tar.gz
3. (Linux only) Grant the permission to modify the extracted files:  chmod -R 777 All-OS-Patch-Downloaders-<platform>-<build>-<version>

4. Depending on the platform, use the instructions in the following topics to set up the utility.
   While preparing the configuration file for a platform, use the backed up configuration file as a reference.

   Platform	Topics
   AIX	Patch Downloader utility for IBM AIX
   Linux	Patch Downloader utility for Oracle Solaris Patch Downloader utility for Red Hat Enterprise Linux Patch Downloader utility for SUSE Linux Enterprise Patch Downloader utility for Oracle Enterprise Linux Patch Downloader utility for Ubuntu and Debian Patch downloader utility for CentOS
   Windows	Patch Downloader utility for Microsoft Windows

Do the following after setting up the Offline Patch Downloader utility

1. Verify that the utility is working properly.
2. Delete the existing All-OS-Patch-Downloaders-<platform>-<build>.<extension> package and its extracted files.
3. Delete the TSSA_LOG4J_<Platform>_<version>_HF_v1.zip file and its extracted files.

 (Optional) Step 5: Apply the hotfix to Live Reporting

1. Stop the Live Reporting by using one of the following methods: 
   • (Windows only) Stop the Live Reporting service (if it exists) using the following command: sc.exe stop Yellowfin
   • Run the following command:
     • (Windows) <LiveReporting_INSTALL_DIR>\Yellowfin\appserver\bin\shutdown.bat
     • (Linux) <LiveReporting_INSTALL_DIR>/Yellowfin/appserver/bin/shutdown.sh
2. (Linux only) Run the following command to find out the java process running in your environment:
   ps -ef | grep java
   If you find any processes related to Live Reporting running, kill them.

3. Download the Yellowfin patch 9.4.2.1 (yellowfin-9.4.2.1-20211217-update.jar) from the following URL: Yellowfin Patch.

4. Install the patch:

   1. Run the following command to start the installation:

      <LiveReporting_INSTALL_DIR>/jre/bin/java -jar yellowfin-9.4.2.1-20211217-update.jar

      The installation wizard starts.

   2. Enter <LiveReporting_INSTALL_DIR/Yellowfin as the installation directory path and click Next.
   3. Complete the installation wizard.
5. Start the Live Reporting by using one of the following methods: 
   • (Windows only) Start the Live Reporting service (if it exists) using the following command: sc.exe start Yellowfin
   • Run the following command:
     • (Windows) <LiveReporting_INSTALL_DIR>\Yellowfin\appserver\bin\startup.bat
     • (Linux) <LiveReporting_INSTALL_DIR>/Yellowfin/appserver/bin/startup.sh
6. Verify that the Live Reporting is working, and then delete the backup directory (created by the Yellowfin patch installer) from the <LiveReporting_INSTALL_DIR>\Yellowfin directory.
7. Update the PostInstaller.jar file:
   1. Navigate to the <LiveReporting_INSTALL_DIR>\liveReportingPostInstaller directory and back up the PostInstaller.jar file outside the <LiveReporting_INSTALL_DIR> directory. 
   2. Delete the PostInstaller.jar file from the <LiveReporting_INSTALL_DIR> directory.
   3. Copy the PostInstaller.jar file from the c:\temp\TSSA_LOG4J_<Platform>_<version>_HF_v1\Live Reporting directory to the <Live-LiveReporting_INSTALL_DIR>\liveReportingPostInstaller directory.

Do the following after applying the hotfix to Live Reporting

1. Verify that the Live Reporting environment is running successfully.
2. Delete the TSSA_LOG4J_<Platform>_<version>_HF_v1.zip file and its extracted files.
3. Delete the PostInstaller.jar file that you backed up in Step 7 (a).

 (Optional) Step 6: Applying the hotfix to Smart Hub Gateway

1. Stop the Smart Hub Gateway service.
2. Back up the following files outside the <Smarthub_INSTALL_DIR> directory and then delete the files:
   • <Smarthub_INSTALL_DIR>\smarthub_gateway\lib\log4j-core-2.13.1.jar
   • <Smarthub_INSTALL_DIR>\smarthub_gateway\lib\log4j-api-2.13.1.jar
   • <Smarthub_INSTALL_DIR>\smarthub_gateway\smarthub_gateway.jar

3. Copy the following files from the c:\temp\TSSA_LOG4J_<Platform>_<version>_HF_v1\smarthub_gateway directory to the directories mentioned in the following table:

   File	Directory
   log4j-core-2.16.0.jar	<Smarthub_INSTALL_DIR>\smarthub_gateway\lib
   log4j-api-2.16.0.jar	<Smarthub_INSTALL_DIR>\smarthub_gateway\lib
   smarthub_gateway.jar	<Smarthub_INSTALL_DIR>\smarthub_gateway

4. Start the Smart Hub Gateway service.

Do the following after applying the hotfix to the Smart Hub Gateway

1. Verify that the  environment is running successfully.
2. Delete the TSSA_LOG4J_<Platform>_<version>_HF_v1.zip file and its extracted files.
3. Delete the files that you backed up in Step 2.