Mitigation for the Apache Log4j vulnerabilities CVE-2021-44228 and CVE-2021-45046

Issue

A zero-day exploit for the following vulnerabilities was publicly released: 

• CVE-2021-44228 (code named Log4Shell) on December 9, 2021
• CVE-2021-45046 on December 14, 2021
• CVE-2021-4104 on December 14, 2021
• CVE-2021-45105 on December 18, 2021
• CVE-2021-44832 on December 28, 2021

A detailed description of the vulnerabilities can be found here: Apache Log4j Security Vulnerabilities. Please follow the BMC Security Advisory Note on BMC Community for continuous updates and details about this issue.

We recommend that you immediately apply the fix as described in this topic.

Resolution

Download the hotfix required for your platform from the Patches tab of the following EPD website page and apply the hotfix. You must provide your BMC Support credentials to access the EPD website. You might also be prompted to complete the Export Compliance form.

Applying the hotfix

Apply the hotfix to various components in the following sequence:

1. Application Server
2. Console (RCP client)
3. (Optional) PXE 
4. (Optional) Offline Patch Downloader Utility
5. (Optional) Live Reporting
6. (Optional) Smart Hub Gateway

Step 1: Apply the hotfix to the Application Server

Before you begin

Before you start applying the hotfix, do the following for each of the Application Servers:

1. If present, back up and remove the following directories from the <TSSA_INSTALL_DIR>\NSH\br\dbm-rcp\configuration directory:
   • org.eclipse.core.runtime
   • org.eclipse.e4.ui.css.swt.theme
   • org.eclipse.equinox.app
   • org.eclipse.equinox.launcher
   • org.eclipse.osgi
   • org.eclipse.update

2. (Windows Application Servers only) Disable the NSH Proxy, if configured, by running the following command on the Application Server:

   secadmin -m default -p 5 -appserver_protocol clear -T encryption_only -e tls
3. (Windows Application Servers only) Remove the nouser entry (if present) from the rsc\users file.
4. (Windows Application Servers only) Make sure that the user mapping is done correctly:
   1. Launch NSH on the Application Server.
   2. Run the following command against the local host name: agentinfo <AppServerhostName>
   3. Make sure that the User Permissions line of the command output includes an administrative user. For example,
      BladeLogicRSCD@appserver5->Administrator@ appserver5:PrivilegeMapped (Identity via trust)
   4. If the User Permissions line does not include an administrative user, resolve the issue before proceeding further. 

5. Depending on your environment, download and extract the hotfix files to a temporary location (for example, c:\temp):

   Environment	Files to download
   Linux Application Servers and Windows consoles (RCP client)	TSSA_LOG4J_LIN_<version>_HF_v2.zip TSSA_LOG4J_WIN_<version>_HF_v2.zip
   Windows Application Servers and Windows consoles (RCP client)	TSSA_LOG4J_WIN_<version>_HF_v2.zip

To apply the hotfix to the Application Server

1. Run the RollingUpdate script on each of the Application Servers.
2. Import the Configuration Objects (COs) into any console (RCP client).

1(a). Run the RollingUpdate script

Do the following for each of the Application Servers:

1. Log in to the Application Server with a user having administrator privileges.

2. From the c:\temp\ TSSA_LOG4J_<Platform>_<version>_HF_v2\ Appserver\RollingUpdate directory, open an nsh terminal and execute the following script:

   nsh rollingUpdateInstaller.nsh

   The following message is displayed when the script execution completes successfully. 

   DB logging completed successfully.
   #### Rolling Updatation Completed Successfully ####
   ##### Rolling Update Version File modified Successfully #####
   ** Please look for the post installtion steps mentioned in readme file.
   As those need to be done in order to complete the update process.
   ** Please find the log file with name - "rollingUpdateInstaller.<serverName>.log" at location - /opt/bmc/bladelogic/NSH/br. Do attach this log for better support.
   |Contact BMC Bladelogic Support: |
   |Toll-Free:  (800) 537 1813      |
   |EMail: customer_support@bmc.com |

2. Import the COs

Do the following in any of the consoles (RCP clients):

1. Go to Configuration Object Dictionary:
   1. Open the console.
   2. Click Configuration > Config Object Dictionary, and then click the + icon.
      A new window opens.
2. Select Server Object and click Next.
3. Browse to the jpavmware.zip file in the c:\temp\TSSA_LOG4J_<Platform>_<version>_HF_v2\Appserver\CO directory, click Next and then click Finish.
   It takes a while to import.
4. After the CO import is complete, run the Distribute Configuration Objects Job against the target servers where this CO was distributed previously.
5. Repeat steps 1 to 4 with the rhev.zip file, present in the c:\temp\TSSA_LOG4J_<Platform>_<version>_HF_v2\Appserver\CO directory.

Do the following after applying the hotfix to the Application Server

1. Verify that the TrueSight Server Automation environment is running successfully.
2. Remove the following directories and files:
   1. Delete the TSSA_LOG4J_<Platform>_<version>_HF_v2.zip file and its extracted files.
   2. Remove the BSA<version>L4JBackup_<Date>_<Time> backup directory (for example, BSA21.01.01.38L4JBackup_17.12.2021_04.28.58), which gets created when installing the hotfix. Typically, the backup directory is located in the <TSSA_INSTALL_DIR>\BladeLogic directory.
   3. On the file server, search for the log4j-api-2.13.1.jar, log4j-core-2.13.1.jar, log4j-api-2.16.0.jar, and log4j-core-2.16.0.jar files and delete them.
      Typically, these .jar files are located in the following directory: \storage\blassetclasses\<uuid>\implementations\all. For example,
      \storage\blassetclasses\09abfd1b-6c86-4e31-9455-266dd4dd2e2f\implementations\all\log4j-api-2.13.1.jar, log4j-api-2.16.0.jar, log4j-core-2.16.0.jar or log4j-core-2.13.1.jar.
   4. After you delete the files, move to their parent <uuid> directory and delete the rhev.zip or jpavmware.zip file (whichever present) from that directory.

3. (Windows Application Servers only) Enable the NSH Proxy, if required, by running the following command on the Application Server:

   secadmin -m default -p 5 -appserver_protocol ssoproxy -T encryption_only -e tls
4. (Windows Application Servers only) Add the nouser entry to the rsc\users file, if required.

Step 2: Apply the hotfix to the console (RCP client)

Depending on which Server Automation components are installed on the console computer, use one of the following methods to apply the hotfix:

• The console is installed, but the Application Server is not installed on the same computer
• Both the Windows 64-bit console and the Application Server are installed on the same computer (not recommended)

Do the following on any of the computers where the console is installed, but the Application Server is not installed:

1. Download and extract the TSSA_LOG4J_Win_<version>_HF_v2.zip file to a temporary location on the computer where the console is installed (for example, c:\temp).
2. From the c:\temp\TSSA_LOG4J_<Platform>_<version>_HF_v2\RCP directory, run the following command:
   
   • (21.02.01, Windows) 
     
     • (32-bit) x32\TSSACONSOLE210201-224-WIN32.exe
     • (64-bit) x64\TSSACONSOLE210201-224-WIN64.exe
   • (21.02.01, Linux, 64-bit) ./TSSACONSOLE210201-224-LIN64.bin
     After the command executes successfully, Console version is updated to 21.2.00.224.
     
   • (21.02.00, Windows) 
     
     • (32-bit) x32\TSSACONSOLE2102-169-WIN32.exe
     • (64-bit) x64\TSSACONSOLE2102-169-WIN64.exe
   • (21.02.00, Linux, 64-bit) ./TSSACONSOLE2102-169-LIN64.bin
     After the command executes successfully, Console version is updated to 21.2.00.169.

Do the following if the Windows 64-bit Console and the Application Server are installed on same server:

1. Close the console and NSH processes.
2. Download and extract the TSSA_LOG4J_Win_<version>_HF_v2.zip file to a temporary location on the computer where the console is installed (for example, c:\temp).

3. From the c:\temp\TSSA_LOG4J_<Platform>_<version>_HF_v2\RCP\x64\RCP_Appserver directory, open an nsh terminal and execute the following script:

   nsh rcp_rollingUpdateInstaller_v2.nsh
4. You are prompted to confirm the following details:
   1. Path where the console is installed. Review the path and enter Y to confirm.
   2. Path to the backup directory: Accept the default value (c:\temp\TSSA_LOG4J_<Platform>_<version>_HF_v2\RCP) and press Y. Or, enter the path where you want to create the backup directory and then press Enter. The backup directory is created with the following naming convention: BSA<version>RCP_L4JBackup_<Date>_<Time> (for example, BSA20.02.01.38L4JBackup_17.12.2021_04.28.58).     

5. After the hotfix installation completes, you are prompted to launch the console. Press Y if you want to launch the console now, or press N if you want to launch the console later manually.
   The following message is displayed when the script execution completes successfully:
   

   #### RCP Rolling Update is Completed Successfully ####
   ##### Rolling Update Version File modified Successfully #####
   ** Please look for the post installtion steps mentioned in readme file.
   As those need to be done in order to complete the update process.
   ** Please find the log file with name - "rollingUpdateInstaller.<serverName>.log" at location - /cygdrive/C/Program Files/BMC Software/BladeLogic/NSH/br/ Do attach this log for better support.
   |Contact BMC Bladelogic Support: |
   |Toll-Free:  (800) 537 1813      |
   |EMail: customer_support@bmc.com |

   Note that Console version remains the same as before applying the hotfix, 21.02.01.211 (for 21.02.01) and 21.02.00.162 (for 21.02.00)

Do the following after applying the hotfix to the console (RCP client)

1. Verify that the TrueSight Server Automation environment is running successfully.
2. Delete the TSSA_LOG4J_<Platform>_<version>_HF_v2.zip file and its extracted files.
3. Delete the backup directory (BSA<version>RCP_L4JBackup_<Date>_<Time>) that is created in Step 5. 

 (Optional) Step 3: Apply the hotfix to PXE

1. Stop the PXE and TFTP services.
2. Back up the following files outside the <PXE_INSTALL_DIR> directory and then delete the files:
   • <PXE_INSTALL_DIR>\br\stdlib\log4j-api-2.13.1.jar or log4j-api-2.16.0.jar
     
   • <PXE_INSTALL_DIR>\br\stdlib\log4j-1.2-api-2.13.1.jar or log4j-1.2-api-2.16.0.jar
     
   • <PXE_INSTALL_DIR>\br\stdlib\log4j-core-2.13.1.jar or log4j-core-2.16.0.jar
   • <PXE_INSTALL_DIR>\br\dbutility\DBConnectionValidator.bat
   • <PXE_INSTALL_DIR>\br\dbutility\DBConnectionValidator.sh

3. Copy the following files from the c:\temp\TSSA_LOG4J_<Platform>_<version>_HF_v2\ PXE directory to the directories mentioned in the following table:

   File	Directory
   log4j-api-2.17.1.jar	<PXE_INSTALL_DIR>\br\stdlib
   log4j-1.2-api-2.17.1.jar	<PXE_INSTALL_DIR>\br\stdlib
   log4j-core-2.17.1.jar	<PXE_INSTALL_DIR>\br\stdlib
   DBConnectionValidator.bat	<PXE_INSTALL_DIR>\br\dbutility
   DBConnectionValidator.sh	<PXE_INSTALL_DIR>\br\dbutility

4. Navigate to the br\deployments directory, search for the rest.war file and back up the file outside the <PXE_INSTALL_DIR> directory, and delete the file from the <PXE_INSTALL_DIR> directory. 
   Search example:
   <PXE_INSTALL_DIR>\br\deployments\<default (or)_postmig (or)_pxe (or) custom_deployment_name>\tomcat\webapps\rest.war
5. Navigate to the br\deployments directory, search for the rest directory, back up the directory outside the <PXE_INSTALL_DIR> directory, and then delete the directory from the br\deployments directory. This directory is re-created after the Application Server restarts, or when a REST call is triggered.
   Search example:
   <PXE_INSTALL_DIR>\br\deployments\<default (or)_postmig (or) _pxe (or) custom_deployment_name>\tomcat\webapps\rest
6. Copy the rest.war file from the c:\temp\TSSA_LOG4J_<Platform>_<version>_HF_v2\PXE directory to directories that you found in step 4 results.
7. (Windows) Update the classpath in the Registry with the exact version number of log4j by using one of the following methods:
   • (Using a script)
     1. Navigate to the c:\temp\TSSA_LOG4J_<Platform>_<version>_HF_v2\PXE\registryupdatescript directory.

     2. Run the following command to update the classpath in the Windows Registry.

        PXE64and32ModifyRegClassPath.bat
   • (Manually)
     1. Open the Windows Registry.
     2. Export the following key:
        • (64-bit PXE) [HKEY_LOCAL_MACHINE\SOFTWARE\BladeLogic\PXE\PXE Server\option2]
        • (32-bit PXE) [HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\BladeLogic\PXE\PXE Server\option2]
     3. Open the exported Registry file with a text editor, replace the string 2.13.1 or 2.16.0 with 2.17.1, and save the Registry file.
     4. Import the edited Registry file back into the Windows Registry.
8. (Linux) Update the classpath in the required files with the exact version number of log4 by using one of following methods:
   • (Using a script)
     1. Navigate to the /tmp/ TSSA_LOG4J_<Platform>_<version>_HF_v2/ PXE/ registryupdatescript directory.
     2. Update PXE64and32ModifyRegClassPath.sh with <PXE_INSTALL_DIR>.

     3. Run the following command to update the classpath in the required files.

        sh PXE64and32ModifyRegClassPath.sh
   • (Manually) 
     1. Open the following files with a text editor and replace the string 2.13.1 or 2.16.0 with 2.17.1, and save the files.
        • <PXE_INSTALL_DIR>/pxe/files/files.reg
        • <PXE_INSTALL_DIR>/pxe/br/dbdiagnostics
        • <PXE_INSTALL_DIR>/pxe/br/bljconsole
        • <PXE_INSTALL_DIR>/pxe/br/blasadmin
        • <PXE_INSTALL_DIR>/pxe/br/bltftp
        • <PXE_INSTALL_DIR>/pxe/br/bljconsole-launcher
        • <PXE_INSTALL_DIR>/pxe/br/blciviewer
        • <PXE_INSTALL_DIR>/pxe/br/bljython
        • <PXE_INSTALL_DIR>/pxe/br/postmigration
        • <PXE_INSTALL_DIR>/pxe/br/blcli
        • <PXE_INSTALL_DIR>/pxe/br/blpxe
        • <PXE_INSTALL_DIR>/pxe/br/blcred
        • <PXE_INSTALL_DIR>/pxe/br/blpxeconf
        • <PXE_INSTALL_DIR>/pxe/br/bl_gen_blcli_user_info
        • <PXE_INSTALL_DIR>/pxe/br/blcli-browse
        • <PXE_INSTALL_DIR>/pxe/br/jmxcli
        • <PXE_INSTALL_DIR>/pxe/br/blmkcert
        • <PXE_INSTALL_DIR>/pxe/br/blcli-generate-html
9. Start the PXE and TFTP services.

Do the following after applying the hotfix to PXE

1. Verify that the TrueSight Server Automation environment is running successfully.
2. Delete the TSSA_LOG4J_<Platform>_<version>_HF_v2.zip file and its extracted files.
3. Delete the files that you backed up in Step 2.
4. Delete the rest.war file that you backed up in Step 4.
5. Delete the rest directory that you backed up in Step 5.

 (Optional) Step 4: Set up the Offline Patch Downloader utility

Replace the existing Offline Patch Downloader utility with the utility bundled in the hotfix.

Before you begin

Back up the configuration file that you had prepared for the existing utility. 

To replace the existing utility

1. Extract LOG4J_TSSA_<Platform>_<version>_HF.zip to a temporary directory. The extracted directory contains the following files:
   • All-OS-Patch-Downloaders-aix-build-<version>.tar
   • All-OS-Patch-Downloaders-linux-build-<version>.tar.gz  
   • All-OS-Patch-Downloaders-windows-build-<version>.zip
2. Depending on the platform, extract the compressed files:
   • (Windows) Extract the ZIP files using a file compression utility.
   • (Linux) Run the following command: tar  -xvf All-OS-Patch-Downloaders-<platform>-<build>-<version>.tar.gz
3. (Linux only) Grant the permission to modify the extracted files: chmod -R 777 All-OS-Patch-Downloaders-<platform>-<build>-<version>

4. Depending on the platform, use the instructions in the following topics to set up the utility.
   While preparing the configuration file for a platform, use the backed up configuration file as a reference.

   Platform	Topics
   AIX	Patch-Downloader-utility-for-IBM-AIX
   Linux	Patch-Downloader-utility-for-Oracle-Solaris Patch-Downloader-utility-for-Red-Hat-Enterprise-Linux Patch-Downloader-utility-for-SUSE-Linux-Enterprise Patch-Downloader-utility-for-Oracle-Enterprise-Linux Patch-Downloader-utility-for-Ubuntu-and-Debian Patch downloader utility for CentOS
   Windows	Patch-Downloader-utility-for-Microsoft-Windows

Do the following after setting up the Offline Patch Downloader utility

1. Verify that the utility is working properly.
2. Delete the existing All-OS-Patch-Downloaders- <platform>-<build>.<extension> package and its extracted files.
3. Delete the TSSA_LOG4J_<Platform>_<version>_HF_v2.zip file and its extracted files.

 (Optional) Step 5: Apply the hotfix to Live Reporting

1. Stop the Live Reporting by using one of the following methods: 
   • (Windows only) Stop the Live Reporting service (if it exists) by using the following command: sc.exe stop Yellowfin
   • Run the following command:
     • (Windows) <LiveReporting_INSTALL_DIR>\Yellowfin\appserver\bin\shutdown.bat
     • (Linux) <LiveReporting_INSTALL_DIR>/Yellowfin/appserver/bin/shutdown.sh
2. (Linux only) Run the following command to find the java processes running in your environment:
   ps -ef | grep java
   If you find any processes related to Live Reporting running, kill them.
3. Back up the following files outside the <LiveReporting_INSTALL_DIR> directory and then delete the files from the <Live-reporting-install-directory>\Yellowfin\appserver\webapps\ROOT\WEB-INF\lib directory.
   • <Live-reporting-install-directory>\Yellowfin\appserver\webapps\ROOT\WEB-INF\lib\log4j-1.2-api-2.13.3.jar or log4j-1.2-api-2.15.0.jar or log4j-1.2-api-2.16.0.jar
   • <Live-reporting-install-directory>\Yellowfin\appserver\webapps\ROOT\WEB-INF\lib\log4j-api-2.13.3.jar or log4j-api-2.15.0.jar or log4j-api-2.16.0.jar
   • <Live-reporting-install-directory>\Yellowfin\appserver\webapps\ROOT\WEB-INF\lib\log4j-core-2.13.3.jar or log4j-core-2.15.0.jar or log4j-core-2.16.0.jar
   • <Live-reporting-install-directory>\Yellowfin\appserver\webapps\ROOT\WEB-INF\lib\log4j-web-2.13.3.jar or log4j-web-2.15.0.jar or log4j-web-2.16.0.jar
4. Copy the following files from the c:\temp\TSSA_LOG4J_<Platform>_<version>_HF_v2\Live Reporting directory to the <Live-reporting-install-directory>\Yellowfin\appserver\webapps\ROOT\WEB-INF\lib directory:
   • log4j-1.2-api-2.17.1.jar
   • log4j-api-2.17.1.jar
   • log4j-core-2.17.1.jar
   • log4j-web-2.17.1.jar
5. Start the Live Reporting by using one of the following methods: 
   • (Windows only) Start the Live Reporting service (if it exists) using the following command: sc.exe start Yellowfin
   • Run the following command:
     • (Windows) <LiveReporting_INSTALL_DIR>\Yellowfin\appserver\bin\startup.bat
     • (Linux) <LiveReporting_INSTALL_DIR>/Yellowfin/appserver/bin/startup.sh
6. Update the PostInstaller.jar file:
   1. Navigate to the <LiveReporting_INSTALL_DIR>\liveReportingPostInstaller directory and back up the PostInstaller.jar file outside the <LiveReporting_INSTALL_DIR> directory. 
   2. Delete the PostInstaller.jar file from the <LiveReporting_INSTALL_DIR> directory.
   3. Copy the PostInstaller.jar file from the c:\temp\TSSA_LOG4J_<Platform>_<version>_HF_v2\Live Reporting directory to the <Live-LiveReporting_INSTALL_DIR>\liveReportingPostInstaller directory.

Do the following after applying the hotfix to Live Reporting

1. Verify that the Live Reporting environment is running successfully.
2. Delete the TSSA_LOG4J_<Platform>_<version>_HF_v2.zip file and its extracted files.
3. Delete the PostInstaller.jar file that you backed up in Step 6 (a).

 (Optional) Step 6: Apply the hotfix to Smart Hub Gateway

1. Stop the Smart Hub Gateway service.
2. Back up the following files outside the <Smarthub_INSTALL_DIR> directory and then delete the files:
   • <Smarthub_INSTALL_DIR>\smarthub_gateway\lib\log4j-core-2.13.1.jar or log4j-core-2.16.0.jar
   • <Smarthub_INSTALL_DIR>\smarthub_gateway\lib\log4j-api-2.13.1.jar or log4j-api-2.16.0.jar
   • <Smarthub_INSTALL_DIR>\smarthub_gateway\smarthub_gateway.jar

3. Copy the following files from the c:\temp\TSSA_LOG4J_<Platform>_<version>_HF_v2\smarthub_gateway directory to the directories mentioned in the following table:

   File	Directory
   log4j-core-2.17.1.jar	<Smarthub_INSTALL_DIR>\smarthub_gateway\lib
   log4j-api-2.17.1.jar	<Smarthub_INSTALL_DIR>\smarthub_gateway\lib
   smarthub_gateway.jar	<Smarthub_INSTALL_DIR>\smarthub_gateway

4. Start the Smart Hub Gateway service.

Do the following after applying the hotfix to the Smart Hub Gateway

1. Verify that the TrueSight Server Automation environment is running successfully.
2. Delete the TSSA_LOG4J_<Platform>_<version>_HF_v2.zip file and its extracted files.
3. Delete the files that you backed up in Step 2.