Troubleshooting upgrade issues

The preferred method for upgrading a Patch, Service Pack, or version is from the appliance’s UI. For more information, see Performing-the-upgrade.

Upgrade stops with "Entering emergency mode" message

Situation

Very rarely the upgrade to an Oracle Linux 9 based appliance (BMC Discovery 23.3 and later) stops, and the console displays output that looks like the following screenshot:

  In particular, in the screenshot, note:

1. The error RPM: error: db5 error(-nnnnn) from dbenv->open at the top of the console.
2. The "Entering emergency mode" error at the bottom.

Root Cause

The problem occurs when the RPM database's lock files become desynchronised during the installation of packages. The desynchronization causes this phase of the upgrade to abort.

The problem is not a cause for concern. The upgrade from a version of BMC Discovery earlier than version 23.3 to BMC Discovery 23.3 or later requires an operating system upgrade to Oracle Linux 9. This part of the upgrade is safe: the process creates a sandboxed copy of the operating system files that it upgrades in isolation, and it only copies those files into the actual operating system if it is successful. The failure means that the sandbox has become corrupted. Continuing the upgrade deletes the corrupted sandbox and creates a new one, resolving the problem.

Resolution

To enable the upgrade to retry the phase and continue:

1. At the console, type exit to exit emergency mode.  The host reboots.
2. Once rebooted, log in to the machine as the tideway  user.
3. Run the following command:
   sudo rm /usr/tideway/var/upgrade/extract/ol9_upgrade/last_script
   Removing this file clears the error condition caused by the failure and permits the upgrade process to try again.
4. Reboot the machine with the following command: sudo reboot now 
   When the host reboots, the upgrade process retries the failed phase.

Steps after upgrade

Recovering from a lost connection using screen

If you lose the connection to the appliance during the upgrade and you have used screen, you can reconnect to the appliance and recover the virtual terminal running the upgrade. To do this:

1. Reconnect to the appliance and login as the tideway user.

2. List the current screen sessions. Enter:

   [tideway@appliance01 ~]$ screen -ls
   There is a screen on:
   23274.pts-0.appliance01 (Detached)
   1 Socket in /var/run/screen/S-tideway.
   

   You can re-attach to it with a simple command:

   [tideway@appliance01 ~]$ screen -r

   The virtual terminal is recovered and you can see how the upgrade is progressing.