Special issues for patch management

The following topics present special issues and limitations that exist while creating a patch catalog or running a patching job. Most issues are caused due to specific third-party patches, and have been classified based on the target that it is deployed on:

Tip

See Troubleshooting-Patch-Management-issues for additional issues.

Issues while patching on Red Hat Enterprise Linux

The following issue exists while patching on Red Hat Enterprise Linux:

  • The patch analysis jobs on RHEL 7 servers end successfully, but cannot find any security patches on the servers, whereas these security updates are applicable on this server.
    The create_repo_wrapper.log shows:
    /data/PatchRepo/RHEL6-7-8/catalog_2000385.part/create_repo_wrapper.sh: line 409: modifyrepo: command not found
    Workaround: Ensure createrepo package version is later than 0.9.6 (for example, 0.16.2) and rerun the jobs.
  • You maybe unable to install or update the open-vm-tools-deploypkg (from the VMware repository) on a RHEL7 target due to a known issue with this patch. See https://access.redhat.com/solutions/1449563, for more information about this known issue.
    Workaround: You can try excluding the all open-vm-tool patches during the patch analysis job or removing open-vm-tools-deploypkg from the target server.
  • Run the ldconfig command on your RHEL 7 target, if you encounter the following type of error during patch analysis:
    Error 03/07/2017 12:22:25 STDERR: There was a problem importing one of the Python modulesrequired to run yum. The error leading to this problem was:   libcurl.so.4: cannot open shared object file: No such file or directoryPlease install a package which provides this module, orverify that the module is installed correctly.It's possible that the above module doesn't match thecurrent version of Python, which is:2.7.5 (default, Feb 11 2014, 07:46:25) [GCC 4.8.2 20140120 (Red Hat 4.8.2-13)]If you cannot solve this problem yourself, please go to the yum faq at:  http://yum.baseurl.org/wiki/Faq  ERROR::cmd: yum -c "./yum.conf"
  • We need to run the ldconfig  command on the machine (ldconfig is a command that is used to maintain the shared library cache. This cache is typically stored in the file /etc/ld.so.cache and is used by the system to map a shared library name to the location of the corresponding shared library file)
  • To cancel an Red Hat patch catalog job that is running with the CDN download option, you must perform the following steps:
    1. Cancel the Catalog Update Job (CUJ) from the Console to stop the job running on the Application Server.
    2. Log on to the repository server and manually kill the CDN download process.

  • The Patch Analysis job fails with the following error on a few Linux target servers when run against a bulk of target servers. For example, 69 failures of 1500 targets.

    STDERR: error: rpmdb: BDB0113 Thread/process 25671/140650515679232 failed: BDB1507 Thread died in Berkeley DB libraryerror: db5 error(-30973) from dbenv->failchk: BDB0087 DB_RUNRECOVERY: Fatal error, run database recoveryerror: cannot open Packages index using db5 -  (-30973)error: cannot open Packages database in /var/lib/rpmCRITICAL:yum.main:Error: rpmdb open failederror: rpmdb: BDB0113 Thread/process 25671/140650515679232 failed: BDB1507 Thread died in Berkeley DB libraryerror: db5 error(-30973) from dbenv->failchk: BDB0087 DB_RUNRECOVERY: Fatal error, run database recoveryerror: cannot open Packages index using db5 -  (-30973)error: cannot open Packages database in /var/lib/rpmCRITICAL:yum.main:Error: rpmdb open failedERROR::cmd: yum -c "./yum.conf" -C list > "/var/tmp/stage/LinuxCatalog_2004521_psrl-rh7-1255/yum.lst" failed!

    Root cause: These rpmdb errors occur during package management (yum/rpm operations) when the RPM database is corrupted.
    Workaround: Perform the following steps to backup and rebuild the local rpmdb database:

    1. (Execute only if stale yum process exists) Reboot the target servers where the job failed.

    2. Execute the following cleanup script on the rebooted target servers using a script or a BLPackage:

      mkdir /var/lib/rpm/backup
      cp -a /var/lib/rpm/__db* /var/lib/rpm/backup/
      rm -f /var/lib/rpm/__db.[0-9][0-9]*
      rpm --quiet -qa
      rpm --rebuilddb
      yum clean all

Issues while patching on SUSE Enterprise Linux

The following issues and limitations exist while patching on SUSE Enterprise Linux:

  • When you upgrade to the latest version from versions earlier than 8.9, and use the Catalog Update Job, the Job fails.

    Workaround: To solve this issue, perform the following steps:
  1. Log on to the BMC Bladelogic Server Automation Console using the RBACAdmin user.
  2. Go to Users.
  3. Double-click the <PatchingUser>.
  4. On the Role Selection tab, from the list of available roles, add any role.
  5. Click File > Save to save the user information.
  6. Remove the role that you added in step 4.
  7. Click File > Save to save the user information.
  8. Run the Catalog Update Job again.
  • To cancel SuSE 12 patch catalog job that is running on the Subscription Management Tool (SMT)-configured repository, you must perform the following steps:
    1. Cancel the Catalog Update Job (CUJ) from the Console to stop the job running on the Application Server.
    2. Log on to the repository server and manually kill the SMT download process.
  • While creating a SuSE Linux Catalog, certain rpm files might cause the Catalog Update Job to fail, with the following message: "BLPAT1700 - RPM verification failed on selected repository location. Please use SuSE platform repository location and run the catalog.". This issue is due to SuSE Enterprise Linux limitations. For this Catalog Update Job, ensure that you specify a repository location on a SuSE Enterprise Linux host computer.
  • While you are creating a SuSE Linux Catalog in the Online mode, the Catalog fails with an rpm-python dependency. The error message states rpm-python needed by createrepo-0.4.6-1.el4.rf.noarch.

    Workaround: You must check the version of create-repo and python-urlgrabber in the repository server and either ensure that the lower versions are installed or remove the higher versions, if they exist. In addition, you must ensure that rpm-python exists on the system and the version of rpm-python is compatible with the create-repo version that is shipped with the product.

  • A SuSE Patch Analysis Job with Install mode and an include list (include all RMP) using Zypper tries to copy the linux_zypper_analyse.sh file from the wrong Application Server location.

  • The Application Server uses the value of the NSH directory from the /c/Windows/rsc/HOME value.

    Workaround: If you encounter this error, ensure that the registry keys and home value reflect the correct Application Server installation location (for example, C:\Program Files\BMC Software\BladeLogic\appserver\NSH\)

  • SuSE's Subscription Management Tool (SMT) internally maintains metadata in a MySQL database on the repository server. By default, SMT creates the MySQL database using the latin1 character set and latin1_swedish_ci collation. Because the  latin1 character set does not support multibyte characters, you will encounter an error if your SuSE patch description contains a multibyte character.

    Workaround: To solve this issue, you must change the character encoding for the DESCRIPTION column in the Patches table of the MySQL database, as follows:

    1. Log on to the MySQL client on the repository server using the following command:
      mysql -u <userName> -D <databaseName> -p
      For example: mysql -u smt -D smt -p
    2. Run the following ALTER command
      ALTER TABLE Patches CHANGE DESCRIPTION DESCRIPTION text CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NOT NULL;

Issues while patching on AIX

The following issues and limitations exist while patching on AIX:

  • A Patch Analysis Job fails with the following error if certain filesets are missing, and the error does not list the missing filesets.
    There are <n> filesets deferred. Please deploy the missing filesets and rerun analysis for the deferred filesets.
    Workaround: Run the Patch Analysis Job in the debug mode, as described on Obtaining-job-diagnostics. The job logs are stored in the <installDirectory>/NSH/tmp/debug/<deploymentName>/<jobName>/<timestamp>/<date>/<targetServer> directory. The analysis.out log file contains information related to the filesets deferred.
  • For AIX, the Patch Analysis Job fails in the Install Mode because of missing filesets in the repository. A single repository will contain updates for all available packages. The Install Mode is intended to install new packages on the target. Hence, BMC recommends the use of Install Mode on selective packages, rather than using the Install Mode on the complete repository.
  • To cancel an AIX patch catalog job that is running with the SUMA download option, you must perform the following steps:
    1. Cancel the Catalog Update Job (CUJ) from the Console to stop the job running on the Application Server.
    2. Log on to the repository server and manually kill the SUMA download process.

  • AIX remediation job fails to deploy 'rsct.lapi.rte 3.1.6.0' on some targets due to a problem in the third-party patch. For more information about this issue, see the following third-party resource:

    https://www-304.ibm.com/support/docview.wss?uid=isg1IZ80922.

  • AIX remediation job fails to deploy 'idsldap.srv64bit62.rte 6.2.0.16' on some targets due to a problem in the third-party patch.

  • AIX remediation job fails to deploy 'rsct.lapi.bsr.bsrpd.odmdel' (rsct.lapi.bsr 3.1.5.0) on some targets due to a problem in the third-party patch. For more information about this issue, see the following third-party resource:

    https://www-304.ibm.com/support/docview.wss?uid=isg1IZ80922.

  • After 'bos.rte.archive 5.3.0.51' is deployed on an AIX 5.3 target, the target goes into reboot mode and fails to restart.

  • After deploying AIX 6.1 TL9SP1 on a target, the 'bos.rte.install 6.1.9.1' fileset enters a broken state if you try to Undo the deploy job. This 'bos.rte.install 6.1.9.1' fileset enters a broken state even if you reject the fileset manually.

    Workaround: To fix the 'bos.rte.install 6.1.9.1' fileset in the broken state you can take the following corrective actions:

    • If the broken fileset is an update, reinstall the update using options in the installation program that will apply, commit, and automatically install the required software without saving the replaced files (-acgN flags).
    • Reinstall the entire fileset at the base level with the desired updates by using the FORCE option (-F flag).

    If you are still unable to fix the fileset in the broken state, contact IBM support team for further assistance.

  • Error while extracting the offline downloader tar package, because of the length of file path input string.
    Workaround: BMC recommends using the GNU tar (gtar) utility to extract the offline downloader package.
  • You might encounter dependency errors while performing patch analysis on AIX 6.1 TL09 for security fixes. Before you run the analysis job, add AIX 6.1 TL08 to the catalog filter and run the catalog update job to add AIX 6.1 TL08 content to the catalog.
  • You cannot edit a patch catalog of a deprecated AIX platform version, even if the patch catalog was created in an earlier version of TrueSight Server Automation. For more information about which AIX platform versions are supported for patching, see the System-requirements section.

Issues while patching on Oracle Solaris

The following issues and limitations exist while patching on Oracle Solaris:

  • Error while extracting the offline downloader tar package, because of length of file path input string.
    Workaround: BMC recommends using the GNU tar (gtar) utility to extract the offline downloader package. 
  • In case of an outage for the Solaris Patch Download URL, update the solaris.download.base.url property in the patch-psu.properties file with a working URL:
    1. Navigate to the <AppServerInstallDir>/NSH/share/patch/resources directory for the online patch catalogs and the <unzipped-downloader>/resources directory for the offline patch catalogs.
    2. Update the property with the working URL:
      #Solaris Patch Download URL
      solaris.download.base.url=<workingDownloadURL> 
    3. Restart the Application Server service.

Issues while patching on Oracle Enterprise Linux

The following issues and limitations exist while patching on Oracle Enterprise Linux:

  • For Oracle Enterprise Linux, the Patch Analysis Job fails in the Install Mode because of missing rpms in the repository. A single repository will contain updates for all available packages. The Install Mode is intended to install new packages on the target. Hence, BMC recommends the use of Install Mode on selective packages, rather than using the Install Mode on the complete repository.

Issues while patching on Windows

The following issues and limitations exist while patching on Windows:

  • Before you apply multiple hotfixes, you must check whether the hotfix changes are available in the latest released service pack. This is because, Microsoft recommends that it is better to apply the latest service pack instead of applying several hofixes on the server. For best practices while applying patches and hotfixes, see MSDN - Best Practices for Applying Service Packs, Hotfixes and Security Patches.
  • If patch KB2810009 is found missing during patch analysis, BMC recommends that you first separately deploy patch KB2810009 and then re-run the remediation job for the other missing patches. This KB2810009 missing patch needs to be deployed separately because of dependency issues.
  • TrueSight Server Automation automatically creates the required pre-installation and post-installation environment on a Windows target server for patching Java installation files. However, TrueSight Server Automation does not decide whether to reboot the Windows target server.. The target must be rebooted after each version of Java installation patch is deployed on the target. You can specify this reboot option while creating a Deploy Job, see Deploy-Job-Job-Options for more information.

  • Patch MS18-02-SO81 might not be found missing on Windows server 2012 if the QualityCompat registry key is not present. For more information, see windows-security-updates-and-antivirus-software in the Microsoft documentation and Knowledge Article ID: 000150075.

    Key="HKEY_LOCAL_MACHINE" Subkey="SOFTWARE\Microsoft\Windows\CurrentVersion\QualityCompat" Value="cadca5fe-87d3-4b96-b7fb-a231484277cc" Type="REG_DWORD"

  • While performing patch analysis, you may sometimes encounter the following error message in the Patching Job log: 

Error: Encountered error 0x800710dd initializing scanner - The operation identifier is not valid. Possible cause is: Signature verification certificates may not have been installed on this server. Re-run the patching job in debug mode and check log file AnalysisTrace.log for further details.
Error: Unable to initialize analysis engine. 
Error: Analysis failed."


    • On certain targets, BlPatchCheck2 fails with error 0x800710dd, when initializing the Shavlik engine and the analysis DPD trace file AnalysisTrace.log (earlier named as Trace.txt) contains an error line containing the message – “Initialize invalid operation: class STCore::CInvalidOperationException at Opc.cpp:1522: Signing certificate validation failed in '<C:\Program Files\BMC Software\BladeLogic\RSCD\tmp\....\WindowsPatchData.zip>”
    • Root cause: New Shavlik content is SHA2-signed, whose signature verification requires certain PKI certificates to be present on the target. These certificates are part of the Microsoft Trusted Root Certificates program and typically require internet access to obtain. If the system being patched does not have internet access or is otherwise prevented from receiving the Root Certificate updates this error can occur.

    • Workaround: Check if the following two certificates are installed on the target machine by performing the following steps:

      1. Go to Start > Run > MMC.exe. A Console Root window opens.
      2. Select File > Add/remove Snap-in.
      3. Select Certificates from the list of available Snap ins.
      4. Check for 'DigiCert Assured ID Root CA' under Trusted Root Certification Authorities\Certificates. 
      5. Check for 'DigiCert SHA2 Assured ID Code Signing CA' under Intermediate Certification Authorities\Certificates. 

      If the certificates are not installed on the target machine, you must install DigiCertAssuredIDRootCA.crtand DigiCertSHA2AssuredIDCodeSigningCA.crt. Perform the following steps for both the certificate files, first for DigiCertAssuredIDRootCA.crt, and repeat the steps for DigiCertSHA2AssuredIDCodeSigningCA.crt:

      1. Download the certificate.
      2. Click Trusted Root Certification Authorities > All Tasks > Import.
      3. Click Next and browse to the location where you have downloaded the certificate.
      4. Select the certificate, you want to import.
      5. Click Next and select the certificate location.
      6. Click Next and Finish.

      Check if the error is addressed after ensuring that the above certificates are installed. 

  • While performing patch analysis, 0 products / patches were detected after running the Patching Job.
    Workaround: Check if you are running patch analysis on the target server using administrative privileges for that target server.

Issues while patching on Ubuntu

You might encounter the following issues while patching on Ubuntu.

Job gets stuck during the Commit phase

When you run a patching job on an Ubuntu 16.04 system, the job gets stuck during the Commit phase. This problem occurs only while installing the following package using the apt-get command. Other packages are installed without any issue.

pulseaudio_8.0-0ubuntu3.15_amd64.deb

The apt-get command runs in the interactive mode even after passing the -yes parameter to run the command in the non-interactive mode.

Workaround:

Add the following package to the Exclude list, and run the patching job.
pulseaudio_8.0-0ubuntu3.15_amd64.deb

You can use other options to force the apt-get command to exit from the interactive mode. However, these options are not recommended. For more details, see Man page of the apt-get command.

Job completes with a warning for the LSB module

When you a Patching Job on an Ubuntu 20.04 system, missing patches are deployed successfully. However, job completes with a warning similar to the following:

Warning 07/13/2021 22:52:04 [stderr: 2] No LSB modules are available. (Time in agent's deploy log:: 07/13/2021 22:51:52) 2 New Debian Package Group

Ignore the warning.


 

Tip: For faster searching, add an asterisk to the end of your partial query. Example: cert*