Troubleshooting Patch Management issues
This topic contains troubleshooting information for patch management issues. The topic includes the following sections:
For troubleshooting patch management on Solaris 11, see the separate page Troubleshooting Patch Management for Solaris 11.
Related BMC Communities article
BMC Customers using Automation for Patching use cases depend on OS vendors for Patches and metadata. To view a document that tracks the service status of the different OS Vendors as known to BMC Support, see the following BMC Communities document:
Troubleshooting Windows patching issues
The errors that you might encounter during a Windows patching task can be divided into the sections that follow.
Errors encountered while creating or updating a patch catalog
The following table lists issues that you might encounter while running a Patch Catalog Job, and provides troubleshooting information and references to knowledge articles for these issues.
Note
For general information about monitoring the progress and viewing the results of a Patch Catalog Job, see Viewing progress and results of a Patch Catalog Job.
Issue | Troubleshooting information |
---|---|
Unable to download shavlik metadata | The shavlik metadata may not be downloaded due to the following reasons:
Steps for debugging the shavlik downloaderTo enable DEBUG logging, add the following text to appserver.cf file: #Downloader debug log4j.logger.com.bladelogic.model.job.compliance.patch.ShavlikResult=DEBUG log4j.logger.com.bladelogic.app.util.DownloadServer=DEBUG |
Patch Catalog Update Job throws the following warning: No mappings were found for the selected product | Product Mappings are subject to change due to updates by Microsoft (and therefore Shavlik). If the updates occur before BMC Server Automation has shipped updates in the product, you can use a Windows Filter Configuration File to update the mappings used by BMC Server Automation. The video at right demonstrates the process of updating the product_categories.xml file. For the latest information on this issue, see the BMC Server Automation Knowledge Article ID: 000091409.
Note If the No mappings were found for the selected product message persists, the Windows Filter Configuration File needs further update. If you need assistance, contact BMC Customer Support. |
ACL issues: Windows Helper Server cannot be reached or access is not authorized | For information about creating an ACL template or ACL policy and controlling server access with agent ACLs, see Managing access. |
Patch object cannot be added or updated due to an RBAC issue | To create Patching Jobs and deploy patches, the patch administrator must be assigned a role that includes the necessary RBAC permissions. For more information, see Minimum permissions for patching. |
Windows Hotfix Patch is not found in the Catalog | For steps on troubleshooting this issue, see Windows Hotfix Patch not found in the Catalog. |
Catalog update job fails because of PD5, or HF7b configuration files. | For Windows patching, you may still be using the PD5.cab, or HF7b.cab ( PD5.xml, or HF7b.xml) configuration files for Windows patching. However, BMC Server Automation do not support the PD5, or HF7b configuration files, and you must use the WindowsPatchData.zip file instead. The Windows catalog update job fails if you use the .cab or .xml configuration files. To update the configuration files used for Windows patching see, Global configuration parameters. |
Errors encountered while running the Patching Job and the Remediation Job
The following table lists issues that you might encounter while running the Patching Job and the Remediation Job, and provides troubleshooting information and references to knowledge articles for these issues.
Note
The Trace.txt log file and the results.xml file are generated by the Windows Patch Analysis Job or by live browsing Hotfixes .The information in these logs can be used to troubleshoot errors. For more information about analyzing the Trace.txt, see How to analyze Trace.txt generated by a Windows Patch Analysis Job.
Issue | Troubleshooting information |
---|---|
Shavlik metadata was not copied to the target | File a ticket with BMC Support and provide the information from the following logs:
Turning on debug tracing for cl5.exeTo validate the payload, the shavlik engine runs the cl5.exe tool on the target. If this tool is not working properly, you can turn on tracing, using the following command: CL5.exe 2097197 1 <Log Directory> The customary value for maximum log size is 5000000. This command writes some values to the registry. To turn off CL5 tracing, use the following command: CL5.exe 2097197 0 |
One of the BLPatchCheck2 phases did not complete successfully | You can refer to the following article in the knowledge base for steps on troubleshooting this issue: If your issue is still not resolved, file a ticket with BMC Support and provide the information from the following logs:
|
Results are not passed back to the Application Server | File a ticket with BMC Support and provide the information from the following logs:
|
Analysis reported unexpected patch as missing | For information about troubleshooting this case, see Windows Patch Analysis Job reports an unexpected patch as missing. You can also refer to the following articles in the BMC knowledge base: |
Analysis did not report expected patch as missing | For information about troubleshooting this case, see Windows Patch Analysis Job does not report an expected missing patch. |
BMC Server Automation analysis does not match with other third-party patch solutions | For more information about troubleshooting this case, see Analyzing differences between Windows Patch Analysis Job results and third-party vendors. |
Troubleshooting Linux patching issues
When troubleshooting yum-based Linux patching job, it is important to know how the analysis process takes place. The high-level steps of the analysis process are listed below, along with the corresponding details for each step:
Step | High-level steps | Details |
---|---|---|
1 | The catalog metadata is copied from the repository location to the target server. | The following steps are executed:
|
2 | A custom yum.conf file is generated on the target that is used by the yum utility. Note that the system's default file is not used during analysis. | A custom yum.conf file is generated at ??STAGING_DIR??/LinuxCatalog_<hash>. The ??STAGING_DIR??/LinuxCatalog_<hash> path contains the following files:
|
3 | Analysis is performed. The analysis results are processed and sent to the application server. | The following steps are executed:
|
Notes
- During analysis, BMC Server Automation only uses BMC Server Automation catalogs. Any other repositories (for example /etc/yum.repos.d) to which the operating system is subscribed are ignored.
- On most Linux platforms, BMC Server Automation uses its own custom binary blyum, which is installed along with the RSCD agent. On RHEL 7, the native yum binary is used instead. Support for native yum is available for yum versions 3.2.22 and later.
The following table discusses issues that you might encounter when performing Linux patching.
Issue | Troubleshooting information |
---|---|
Yum failed to execute | Scenario 1: STDERR: cat: rpm-includes.lst: No such file or directory ERROR::YUM dry run failed. ERROR::cmd: failed! For more information, see Yum failed to execute - Scenario 1. |
Scenario 2: Could not find repodata.tar.gz corresponding to OsArch: ‘[ARCH]' in the catalog ( at location '//<repo>/catalog_XXX/[ARCH]/repodata.tar.gz'). For more information, see Yum failed to execute - Scenario 2. | |
Yum failed to complete | Scenario 1: For more information, see Yum failed to complete - Scenario 1. |
Scenario 2: For more information, see Yum failed to complete - Scenario 2. | |
Scenario 3: For more information, see Yum failed to complete - Scenario 3. | |
Deploy Job failed | Scenario 1: [rpm-version.arch]: Caching enabled but no local cache of //<staging>/blrepos/repo/packages/[rpm-version.arch].rpm from repo. For more information, see Deploy Job failed - Scenario 1. |
Scenario 2: For more information, see Deploy Job failed - Scenario 2. | |
Kernel RPMs removed during Patch Analysis Job | During Patch Analysis Job, warning messages notify you that old RPMs were removed. This happens when the native yum is used, and old packages are automatically removed, as controlled by the installonly_limit option in the yum.conf file. By default, only the last 3 packages are kept. To turn off the removal of old RPMs, set the installonly_limit option to a value of 0 on the yum.conf tab in the Patch Global Configuration dialog box or the Patching Job - Analysis Options panel. |
Yum failed to execute scenarios
Yum failed to execute - Scenario 1
The following scenario describes the No such file or directory
error that you might encounter when the yum file fails to execute, as well as steps to troubleshoot the issue.
Error Message | STDERR: cat: rpm-includes.lst: No such file or directory ERROR::YUM dry run failed. ERROR::cmd: failed! |
Description | The error messages suggest that either blyum is not found on the target or it is found but some libraries that are needed to run blyum were missing. |
Troubleshooting | To fix this error or research further, review the analysis_err.log error log file in the staging directory. This error is usually encountered when the supported agent version or architecture is not installed. So, to troubleshoot this issue, validate that the supported agent version or architecture is installed. |
Logs to collect | Log files to collect from the application server:
Log files to collect from the target:
|
Yum failed to execute - Scenario 2
The following scenario describes an error in the execution of the yum file, as well as steps to troubleshoot the issue.
Error Message |
|
Description | This error message indicates that the Catalog Job did not succeed at a target where the repodata.tar.gz file was not generated. As a result, the Analysis Job was run against an invalid target whose OS level or architecture did not match. This problem might be caused by an issue on the Repo Server. |
Troubleshooting | To fix this error, perform the following:
|
Logs to collect |
From the target
From the Repo server
|
Yum failed to complete scenarios
Yum failed to complete - Scenario 1
The following scenario describes an error where the yum file fails to complete, as well as steps to troubleshoot the issue.
Error Message |
|
Description | Both rpm1 of the specified version and rpm2 are installed on the target. A newer version of rpm1 is found in the Catalog and set to be updated. The installed version of rpm1 is also a dependency to rpm2. To approve the installation of a newer rpm1, yum also needs to update rpm2. However, yum cannot find a newer version of rpm2 in repodata.tar.gz, so rpm2 is excluded from analysis. As a new version of rpm2 is not found, the rpm is not offered an update. Because rpm2 is not updated, yum cannot allow the update of rpm1. An error is logged, alerting that rpm1 needs to remain installed to preserve the dependency of rpm2. |
Troubleshooting | To troubleshoot this error, follow these steps:
|
Logs to collect | Log files to collect from the application server Patch Analysis Job log Log files to collect from the target analysis bundle.log (including repodata.tar.gz) |
Yum failed to complete - Scenario 2
The following scenario describes an error where the yum file fails to complete, as well as steps to troubleshoot the issue.
Error Message |
|
Description | Neither rpm1 of the specified version nor rpm2 are installed on the target. Yum sets rpm2 to be updated for one of two reasons:
Yum checks if rpm2 needs any dependencies of its own, and detects that rpm1 of the specified version is needed as a dependency. Therefore, yum attempts to set rpm1 to be updated as well, but fails. An error is logged, alerting that rpm1 is required for rpm2 to be installed. |
Troubleshooting | To troubleshoot this error, follow these steps:
|
Logs to collect | Log files to collect from the application server Patch Analysis Job log Log files to collect from the target
|
Yum failed to complete - Scenario 3
The following scenario describes an error where the yum file fails to complete, as well as steps to troubleshoot the issue.
Error Message | [rpm1] conflicts with [rpm2] |
Description | Rpm2 is installed on the target, while rpm1 is not. Yum attempts to offer rpm1 to be installed or updated, but then yum detects that some of the files to be updated by rpm1 are used by rpm2. Therefore, yum rejects the installation or update of rpm1. An error is logged, alerting that rpm1 cannot be installed. In general, the conflict suggests that rpm1 and rpm2 cannot coexist if rpm versions matter. |
Troubleshooting |
|
Logs to collect | Log files to collect from the application server Patch Analysis Job log Log files to collect from the target
|
Deploy Job failed scenarios
Deploy Job failed - Scenario 1
Error Message | [rpm-version.arch]: Caching enabled but no local cache of //<staging>/blrepos/repo/packages/[rpm-version.arch].rpm from repo |
Description | Prior to installing the packages, the Deploy Job runs a preliminary yum analysis. During this analysis scan, rpm-version.arch is found as missing. The Deploy Job does not find rpm-version.arch in the list of staged missing patches, and, therefore, the job cannot proceed. An error is logged, alerting you that the patch payload is not found and cannot be installed. This error indicates that a discrepancy was found between the regular Patch Analysis and the preliminary yum scan during Deploy, or that rpm-version.arch was not copied during staging. The error message is written in either the Deploy Job log or the bldeploy log. |
Troubleshooting |
|
Deploy Job failed - Scenario 2
Error Message | package [rpm-version] is already installed |
Description | The Deploy Job installs the patches that were found missing during Analysis. In this case, the Analysis Job packaged the rpm-version, even though it is already installed. |
Troubleshooting |
|
Logs to collect | Log files to collect from the application server
Log files to collect from the target
|
Third party issues and limitations for patch management
Patch management issue for version 8.7 agents
The following OS versions are supported as patching targets, only if the target system has an 8.7 P4 agent version installed on it.
- RedHat Enterprise Linux version 5 (and earlier)
- Oracle Enterprise Linux version 5 (and earlier)
- SuSE version 10 (and earlier)
For targets on any of these platforms, do one of the following:
- Update the version 8.8 agents with the component template fix available from Knowledge Article 000120867.
- Use an earlier version of the agent (for example, version 8.6).
Issues while patching on Red Hat Enterprise Linux
The following issue exists while patching on Red Hat Enterprise Linux:
- You maybe unable to install or update the open-vm-tools-deploypkg (from the VMware repository) on a RHEL7 target due to a known issue with this patch. See https://access.redhat.com/solutions/1449563, for more information about this known issue.
Workaround: You can try excluding the all open-vm-tool patches during the patch analysis job or removing open-vm-tools-deploypkg from the target server. - Run the
ldconfig
command on your RHEL 7 target, if you encounter the following type of error during patch analysis:
Error 03/07/2017 12:22:25 STDERR: There was a problem importing one of the Python modulesrequired to run yum. The error leading to this problem was: libcurl.so.4: cannot open shared object file: No such file or directoryPlease install a package which provides this module, orverify that the module is installed correctly.It's possible that the above module doesn't match thecurrent version of Python, which is:2.7.5 (default, Feb 11 2014, 07:46:25) [GCC 4.8.2 20140120 (Red Hat 4.8.2-13)]If you cannot solve this problem yourself, please go to the yum faq at: http://yum.baseurl.org/wiki/Faq ERROR::cmd: yum -c "./yum.conf" - We need to run the ldconfig command on the machine (ldconfig is a command that is used to maintain the shared library cache. This cache is typically stored in the file /etc/ld.so.cache and is used by the system to map a shared library name to the location of the corresponding shared library file)
Issues while patching on SUSE Enterprise Linux
The following issues and limitations exist while patching on SUSE Enterprise Linux:
- While updating a SUSE catalog with the SUSE 9-x86_64 filter, the count of the Downloaded patches displayed by BMC Server Automation may be incorrect. The count of RPMs that do not contain release version and osArch information do not appear in Downloaded count field, even though they are added to the catalog.
- While creating a SuSE Linux Catalog, certain rpm files might cause the Catalog Update Job to fail, with the following message: "BLPAT1700 - RPM verification failed on selected repository location. Please use SuSE platform repository location and run the catalog.". This issue is due to SuSE Enterprise Linux limitations. For this Catalog Update Job, ensure that you specify a repository location on a SuSE Enterprise Linux host computer.
Note
If this error occurs even when you are using a SuSE repository location, the issue might be caused due to zero byte rpms or corrupted rpms in the repository location. To resolve this issue, delete all zero byte size rpms and corrupted rpms from the repository location and update the catalog. Use the following commands on the SuSE repository server:
To list all the zero byte rpms
ls -s|grep -E "^ *0"
To delete all zero byte rpms
ls -s|grep -E "^ *0"|sed "s/^ *0 //g"|xargs -i rm "{}"
- While you are creating a SuSE Linux Catalog in the Online mode, the Catalog fails with an rpm-python dependency. The error message states rpm-python needed by createrepo-0.4.6-1.el4.rf.noarch.
Workaround: You must check the version of create-repo and python-urlgrabber in the repository server and either ensure that the lower versions are installed or remove the higher versions, if they exist. In addition, you must ensure that rpm-python exists on the system and the version of rpm-python is compatible with the create-repo version that is shipped with the product. - For SUSE Linux Enterprise version 10 SP1, hplip17-hpijs-1.7.2-0.15.x86_64.rpm and hplip-hpijs-0.9.7-19.9.x86_64.rpm packages obsolete each other during installation. If hplip17-hpijs-1.7.2-0.15.x86_64.rpm is installed, the hplip-hpijs-0.9.7-19.9.x86_64.rpm package is removed from the SUSE target and the other way around. Therefore, when one of these packages is installed, the other package is shown as missing on the next Patch Analysis Job run.
Workaround: You must decide which of these files must exist on the SUSE target and mark the other file as excluded. - For SUSE Linux Enterprise version 10, the SP3 Pool catalog contains a lower version of the SPident-0.9-74.27.8.noarch.rpm file. If you perform an analysis on a SUSE Linux Enterprise version 10 SP3 target (for example, by using the Online mode catalog), the SPident-0.9-74.27.8.noarch.rpm is reported as missing. This issue occurs due to a conflict in the repositories provided by Novell and can be ignored. This issue will be resolved when Novell updates its repository with a newer version of the rpm.
For SUSE Linux Enterprise version 10 SP4, the Patch Analysis Job fails with an error message for the kernel-default rpm: STDERR: cat: rpm-includes.lst: No such file or directoryERROR::YUM dry run failedERROR::sysconfig conflicts with kernel-defaultERROR::cmd: failed! This issue occurs due to a conflict in the repositories provided by Novell and will be resolved when Novell updates its repository with a newer version of the kernel-default rpm.
Workaround: If you encounter this package conflict, you must exclude the conflicting kernel-default rpm by using the include/exclude functionality in the Patching Job.
Issues while patching on AIX
The following issues and limitations exist while patching on AIX:
- For AIX, the Patch Analysis Job fails in the Install Mode because of missing filesets in the repository. A single repository will contain updates for all available packages. The Install Mode is intended to install new packages on the target. Hence, BMC recommends the use of Install Mode on selective packages, rather than using the Install Mode on the complete repository.
To cancel an AIX patch catalog job that is running with the SUMA download option, you must perform the following steps:
Cancel the Catalog Update Job (CUJ) from the Console to stop the job running on the Application Server.
Log on to the repository server and manually kill the SUMA download process.
AIX remediation job fails to deploy 'rsct.lapi.rte 3.1.6.0' on some targets due to a problem in the third-party patch. For more information about this issue, see the following third-party resource:
https://www-304.ibm.com/support/docview.wss?uid=isg1IZ80922.
- AIX remediation job fails to deploy 'idsldap.srv64bit62.rte 6.2.0.16' on some targets due to a problem in the third-party patch.
AIX remediation job fails to deploy 'rsct.lapi.bsr.bsrpd.odmdel' (rsct.lapi.bsr 3.1.5.0) on some targets due to a problem in the third-party patch. For more information about this issue, see the following third-party resource:
https://www-304.ibm.com/support/docview.wss?uid=isg1IZ80922.
- After 'bos.rte.archive 5.3.0.51' is deployed on an AIX 5.3 target, the target goes into reboot mode and fails to restart.
After deploying AIX 6.1 TL9SP1 on a target, the 'bos.rte.install 6.1.9.1' fileset enters a broken state if you try to Undo the deploy job. This 'bos.rte.install 6.1.9.1' fileset enters a broken state even if you reject the fileset manually.
Workaround: To fix the 'bos.rte.install 6.1.9.1' fileset in the broken state you can take the following corrective actions:
- If the broken fileset is an update, reinstall the update using options in the installation program that will apply, commit, and automatically install the required software without saving the replaced files (-acgN flags).
- Reinstall the entire fileset at the base level with the desired updates by using the FORCE option (-F flag).
If you are still unable to fix the fileset in the broken state, contact IBM support team for further assistance.
- Error while extracting the offline downloader tar package, because of the length of file path input string.
Workaround: BMC recommends using the GNU tar (gtar) utility to extract the offline downloader package. You might encounter dependency errors while performing patch analysis on AIX 6.1 TL09 for security fixes. Before you run the analysis job, add AIX 6.1 TL08 to the catalog filter and run the catalog update job to add AIX 6.1 TL08 content to the catalog.
- You cannot edit a patch catalog of a deprecated AIX platform version, even if the patch catalog was created in an earlier version of BMC Server Automation. For more information about which AIX platform versions are supported for patching, see the System requirements section.
Issues while patching on Oracle Solaris
The following issues and limitations exist while patching on Oracle Solaris:
- Error while extracting the offline downloader tar package, because of length of file path input string.
Workaround: BMC recommends using the GNU tar (gtar) utility to extract the offline downloader package.
Issues while patching on Oracle Enterprise Linux
The following issues and limitations exist while patching on Oracle Enterprise Linux:
- For Oracle Enterprise Linux, the Patch Analysis Job fails in the Install Mode because of missing rpms in the repository. A single repository will contain updates for all available packages. The Install Mode is intended to install new packages on the target. Hence, BMC recommends the use of Install Mode on selective packages, rather than using the Install Mode on the complete repository.
Issues while patching on Windows
The following issues and limitations exist while patching on Windows:
- You may encounter errors if you try applying more than one patch or hotfix, on a Windows target, at the same time. To resolve this issue, BMC recommends applying only a single patch or a single hotfix at a time. Note that patches and hotfixes must be applied on Windows targets only if it is required to solve a specific issue. For best practices while applying patches and hotfixes, see MSDN - Best Practices for Applying Service Packs, Hotfixes and Security Patches.
- Before you apply multiple hotfixes, you must check whether the hotfix changes are available in the latest released service pack. This is because, Shavlik recommends that it is better to apply the latest service pack instead of applying several hofixes on the server. For best practices while applying patches and hotfixes, see MSDN - Best Practices for Applying Service Packs, Hotfixes and Security Patches.
If patch KB2810009 is found missing during patch analysis, BMC recommends that you first separately deploy patch KB2810009 and then re-run the remediation job for the other missing patches. This KB2810009 missing patch needs to be deployed separately because of dependency issues.
- BMC Server Automation automatically creates the required pre-installation and post-installation environment on a Windows target server for patching Java installation files. However, BMC Server Automation does not decide whether to reboot the Windows target server.. The target must be rebooted after each version of Java installation patch is deployed on the target. You can specify this reboot option while creating a Deploy Job, see Deploy Job - Job Options for more information.
Comments