Troubleshooting Patch Management issues
This topic contains troubleshooting information for patch management issues. The topic includes the following sections:
For troubleshooting patch management on Solaris 11, see the separate page Troubleshooting-Patch-Management-for-Solaris-11.
Troubleshooting Windows patching issues
The errors that you might encounter during a Windows patching task can be divided into the sections that follow.
Errors encountered while creating or updating a patch catalog
The following table lists issues that you might encounter while running a Patch Catalog Job, and provides troubleshooting information and references to knowledge articles for these issues.
Issue | Troubleshooting information |
---|---|
Unable to download shavlik metadata | The shavlik metadata may not be downloaded due to the following reasons:
Steps for debugging the shavlik downloaderTo enable DEBUG logging, add the following text to appserver.cf file: #Downloader debug |
Patch Catalog Update Job throws the following warning: No mappings were found for the selected product | Product Mappings are subject to change due to updates by Microsoft (and therefore Shavlik). If the updates occur before TrueSight Server Automation has shipped updates in the product, you can use a Windows Filter Configuration File to update the mappings used by TrueSight Server Automation. The video at right demonstrates the process of updating the product_categories.xml file. For the latest information on this issue, see the TrueSight Server Automation Knowledge Article ID: 000091409.
|
ACL issues: Windows Helper Server cannot be reached or access is not authorized | For information about creating an ACL template or ACL policy and controlling server access with agent ACLs, see Managing-access. |
Patch object cannot be added or updated due to an RBAC issue | To create Patching Jobs and deploy patches, the patch administrator must be assigned a role that includes the necessary RBAC permissions. For more information, see Minimum-permissions-for-patching. |
Windows Hotfix Patch is not found in the Catalog | For steps on troubleshooting this issue, see Windows-Hotfix-Patch-not-found-in-the-Catalog. |
Catalog update job fails because of PD5, or HF7b configuration files. | For Windows patching, you may still be using the PD5.cab, or HF7b.cab ( PD5.xml, or HF7b.xml) configuration files for Windows patching. However, TrueSight Server Automation 8.9.03 and later versions do not support the PD5, or HF7b configuration files, and you must use the WindowsPatchData.zip file instead. The Windows catalog update job fails if you use the .cab or .xml configuration files. To update the configuration files used for Windows patching see, Global configuration parameters. |
Errors encountered while running the Patching Job and the Remediation Job
The following table lists issues that you might encounter while running the Patching Job and the Remediation Job, and provides troubleshooting information and references to knowledge articles for these issues.
Issue | Troubleshooting information |
---|---|
Shavlik metadata was not copied to the target | File a ticket with BMC Support and provide the information from the following logs:
Turning on debug tracing for cl5.exeTo validate the payload, the shavlik engine runs the cl5.exe tool on the target. If this tool is not working properly, you can turn on tracing, using the following command: CL5.exe 2097197 1 <Log Directory> The customary value for maximum log size is 5000000. This command writes some values to the registry. To turn off CL5 tracing, use the following command: CL5.exe 2097197 0 |
One of the BLPatchCheck2 phases did not complete successfully | You can refer to the following article in the knowledge base for steps on troubleshooting this issue: If your issue is still not resolved, file a ticket with BMC Support and provide the information from the following logs:
|
Results are not passed back to the Application Server | File a ticket with BMC Support and provide the information from the following logs:
|
Analysis reported unexpected patch as missing | For information about troubleshooting this case, see Windows-Patch-Analysis-Job-reports-an-unexpected-patch-as-missing. |
Analysis did not report expected patch as missing | For information about troubleshooting this case, see Windows-Patch-Analysis-Job-does-not-report-an-expected-missing-patch. |
TrueSight Server Automation analysis does not match with other third-party patch solutions | For more information about troubleshooting this case, see Analyzing-differences-between-Windows-Patch-Analysis-Job-results-and-third-party-vendors. |
Troubleshooting Linux patching issues
When troubleshooting yum-based Linux patching job, it is important to know how the analysis process takes place. The high-level steps of the analysis process are listed below, along with the corresponding details for each step:
Step | High-level steps | Details |
---|---|---|
1 | The catalog metadata is copied from the repository location to the target server. | The following steps are executed:
|
2 | A custom yum.conf file is generated on the target that is used by the yum utility. Note that the system's default file is not used during analysis. | A custom yum.conf file is generated at ??STAGING_DIR??/LinuxCatalog_<hash>. The ??STAGING_DIR??/LinuxCatalog_<hash> path contains the following files:
|
3 | Analysis is performed. The analysis results are processed and sent to the application server. | The following steps are executed:
|
The following table discusses issues that you might encounter when performing Linux patching.
Issue | Troubleshooting information |
---|---|
Yum failed to execute | Scenario 1: STDERR: cat: rpm-includes.lst: No such file or directory ERROR::YUM dry run failed. ERROR::cmd: failed! For more information, see Yum failed to execute - Scenario 1. |
Scenario 2: Could not find repodata.tar.gz corresponding to OsArch: ‘[ARCH]' in the catalog ( at location '//<repo>/catalog_XXX/[ARCH]/repodata.tar.gz'). For more information, see Yum failed to execute - Scenario 2. | |
Yum failed to complete | Scenario 1: Missing Dependency: [rpm1] = [version] is needed by package [rpm2-version.arch] (installed) For more information, see Yum failed to complete - Scenario 1. |
Scenario 2: Missing Dependency: [rpm1] = [version] is needed by package [rpm2-version.arch] (repo) For more information, see Yum failed to complete - Scenario 2. | |
Scenario 3: [rpm1] conflicts with [rpm2] For more information, see Yum failed to complete - Scenario 3. | |
Deploy Job failed | Scenario 1: [rpm-version.arch]: Caching enabled but no local cache of //<staging>/blrepos/repo/packages/[rpm-version.arch].rpm from repo. For more information, see Deploy Job failed - Scenario 1. |
Scenario 2: Transaction Check Error: package [rpm-version] is already installed. For more information, see Deploy Job failed - Scenario 2. | |
Kernel RPMs removed during Patch Analysis Job | During Patch Analysis Job, warning messages notify you that old RPMs were removed. This happens when the native yum is used, and old packages are automatically removed, as controlled by the installonly_limit option in the yum.conf file. By default, only the last 3 packages are kept. To turn off the removal of old RPMs, set the installonly_limit option to a value of 0 on the yum.conf tab in the Patch Global Configuration dialog box or the Patching Job - Analysis Options panel. |
Yum failed to execute scenarios
Yum failed to execute - Scenario 1
The following scenario describes the No such file or directory error that you might encounter when the yum file fails to execute, as well as steps to troubleshoot the issue.
Error Message | STDERR: cat: rpm-includes.lst: No such file or directory ERROR::YUM dry run failed. ERROR::cmd: failed! |
Description | The error messages suggest that either blyum is not found on the target or it is found but some libraries that are needed to run blyum were missing. |
Troubleshooting | To fix this error or research further, review the analysis_err.log error log file in the staging directory. This error is usually encountered when the supported agent version or architecture is not installed. So, to troubleshoot this issue, validate that the supported agent version or architecture is installed. |
Logs to collect | Log files to collect from the application server:
Log files to collect from the target:
|
Yum failed to execute - Scenario 2
The following scenario describes an error in the execution of the yum file, as well as steps to troubleshoot the issue.
Error Message | Could not find repodata.tar.gz corresponding to OsArch: ‘[ARCH]' in the catalog ( at location '//<repo>/catalog_XXX/[ARCH]/repodata.tar.gz') |
Description | This error message indicates that the Catalog Job did not succeed at a target where the repodata.tar.gz file was not generated. As a result, the Analysis Job was run against an invalid target whose OS level or architecture did not match. This problem might be caused by an issue on the Repo Server. |
Troubleshooting | To fix this error, perform the following:
|
Logs to collect |
From the target
From the Repo server
|
Yum failed to complete scenarios
Yum failed to complete - Scenario 1
The following scenario describes an error where the yum file fails to complete, as well as steps to troubleshoot the issue.
Error Message | {{code language="none"}} Missing Dependency: [rpm1] = [version] is needed by package [rpm2-version.arch] (installed) {{/code}} |
Description | Both rpm1 of the specified version and rpm2 are installed on the target. A newer version of rpm1 is found in the Catalog and set to be updated. The installed version of rpm1 is also a dependency to rpm2. To approve the installation of a newer rpm1, yum also needs to update rpm2. However, yum cannot find a newer version of rpm2 in repodata.tar.gz, so rpm2 is excluded from analysis. As a new version of rpm2 is not found, the rpm is not offered an update. Because rpm2 is not updated, yum cannot allow the update of rpm1. An error is logged, alerting that rpm1 needs to remain installed to preserve the dependency of rpm2. |
Troubleshooting | To troubleshoot this error, follow these steps:
|
Logs to collect | Log files to collect from the application server Patch Analysis Job log Log files to collect from the target |
Yum failed to complete - Scenario 2
The following scenario describes an error where the yum file fails to complete, as well as steps to troubleshoot the issue.
Error Message | {{code language="none" source="string:{{code language=~"none~"~}~} Missing Dependency: [rpm1] = [version] is needed by package [rpm2-version.arch] (repo) {{/code~}~}"/}} {{/code}} ))) |((( **Description** )))|((( Neither rpm1 of the specified version nor rpm2 are installed on the target. Yum sets rpm2 to be updated for one of two reasons: * Yum detected an older version of rpm2 installed. * Yum offered rpm2 as a dependency to some other rpm. Yum checks if rpm2 needs any dependencies of its own, and detects that rpm1 of the specified version is needed as a dependency. Therefore, yum attempts to set rpm1 to be updated as well, but fails. An error is logged, alerting that rpm1 is required for rpm2 to be installed. ))) |((( **Troubleshooting** )))|((( To troubleshoot this error, follow these steps: 1. Check if rpm1 of the specified version exists in the Catalog. 1. If rpm1 is not in the Catalog, check if it exists in the RHN Channel. 1*. If not in RHN Channel, then analysis happens as expected. 1*1. Acquire rpm1 by other means and install it manually. 1*1. Exclude rpm2 from the analysis or uninstall rpm2. When you uninstall rpm2, consult your administrator and use caution. Note that another rpm might require the newer version of the excluded rpm. Furthermore, this step will not work if yum offered rpm2 as a dependency to some other rpm. 1*. If present in the RHN Channel, run the Catalog Update Job again. 1*1. If rpm1 is still not in catalog, and this is not an RBAC-related issue, then collect logs and contact BMC Support. 1*1. If rpm1 is now present in the catalog, run the analysis again. If Analysis still shows the same error, continue to the next step. 1. If rpm1 is present in the catalog, check whether rpm1 is present in **repodata.tar.gz** on the target by running the following command:{{code language="none"}}# cd <staging>/LinuxCatalog_XXX_[target]/{{/code}}{{code language="none"}}# yum -C -c yum.conf search [rpm1] {{/code}}Proceed according to the results that you obtain: 1*. If rpm1 is not found, then this could be why it is not offered during Analysis. 1*. If rpm1 is found, something might be preventing it from being installed. In such a case, try performing a //dry run// installation of rpm1 using the following command: {{code language="none"}}# rpm -iUv --test rpm1{{/code}} If the dry run fails, then yum predicted this failure and that is why yum rejected rpm1 from being updated, and rpm1 was no longer considered to be updated. An error is logged, indicating rpm1 as a missing dependency. ))) |((( **Logs to collect** )))|((( (% class="O1" %) ((( **Log files to collect from the application server** ))) (% class="O1" %) ((( Patch Analysis Job log \\ ))) (% class="O1" %) ((( (% class="O1" %)**Log files to collect from the target** ))) * rscd.log * analysis bundle.log (including repodata.tar.gz) * rpm –qa ))) {{export-ignore}} ,,[[Back to top>>doc:||anchor="TroubleshootingPatchManagementissues-Top"]],, {{/export-ignore}} ==== {{id name="TroubleshootingPatchManagementissues-Yumfailedtocomplete-Scenario3"/}}{{id name="TroubleshootingPatchManagementissues-yum_fail_complete_3"/}}Yum failed to complete - Scenario 3 ==== The following scenario describes an error where the yum file fails to complete, as well as steps to troubleshoot the issue. |((( **Error Message** )))|((( (% style="margin-left: 0.3in;" %) ((( {{code language="none"}}[rpm1] conflicts with [rpm2]{{/code}} (% style="color: black;" %) ))) ))) |((( **Description** )))|((( Rpm2 is installed on the target, while rpm1 is not. Yum attempts to offer rpm1 to be installed or updated, but then yum detects that some of the files to be updated by rpm1 are used by rpm2. Therefore, yum rejects the installation or update of rpm1. An error is logged, alerting that rpm1 cannot be installed. In general, the conflict suggests that rpm1 and rpm2 cannot coexist if rpm versions matter. ))) |((( **Troubleshooting** )))|((( 1. Check the RHN site for known conflict issues. A conflict may arise if either rpm1 or rpm2 are not in the repository. 1. Validate that the rpm is in RHN channel, in the Catalog, and in **repodata.tar.gz**, as described in [[scenario 2>>doc:||anchor="TroubleshootingPatchManagementissues-deploy_job_fail_scenario_2"]]. Note that resolution cannot be guaranteed if the RPMs come from different channels. 1. Proceed according to your needs: 1*. If rpm1 must be installed, check whether rpm2 can be uninstalled, downgraded, or upgraded to a non-conflicting version. 1*. If rpm2 must be preserved, consider the following actions: 1**. Check whether rpm1 can be excluded. Note that another rpm might require the newer version of the excluded rpm. 1**. Check whether another non-conflicting version of rpm1 can be installed. 1**. Check whether multiple versions of rpm2 are installed, where the conflicting one can be uninstalled. 1. If the previous steps do not resolve the problem, collect logs and contact BMC Support. ))) |((( **Logs to collect** )))|((( (% class="O1" %) ((( **Log files to collect from the application server** ))) (% class="O1" %) ((( Patch Analysis Job log \\ ))) (% class="O1" %) ((( (% class="O1" %)**Log files to collect from the target** ))) * analysis bundle.log (including repodata.tar.gz) * rpm –qa ))) {{export-ignore}} ,,[[Back to top>>doc:||anchor="TroubleshootingPatchManagementissues-Top"]],, {{/export-ignore}} === {{id name="TroubleshootingPatchManagementissues-DeployJobfailedscenarios"/}}Deploy Job failed scenarios === ==== {{id name="TroubleshootingPatchManagementissues-DeployJobfailed-Scenario1"/}}{{id name="TroubleshootingPatchManagementissues-deploy_job_fail_scenario_1"/}}Deploy Job failed - Scenario 1 ==== |((( **Error Message** )))|((( {{code language="none"}} [rpm-version.arch]: Caching enabled but no local cache of //<staging>/blrepos/repo/packages/[rpm-version.arch].rpm from repo {{/code}} ))) |((( **Description** )))|((( Prior to installing the packages, the Deploy Job runs a preliminary yum analysis. During this analysis scan, **rpm-version.arch** is found as missing. The Deploy Job does not find **rpm-version.arch** in the list of staged missing patches, and, therefore, the job cannot proceed. An error is logged, alerting you that the patch payload is not found and cannot be installed. This error indicates that a discrepancy was found between the regular Patch Analysis and the preliminary yum scan during Deploy, or that **rpm-version.arch** was not copied during staging. The error message is written in either the Deploy Job log or the **bldeploy** log. ))) |((( **Troubleshooting** )))|((( (% class="content-wrapper" %) ((( 1. Review Patch Analysis results and check whether **rpm-version.arch** was found as missing. 1. If the file is missing, check whether it was packaged during patch remediation.\\ 1*. If it was packaged, check whether it points to an existing source. Open the object from inside the Catalog and review its **Installable source** value.\\ 1**. The root cause of the problem might be that the installable source //points to a missing payload//. In this case, rerun the Catalog or download the patch manually. 1**. If the source points to an //available payload//, review the stage phase of the Deploy Job. Check for issues that might have occurred during the copying of the payload to the target. 1*. If the object was not packaged, check for possible RBAC issues. 1. ((( If patch was not found missing, compare yum scans of Analysis and Deploy: * Analysis: **<staging>/LinuxCatalog_XXX_[target]/yum_analysis.res** * Deploy: **<staging>/XXXXX/blres.yum.depl** {{confluence_note}} The deploy log exists if the Deploy Job was set to //preserve staging area on failure//. If not, configure this setting now, then reproduce the error and collect the log file. {{/confluence_note}} ))) 1. Compare the lists of //set to be updated// patches. This comparison might reveal that for some RPMs, the analysis finds one **arch** file missing, while Deploy finds the other one missing. This might be the cause of the original issue. If so, a fix is available in TrueSight Server Automation 8.1 Service Pack 3. 1. If the preceding steps did not reveal the root cause of the issue, collect several logs and contact BMC Support. From the application server, obtain the Patch Analysis Job log. In addition, obtain the following logs from the target: 1*. Analysis bundle log 1*. bldeploy log 1*. **<staging>/XXXXX/blres.yum.depl** ))) ))) {{export-ignore}} ,,[[Back to top>>doc:||anchor="TroubleshootingPatchManagementissues-Top"]],, {{/export-ignore}} ==== {{id name="TroubleshootingPatchManagementissues-DeployJobfailed-Scenario2"/}}{{id name="TroubleshootingPatchManagementissues-deploy_job_fail_scenario_2"/}}Deploy Job failed - Scenario 2 ==== |((( **Error Message** )))|((( (% style="margin-left: 0.3in;" %) ((( (% class="code" %) ((( package [rpm-version] is already installed ))) (% style="color: black;" %) ))) ))) |((( **Description** )))|((( (% class="O1" %) The Deploy Job installs the patches that were found missing during Analysis. In this case, the Analysis Job packaged the rpm-version, even though it is already installed. ))) |((( **Troubleshooting** )))|((( 1. ((( Review the Analysis logs to troubleshoot the issue. (% class="O1" %) ((( The error message is located in either the Deploy Job log or the bldeploy log. You might see the following log records: ))) * The analysis_log.log is expected to reveal that the Patch Analysis ran with an include filter. The signs for this are:\\ ** After 'update' you see a list of RPMs. ** YUM Command: for i in 0 1 2 3 4 5 :; {{code language="none"}}do echo n; done | blyum -c ./yum.conf -C update "rpm-version.arch" "…" ...{{/code}} * Command {{code language="none"}}rpm –q [rpm]{{/code}} reveals that multiple versions of the rpm are installed. ))) 1. The expected findings described in the previous step are not the standard configuration. Therefore, perform the following further testing: 1*. Run yum update with include list. This is expected to produce the following error:{{code language="none"}}latest rpm as missing{{/code}} [issue reproduced] The updated rpm is offered for the older version of RPM that is installed even if the new version is also already installed. 1*. Run yum update without include list. This should return the following message:{{code language="none"}}no missing rpm{{/code}} [good result] If we do not forcefully analyze for patches (with an include filter), then if the latest version is found, the rpm is not offered. 1*. Run yum install with include list. This should return the following message:{{code language="none"}}no missing rpm{{/code}} [good result] The //Install// option checks only for the specific version of the rpm that we are attempting to install. Since our specific version of rpm is already installed, it will not be installed again. 1. If you must preserve multiple versions of such RPMs, create a separate analysis job where you would use //Install //option and include the RPM. 1. Modify the existing Analysis Job include filter and remove the RPM. If the RPM must be included, try excluding the opposite set of RPMs. For example, exclude all where RPM does not equal [rpm-version]. 1. Yum does not install multiple versions of such RPMs by default, so if they are not required, then uninstall the old versions. The Analysis Job should no longer report the latest RPM as missing, because there will no longer be multiple versions. ))) |((( **Logs to collect** )))|((( (% class="O1" %) ((( **Log files to collect from the application server** ))) (% class="O1" %) ((( * Patch Analysis Job log ))) (% class="O1" %) ((( (% class="O1" %)**Log files to collect from the target** ))) (% class="O1" %) ((( * analysis bundle log * bldeploy log * <staging>/XXXXX/blres.yum.depl * rpm –q [rpm] ))) ))) {{export-ignore}} ,,[[Back to top>>doc:||anchor="TroubleshootingPatchManagementissues-Top"]],, {{/export-ignore}} \\ == {{id name="TroubleshootingPatchManagementissues-Archivingmechanismforpatchanalysisfiles"/}}Archiving mechanism for patch analysis files == As an alternative to running a Patch Analysis Job in [[debug mode>>doc:Automation-DevSecOps.Server-Automation.TrueSight-Server-Automation.tssa244.Troubleshooting.Collecting-diagnostics.Obtaining-job-diagnostics.WebHome||anchor="Obtainingjobdiagnostics-Runningajobindebugmode"]], an archiving mechanism introduced in TrueSight Server Automation 8.9.01 saves patch analysis files on Linux or Windows target servers. This archiving mechanism stores patch analysis files from the 3 most recent job runs, and deletes files from any older job runs during every job run. The patch analysis files are stored in each target agent's **Transactions** directory, in a subdirectory named **analysis_archive** (for example, **C:\Program Files\BMC Software\BladeLogic\RSCD\Transactions\analysis_archive\**). Files are stored separately for each job run, in a subdirectory with the following naming convention: ~_~_//JobId//,//JobVersionId//-//JobRunId//~_~_ (for example, **~_~_PatchJob5,1-71~_~_**).\\ You can use blasadmin parameters to turn off this archiving mechanism and turn it back on, as well as to control the number of job runs for which to save patch analysis files. For more information, see [[doc:Automation-DevSecOps.Server-Automation.TrueSight-Server-Automation.tssa244.Administering.Managing-Application-Server-behavior-using-the-Application-Server-Administration-console.Controlling-patch-analysis-file-archiving.WebHome]].\\ {{export-ignore}} ,,[[Back to top>>doc:||anchor="TroubleshootingPatchManagementissues-Top"]],, {{/export-ignore}} \\ == {{id name="TroubleshootingPatchManagementissues-Thirdpartyissuesandlimitationsforpatchmanagement"/}}Third party issues and limitations for patch management == {{excerpt-include 0="Automation-DevSecOps.Server-Automation.TrueSight-Server-Automation.tssa244.Troubleshooting.Troubleshooting-by-component-or-functional-area.Troubleshooting-Patch-Management-issues.Special-issues-for-patch-management.WebHome" name="third_party_patch_issues" nopanel="true"/}} {{export-ignore}} ,,[[Back to top>>doc:||anchor="TroubleshootingPatchManagementissues-Top"]],, {{/export-ignore}} == {{id name="TroubleshootingPatchManagementissues-Relatedtopics"/}}Related topics == [[doc:Automation-DevSecOps.Server-Automation.TrueSight-Server-Automation.tssa244.Troubleshooting.Troubleshooting-by-component-or-functional-area.Troubleshooting-Patch-Management-issues.Special-issues-for-patch-management.WebHome]]\\ |