Troubleshooting Patch Management issues

Unable to download shavlik metadata

The shavlik metadata may not be downloaded due to the following reasons:

• Files are not available at the shavlik site
• Firewall has blocked access to shavlik URLs

Steps for debugging the shavlik downloader

To enable DEBUG logging, add the following text to appserver.cf file:

#Downloader debug
log4j.logger.com.bladelogic.model.job.compliance.patch.ShavlikResult=DEBUG
log4j.logger.com.bladelogic.app.util.DownloadServer=DEBUG

1. Navigate to the following BMC Communities document: Windows Patching- product_categories.xml
2. Download the most recent Windows Filter Configuration File file to a location available to the BMC Server Automation console.
3. From the console, select Configuration > Patch Global Configuration and click the Windows tab.
4. Next to the Windows Filter Configuration File (the last item in the tab), click the ellipsis button (...) and navigate to the downloaded file.
5. Select the product categories mapping file you downloaded from BMC Communities and click OK.
6. Click the Save icon (the blue diskette) at the top left corner of Patch Global Configuration window.
7. Edit the filter list in the patch catalog and select the correct Product Filter.
8. Run the Patch Catalog Update Job again.

For information about creating an ACL template or ACL policy and controlling server access with agent ACLs, see Managing access.

Windows Hotfix Patch is not found in the Catalog

For steps on troubleshooting this issue, see Windows Hotfix Patch not found in the Catalog.

Catalog update job fails because of PD5, or HF7b configuration files.

For Windows patching, you may still be using the PD5.cab, or HF7b.cab ( PD5.xml, or HF7b.xml) configuration files for Windows patching. However, BMC Server Automation do not support the PD5, or HF7b configuration files, and you must use the WindowsPatchData.zip file instead. The Windows catalog update job fails if you use the .cab or .xml configuration files.

To update the configuration files used for Windows patching see, Global configuration parameters.

File a ticket with BMC Support and provide the information from the following logs:

• Patch Analysis Job log
• rscd.log from file server and target

Turning on debug tracing for cl5.exe

To validate the payload, the shavlik engine runs the cl5.exe tool on the target. If this tool is not working properly, you can turn on tracing, using the following command:

CL5.exe 2097197 1 <Log Directory>

The customary value for maximum log size is 5000000. This command writes some values to the registry.

To turn off CL5 tracing, use the following command:

CL5.exe 2097197 0

You can refer to the following article in the knowledge base for steps on troubleshooting this issue:
Windows Patch Analysis or Catalog Update Job error: Unable to load the shavlik scan engine dll

If your issue is still not resolved, file a ticket with BMC Support and provide the information from the following logs:

• Patch Analysis Job log
• rscd.log and Trace.txt from target

File a ticket with BMC Support and provide the information from the following logs:

• Patch Analysis Job log
• rscd.log and Trace.txt from target
• appserver.log

For information about troubleshooting this case, see Windows Patch Analysis Job reports an unexpected patch as missing.

You can also refer to the following articles in the BMC knowledge base:

• Windows Patch Deploy Job error: Could not load patch details file
• Patch Analysis or Deploy error: Unable to load the shavlik packager engine dll
• Windows Patch Analysis error: Difference in results with and without using Include filter (whitelist)

For information about troubleshooting this case, see Windows Patch Analysis Job does not report an expected missing patch.

For more information about troubleshooting this case, see Analyzing differences between Windows Patch Analysis Job results and third-party vendors.

The following steps are executed:

1. BLPackage of repodata.tar.gz and linux_analyze is created
2. The BLPackage is deployed to the target server at ??STAGING_DIR??/LinuxCatalog_<hash>

A custom yum.conf file is generated on the target that is used by the yum utility. Note that the system's default file is not used during analysis.

A custom yum.conf file is generated at ??STAGING_DIR??/LinuxCatalog_<hash>. The ??STAGING_DIR??/LinuxCatalog_<hash> path contains the following files:

• Compressed yum repository data from the catalog (repodata.tar.gz). The contents of the repodata.tar.gz file are extracted in the repository directory during analysis.
• Yum configuration file (yum.conf) is generated by the analysis job based on yum.conf settings in the Patch Global Configuration.
• Binary that builds the include lists and runs yum with options (linux_analyze)

The following steps are executed:

1. linux_analyze calls <rscd install dir>/bin/blyum (or just /usr/bin/yum on platforms where the native yum is used) -C -c ./yum.conf [ update | install ] <include list>
2. A number of output files are generated. These files contain various outputs:
   • Result of the yum upgrade/install command (yum_analysis.res)
   • Stderr of the linux_analyze command (analysis_err.log).
   • Stdout of the linux_analyze command (analysis_log.log)
   • Count of installed RPMs (installed_rpms.log)
   • List of installed RPMs (yum.lst)
   • Analysis results (analysis_res.log)
3. If the analysis is successful the required RPMs are sent back to the application server and displayed in the analysis results.
4. Depending on whether the DEBUG_MODE_ENABLED property on the patching job is set to 'true' or 'false' the following is executed:
   
   • If set to True: The output files (listed above) from each target server are copied to the <application server install dir>/NSH/tmp/debug/<appserver instance name> directory for investigation
   • If set to False: The <application server install dir>/NSH/tmp/debug/<appserver instance name> directory is deleted.

Scenario 1: Missing Dependency: [rpm1] = [version] is needed by package [rpm2-version.arch] (installed)

For more information, see Yum failed to complete - Scenario 1.

Scenario 2: Missing Dependency: [rpm1] = [version] is needed by package [rpm2-version.arch] (repo)

For more information, see Yum failed to complete - Scenario 2.

Scenario 3: [rpm1] conflicts with [rpm2]

For more information, see Yum failed to complete - Scenario 3.

Scenario 2: Transaction Check Error: package [rpm-version] is already installed.

For more information, see Deploy Job failed - Scenario 2.

Kernel RPMs removed during Patch Analysis Job

During Patch Analysis Job, warning messages notify you that old RPMs were removed. This happens when the native yum is used, and old packages are automatically removed, as controlled by the installonly_limit option in the yum.conf file. By default, only the last 3 packages are kept.

To turn off the removal of old RPMs, set the installonly_limit option to a value of 0 on the yum.conf tab in the Patch Global Configuration dialog box or the  Patching Job - Analysis Options panel.

The error messages suggest that either blyum is not found on the target or it is found but some libraries that are needed to run blyum were missing.

To fix this error or research further, review the analysis_err.log error log file in the staging directory.

This error is usually encountered when the supported agent version or architecture is not installed. So, to troubleshoot this issue, validate that the supported agent version or architecture is installed.

This error message indicates that the Catalog Job did not succeed at a target where the repodata.tar.gz file was not generated. As a result, the Analysis Job was run against an invalid target whose OS level or architecture did not match. This problem might be caused by an issue on the Repo Server.

To fix this error, perform the following:

1. Run the Catalog Update Job again, and ensure that it ran successfully at all targets.
2. Validate that the target is correct by right-clicking and verifying the OS information.
3. Validate the ACLs on the Repo server.

• Patch Analysis Job log
• Catalog Update Job log

• rscd.log
• agentinfo.log

From the Repo server

• rscd.log

Both rpm1 of the specified version and rpm2 are installed on the target. A newer version of rpm1 is found in the Catalog and set to be updated. The installed version of rpm1 is also a dependency to rpm2. To approve the installation of a newer rpm1, yum also needs to update rpm2. However, yum cannot find a newer version of rpm2 in repodata.tar.gz, so rpm2 is excluded from analysis.

As a new version of rpm2 is not found, the rpm is not offered an update. Because rpm2 is not updated, yum cannot allow the update of rpm1. An error is logged, alerting that rpm1 needs to remain installed to preserve the dependency of rpm2.

To troubleshoot this error, follow these steps:

1. Validate that rpm2 is missing from the catalog.
2. If rpm2 is missing from the Catalog, check if it exists in the RHN Channel.
   • If not in the RHN Channel, then analysis happens as expected.
     1. Acquire rpm2 by other means and update it along with rpm1.
     2. Exclude Rpm1 from the analysis.
        Note that another rpm might require the newer version of the excluded rpm.
     3. Uninstall rpm2.
        When you uninstall rpm2, consult your administrator and use caution.
   • If present in RHN Channel, run the Catalog Update Job again.
     • If rpm2 is still not in the catalog, and this is not an RBAC-related issue, then collect logs and contact BMC Support.
     • If rpm2 is now present in the catalog, run the analysis again.
       If Analysis still shows the same error, continue to the next step .
3. If rpm2 is present in the catalog, check whether rpm2 is present in repodata.tar.gz on the target by running the following command:
   

   # cd <staging>/LinuxCatalog_XXX_[target]/
# yum -C -c yum.conf search [rpm2]
   
   Proceed according to the results that you obtain:

   • If rpm2 is not found, then this could be why it is not offered during Analysis.

   • If rpm2 is found, collect logs and contact BMC Support.

Neither rpm1 of the specified version nor rpm2 are installed on the target. Yum sets rpm2 to be updated for one of two reasons:

• Yum detected an older version of rpm2 installed.
• Yum offered rpm2 as a dependency to some other rpm.

Yum checks if rpm2 needs any dependencies of its own, and detects that rpm1 of the specified version is needed as a dependency. Therefore, yum attempts to set rpm1 to be updated as well, but fails. An error is logged, alerting that rpm1 is required for rpm2 to be installed.

To troubleshoot this error, follow these steps:

1. Check if rpm1 of the specified version exists in the Catalog.
2. If rpm1 is not in the Catalog, check if it exists in the RHN Channel.
   • If not in RHN Channel, then analysis happens as expected.
     1. Acquire rpm1 by other means and install it manually.
     2. Exclude rpm2 from the analysis or uninstall rpm2.
        When you uninstall rpm2, consult your administrator and use caution.
        Note that another rpm might require the newer version of the excluded rpm. Furthermore, this step will not work if yum offered rpm2 as a dependency to some other rpm.
   • If present in the RHN Channel, run the Catalog Update Job again.
     1. If rpm1 is still not in catalog, and this is not an RBAC-related issue, then collect logs and contact BMC Support.
     2. If rpm1 is now present in the catalog, run the analysis again.
        If Analysis still shows the same error, continue to the next step.
3. If rpm1 is present in the catalog, check whether rpm1 is present in repodata.tar.gz on the target by running the following command:
# cd <staging>/LinuxCatalog_XXX_[target]/
# yum -C -c yum.conf search [rpm1]

Proceed according to the results that you obtain:
   • If rpm1 is not found, then this could be why it is not offered during Analysis.
   • If rpm1 is found, something might be preventing it from being installed.
     In such a case, try performing a dry run installation of rpm1 using the following command:
     # rpm -iUv --test rpm1
     If the dry run fails, then yum predicted this failure and that is why yum rejected rpm1 from being updated, and rpm1 was no longer considered to be updated. An error is logged, indicating rpm1 as a missing dependency.

• rscd.log
• analysis bundle.log (including repodata.tar.gz)
• rpm –qa

Rpm2 is installed on the target, while rpm1 is not. Yum attempts to offer rpm1 to be installed or updated, but then yum detects that some of the files to be updated by rpm1 are used by rpm2. Therefore, yum rejects the installation or update of rpm1. An error is logged, alerting that rpm1 cannot be installed. In general, the conflict suggests that rpm1 and rpm2 cannot coexist if rpm versions matter.

1. Check the RHN site for known conflict issues.
   A conflict may arise if either rpm1 or rpm2 are not in the repository.
2. Validate that the rpm is in RHN channel, in the Catalog, and in repodata.tar.gz, as described in scenario 2.
   Note that resolution cannot be guaranteed if the RPMs come from different channels.
3. Proceed according to your needs:
   • If rpm1 must be installed, check whether rpm2 can be uninstalled, downgraded, or upgraded to a non-conflicting version.
   • If rpm2 must be preserved, consider the following actions:
     • Check whether rpm1 can be excluded. Note that another rpm might require the newer version of the excluded rpm.
     • Check whether another non-conflicting version of rpm1 can be installed.
     • Check whether multiple versions of rpm2 are installed, where the conflicting one can be uninstalled.
4. If the previous steps do not resolve the problem, collect logs and contact BMC Support.

• analysis bundle.log (including repodata.tar.gz)
• rpm –qa

Prior to installing the packages, the Deploy Job runs a preliminary yum analysis. During this analysis scan, rpm-version.arch is found as missing. The Deploy Job does not find rpm-version.arch in the list of staged missing patches, and, therefore, the job cannot proceed.

An error is logged, alerting you that the patch payload is not found and cannot be installed. This error indicates that a discrepancy was found between the regular Patch Analysis and the preliminary yum scan during Deploy, or that rpm-version.arch was not copied during staging. The error message is written in either the Deploy Job log or the bldeploy log.

1. Review Patch Analysis results and check whether rpm-version.arch was found as missing.
2. If the file is missing, check whether it was packaged during patch remediation.
   
   • If it was packaged, check whether it points to an existing source. Open the object from inside the Catalog and review its Installable source value.
     
     • The root cause of the problem might be that the installable source points to a missing payload. In this case, rerun the Catalog or download the patch manually.
     • If the source points to an available payload, review the stage phase of the Deploy Job. Check for issues that might have occurred during the copying of the payload to the target.
   • If the object was not packaged, check for possible RBAC issues.

3. If patch was not found missing, compare yum scans of Analysis and Deploy:

   • Analysis: <staging>/LinuxCatalog_XXX_[target]/yum_analysis.res

   • Deploy: <staging>/XXXXX/blres.yum.depl

   The deploy log exists if the Deploy Job was set to preserve staging area on failure. If not, configure this setting now, then reproduce the error and collect the log file.

4. Compare the lists of set to be updated patches.
   This comparison might reveal that for some RPMs, the analysis finds one arch file missing, while Deploy finds the other one missing. This might be the cause of the original issue. If so, a fix is available in BMC Server Automation 8.1 Service Pack 3.

5. If the preceding steps did not reveal the root cause of the issue, collect several logs and contact BMC Support. From the application server, obtain the Patch Analysis Job log. In addition, obtain the following logs from the target:

   • Analysis bundle log
   • bldeploy log
   • <staging>/XXXXX/blres.yum.depl

The Deploy Job installs the patches that were found missing during Analysis. In this case, the Analysis Job packaged the rpm-version, even though it is already installed.

1. Review the Analysis logs to troubleshoot the issue. 
   The error message is located in either the Deploy Job log or the bldeploy log.
   You might see the following log records:
   • The analysis_log.log is expected to reveal that the Patch Analysis ran with an include filter. The signs for this are:
     
     • After 'update' you see a list of RPMs.
     • YUM Command: for i in 0 1 2 3 4 5 :; do echo n; done | blyum -c ./yum.conf -C update "rpm-version.arch" "…" ...
   • Command rpm –q [rpm] reveals that multiple versions of the rpm are installed.
2. The expected findings described in the previous step are not the standard configuration. Therefore, perform the following further testing:
   • Run yum update with include list.
     This is expected to produce the following error:
latest rpm as missing [issue reproduced]
     The updated rpm is offered for the older version of RPM that is installed even if the new version is also already installed.
   • Run yum update without include list.
     This should return the following message:
no missing rpm [good result]
     If we do not forcefully analyze for patches (with an include filter), then if the latest version is found, the rpm is not offered.
   • Run yum install with include list.
     This should return the following message:
no missing rpm [good result]
     The Install option checks only for the specific version of the rpm that we are attempting to install. Since our specific version of rpm is already installed, it will not be installed again.
3. If you must preserve multiple versions of such RPMs, create a separate analysis job where you would use Install option and include the RPM.
4. Modify the existing Analysis Job include filter and remove the RPM.
   If the RPM must be included, try excluding the opposite set of RPMs. For example, exclude all where RPM does not equal [rpm-version].
5. Yum does not install multiple versions of such RPMs by default, so if they are not required, then uninstall the old versions.
   The Analysis Job should no longer report the latest RPM as missing, because there will no longer be multiple versions.