How to analyze Trace.txt generated by a Windows Patch Analysis Job

This article illustrates what information can be learned from the Trace.txt log file (and the additional results.xml file) generated by the Windows Patch Analysis Job or by live browsing Hotfixes objects.  This document is based on Knowledge Article ID: 000096560

The information below is applicable to the Trace.txt generated by Shavlik engine 6.0, which is used in BMC Server Automation versions 8.2.or earlier. Most of this information also applies to Shavlik engine 8.0, which is used in BMC Server Automation 8.2 SP1 and later.

The following check points are discussed:

How and where the files are generated

Set the target Agent to debug, so you can see the exact command that is run to perform the scan.
Executable used for the scan: RSCD_DIR/sbin/BLPatchCheck2.exe

Location of the logs:

• C:\Trace.txt and C:\Trace.txt.old — By default, the Trace.txt file is deleted after a Patch Analysis Job is executed. To download a copy of the the Trace.txt file in the tmp directory on the Application Server, set the DEBUG_MODE_ENABLED property of the job to TRUE.
• RSCD_DIR/tmp/xxx/shavlik_results.xml — deleted after scan

The following command is executed during live browse Hotfixes, and it only scans for security patches:

BLPatchCheck2.exe 0 hf7b.xml shavlik_results.xml

The -pt option is used during Patch Analysis to specify patch types:

BLPatchCheck2.exe 0 -pt 13 hf7b.xml shavlik_results.xml

The following values are available for use with the -pt option:
    (1) Security patches (default)
    (4) Security tools
    (5) Security Patches, Security Tools
    (8) Non-Security patches
    (9) Security and non-security patches
    (12) Security Tools, Non-Security Patches
    (13) Security, non-security and tools

In the Trace.txt file, you can also see the following corresponding line:
... MultiMachineScanner.cpp:625 After License check - Scanning for patchtypes [13]

The -qf option is used to specify the whitelist:

BLPatchCheck2.exe 0 -qf whitelist.txt hf7b.xml shavlik_results.xml

The -xqf option is used to specify the blacklist:

BLPatchCheck2.exe 0 -xqf blacklist.txt hf7b.xml shavlik_results.xml

Was the CAB or XML metadata file passed to the scan?

The BLPatchCheck2.exe command in rscd.log references either hf7b.cab or hf7b.xml. The CAB file is the same XML, only packaged. The information can also be obtained from the Trace.txt file in the beginning:

The following is logged if the CAB file was passed to the scan (notice: extraction in process):

... CabExtracter.cpp:138 Extracting files from hf7b.cab
... CabExtracter.cpp:63 Extracting hf7b.xml to \hf7b.xml
... CabExtracter.cpp:49 Extracted \hf7b.xml from hf7b.cab
... ShavlikXMLDocumentSource.cpp:218 Using XML name of *\hf7b.xml*

The following is logged if the XML file was passed to the scan (notice: no extraction):

... ShavlikXMLDocumentSource.cpp:218 Using XML name of *hf7b.xml*

In the end, the actual analysis scan is performed against the XML file, either way.

Was a whitelist or Include Filter used?

If the rscd.log in debug is available, then the -qf option is shown with the BLPatchCheck2.exe command.
In Trace.txt you should see the whitelist logged as such:

... MultiMachineScanner.cpp:564 Qs to scan for
... MultiMachineScanner.cpp:571  Scanning for [Q2633880]
... MultiMachineScanner.cpp:571  Scanning for [Q2633870]
... MultiMachineScanner.cpp:571  Scanning for [Q2633874]
... MultiMachineScanner.cpp:571  Scanning for [Q2633879]
... MultiMachineScanner.cpp:571  Scanning for [Q2633873]

Scanning for [Qxxxxxx] messages will not be present in the Trace.txt if the whitelist was not used.

Sometimes you might see that the Q numbers repeat in the "Scanning for [Qxxxxxx]" column. This is most likely because you had a Bulletin added to the Include Filter instead of a Q number. The Bulletin may contain multiple hotfixes of the same KB or Q numbers for different Windows platforms or Product versions. To construct the whitelist, all KB numbers are extracted from the Bulletin.

Reviewing the logic stack of the patch or 'Reason'

The following examples illustrate what to look for when reviewing the logic stack.

Example 1

Here is a snippet from scanning the MS12-016 bulletin. The KB numbers are not usually listed during this check, so search the Trace.txt files using Bulletin IDs.

[target_hostname] ... PatchTest.cpp:354 TESTING MS12-016
[target_hostname] ... PatchTest.cpp:403
     Bulletin ID = MS12-016
     Filecount = 2
     Regcount = 0
[target_hostname] ... PatchTest.cpp:783 File C:\Windows\MICROSOFT.NET\FRAMEWORK\V2.0.50727\SYSTEM.DLL ErrNum=0
[target_hostname] ... PatchTest.cpp:785 File C:\Windows\MICROSOFT.NET\FRAMEWORK\V2.0.50727\SYSTEM.DLL (2.0.50727.4927)  C 2 2.0.50727.5703 (AC 5)
[target_hostname] ... PatchTest.cpp:783 File C:\Windows\MICROSOFT.NET\FRAMEWORK\V2.0.50727\SYSTEM.DLL ErrNum=0
[target_hostname] ... PatchTest.cpp:785 File C:\Windows\MICROSOFT.NET\FRAMEWORK\V2.0.50727\SYSTEM.DLL (2.0.50727.4927)  C 2 2.0.50727.4968 (AC 5)
[target_hostname] ... PatchTest.cpp:798 Is *A* FileChange
[target_hostname] ... PatchTest.cpp:427 A file was tested
[target_hostname] ... PatchTest.cpp:429 Did not pass file tests

Key notes

"Did not pass file tests" – suggests that the file version found on the target was less than expected, so the patch is considered Missing.
(2.0.50727.4927) – the file version in parentheses is the version found on the target
2.0.50727.5703 – the file version that appears next is where we expect to be

In this particular case we see two comparisons to the SYSTEM.DLL file of two separate versions. Sometime you might see even more comparisons under the same Bulletin, and they might not be against the same file. This happens because multiple hotfixes might be under the same Bulletin and they might fall under different branches (GDR and LDR), where each variation might be looking for a specific file/version. The results.xml file (or the Patch Analysis results from the BMC Server Automation Console) would have the correct one listed. Note that in rare cases, due to a Shavlik defect, the incorrect one might be listed; this is to be determined when cross-checking the specifications on a vendor site:
Reason="File version is less than expected. [C:\Windows\MICROSOFT.NET\FRAMEWORK\V2.0.50727\SYSTEM.DLL 2.0.50727.4927 < 2.0.50727.5703]"

Example 2

Here is a snippet from analyzing for the MS12-017 bulletin.

[target_hostname] ... PatchTest.cpp:354 TESTING MS12-017
[target_hostname] ... PatchTest.cpp:403
     Bulletin ID = MS12-017
     Filecount = 2
     Regcount = 0
[target_hostname] ... PatchTest.cpp:783 File C:\windows\system32\dns.exe ErrNum=1
[target_hostname] ... PatchTest.cpp:783 File C:\windows\system32\dns.exe ErrNum=1
[target_hostname] ... PatchTest.cpp:427 A file was tested
[target_hostname] ... PatchTest.cpp:1031 Reg passed

Key Notes

"Reg passed" – suggests that the condition passed, and the patch is not considered missing. File version comparisons are not listed.
"File C:\windows\system32\dns.exe ErrNum=1" – suggests that the file is missing from the system. If the file is missing, a certain service is not running; in this case DNS is most likely not activated. If the file does not exist on the system, then Shavlik considers the vulnerability not present, and therefore the patch is not considered missing. If the patch is not 'missing', and it is not explicitly 'installed', then such patch would be reported as 'EffectivelyInstalled'. 

Example 3

Sometimes the registry keys are referenced when performing the checks, so the reason is not only limited to files. Here is a snippet for the MS12-A04 bulletin.

... PatchTest.cpp:528 Testing 'MS12-A04'.
... PatchTest.cpp:1418 Key 'SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall\{06b2b7ed-809a-44e6-8538-ca0f5b74ecc4}.sdb'.
... PatchTest.cpp:1572 Reg failed reason - The registry key 'SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall\{06b2b7ed-809a-44e6-8538-ca0f5b74ecc4}.sdb\DisplayName' does not exist. It is required for this patch to be considered installed.

Key notes

"Reg failed reason - The registry key 'xxxxx' does not exist. It is required for this patch to be considered installed." – suggests that the patch is missing. If this patch were installed, then this particular registry entry would be present.

Did the scan complete gracefully or crash?

If the scan completes and BLPatchCheck2 exits gracefully, the following messages are logged to Trace.txt towards the end:

[target_hostname] ... MultiMachineScanner.cpp:955     Generating output for = [target_hostname]
[target_hostname] ... MultiMachineScanner.cpp:956     Scanned Server Count = 0, MAX count = 1

If the lines are not present, then the Patch Analysis Job most likely failed. Often, another error message appears towards the end of the Trace.txt log file.

Error messages and troubleshooting

Check if the error message from Trace.txt is documented in KCS.
When validating a conditions stack (reason), cross reference against the vendor site that the correct file and version is checked, and, if necessary, which branch is the applicable one (GDR or LDR).