Skip to Content

Troubleshooting Compliance job errors

A TrueSight Server Automation Compliance Job is failing with errors against one or more Target Servers.

This topic helps the reader to locate and review the appropriate log files to determine why the Compliance job is failing and either identify and resolve the issue or create a BMC Customer Support case.

Issue symptoms

A TrueSight Server Automation Compliance Job fails with an error message in the job run log.

Issue scope

  • This Troubleshooting Guide covers the Compliance phase of a Compliance and not Component Discovery or Remediation (Deploy).
  • The issue may occur on all targets against which the Compliance Job is executed or may be limited to a subset of the targets.

Diagnosing and reporting an issue

Task

Action

Steps

Reference

1

Understand the problem scope.

  • Is this Compliance Job using a custom (customer-created) Component Template or BMC-provided Component Template?
  • If the Component Template is BMC-provided content, what is the name and version of the Component Template?(refer to screenshot on the right for details on how to determine the Template name and version) For example,
    CIS Windows 2019 (Version 1.1.0)
    CIS - Red Hat Enterprise Linux 8 (Version 1.0.0)
    PCI Data Security Standard v3 - Windows Server 2016 (Version 3.2.1)
    PCI Data Security Standard v3 - Red Hat Enterprise Linux 7 (Version 3.2.1)
  • Is the Compliance Job performing Auto-Discovery or was a previous Component Discovery Job run?
  • What Compliance Job rule(s) are failing? (refer to screenshot on the right)
  • What is the Operating System vendor and version of the Target Servers? For example,. MS Windows 2016, RHEL 7 and so on.
  • How many total Target Servers is the Compliance Job running against?
  • Are the errors happening against all the Target Servers or a subset of the Target Servers?
    • If a subset, how many affected Target Servers?
    • What are the names of some of the affected Target Servers?
  • What is the version of the TSSA Application Server?
  • What are the versions of the affected TSSA RSCD Agents?

Template name and version details:

Teamplate_version_identifier.png


Identify failing rule(s)

Compliance error1.jpg

Compliance error.jpg

2

Test the individual Compliance Rule from the TrueSight Server Automation Console.

Can the errors be reproduced using the "Test Rule" functionality of the TrueSight Server Automation Console?

Testing the rule from the TrueSight Server Automation Console allows the user to reproduce and troubleshoot the behavior outside the context of a Compliance Job.

See steps in the Reference section on the right for details on testing a Compliance Rule from the TrueSight Server Automation Console.

  1. Open the Component Template.
  2. Open the Rule which appears to be displaying incorrect results.
  3. Click Play to test the rule.

    10.png
  4. Add the Component or Target Server (you need to do this only once).

    11.png

  5. Click Run Test.

    12.png

  6. Review the results.

For more information, see Testing-a-compliance-rule.

3

Test the command(s) run by the rule directly on the Target Server.

If the Compliance Job errors are confined to specific rules , can the commands used by the rules be run directly on the Target Server?

Does this manual run generate the same error?

This step may not apply if the errors are not related to specific rules/Targets but are a general issue with the Compliance Job.

For example, if the Compliance Rule is checking the permissions of a file, this can be validated directly on the Target Server and also via a Live Browse from the TrueSight Server Automation Console. For example,

  1. Login to the impacted Target Server OR Live Browse the Target Server from the TrueSight Server Automation Console.
  2. Go to the file location in question For example, cd /etc
  3. Check the permissions on the file
    In this example, it is 000.

See reference section on the right for an example of checking file permissions directly from a Target Server and from the TrueSight Server Automation Console.

Different Compliance Rules will check for other conditions which can similarly be checked. For example,

  • Results of a script executed via an Extended Object
  • Does a file exist
  • Does a registry entry exist
  • checksum of a specific file
  • Value of a registry entry
  • ownership of a file
  • Presence of a specific entry in a Configuration file

Checking file permission directly from Target Server:

5.png

To check the file permission by Live Browsing the Target Server:

  1. Right click the target server and select Browse.

    7.png
  2. Go to 'File System' > /etc/shadow and validate the permissions.

    8.png

    9.png

4

Enable Debug Logging.

The log4j.properties or log4j2.xml file (depending on TSSA version) on an Application Server can be modified to enable debug-level logging for two specific Compliance Level logger classes.

See detailed steps in reference section on the right.

An Application Server restart is not required after enabling this debug.

For simplicity, the debug-level logging can be enabled on just one Application Server and a Job Routing rule can then be added to route the next run of the Compliance Job to this specific Application Server.

Once the additional debug-level logging has been enabled, and the job routing rule has been created, rerun the Compliance Job to generate the additional debug-level logging in the Application Server log.

Logging properties file location:

<installation_directory>/NSH/br/deployments/<deployment>

  • Log4j.properties (TrueSight Server Automation version 8.9.04 P3 or earlier)
    • log4j.logger.com.bladelogic.om.infra.ast.visitor.conditionresult.evaluator.NewCondition
      ResultSuccessVisitor=Debug
    • log4j.logger.com.bladelogic.om.infra.model.asset=Debug
  • Log4j2.xml (TrueSight Server Automation version 20.02)
    • <Logger name="com.bladelogic.om.infra.ast.visitor.conditionresult.evaluator.NewCondition
      ResultSuccessVisitor" level="DEBUG"/>
    • <Logger name="com.bladelogic.om.infra.model.asset" level="DEBUG"/>

See KA 000389694 for additional details.

5

Export the Compliance Job Results.

Export the detailed Compliance Job results by Rule or by Server.

See detailed steps in reference section on the right.

  1. Expand the Compliance run.
    Rule1.jpg
  2. Right click Rules View and select Export Compliance Results.
    Rule2.jpg

  3. Or Right click Server View and select Export Compliance Results.
    Rule3.jpg

Both type of log export provide option for “Generate detailed reports for individual servers.”

6

Generate the Compliance Job Log Package.

Generate the Compliance Job Log Package for review by BMC Customer Support.

Right-click a failed Compliance Job Run (where debug-level logging had been enabled in step 4) and select "Download Log Package" to capture the required logs. (refer to screenshot on the right for details)

  1. Right-click the job run and select Download Log package.
    Step1.jpg
  2. Specify a location on the local system:
    Step2.jpg
  3. Filter the list of Target Servers by status to select the Target Servers from which to collect agent logs.
    • All Success
    • All Failure
    • All Warning
  2. Select the desired set of targets:
  3. Select the Save button in order to begin the download process in the background. The download progress can be tracked in the bottom right corner of the console:
    step5.jpg

Once process is complete it will show a popup window confirming the logs are downloaded/generated:

Reference Video:

7

Analyze error(s) from Compliance Job Run Log

Review the specific error message gathered from the Compliance logs (steps 4-6 above)

Use the table in the "Resolutions for common issues" section below to review common errors which can result in Compliance Job failures along with how they can typically be resolved.

If unable to identify and resolve the problem, see Step 8 to create a BMC Support Case.


8

Creating a BMC Support Case

Provide the following information and log files when creating a case with BMC Customer Support:

  • Scope of the issue as identified in step 1 above.
  • Results of the "Test Rule" performed in step 2
  • Results of the tests performed directly on the Target Server and via TSSA Live Browse in step 3
  • Export of detailed Compliance Job results generated in step 5
  • Job Log Package generated in step 6


Resolutions for common issues

Symptom

Action

Reference

Compliance job fails with this error:

"Unable to copy script "<SCRIPT_NAME>" to host TARGET: No such file or directory

or

The file <FILE_NAME> doesn't exist.

This issue occurs when required sensors (script) files are not present on the application server.

Make sure the Compliance content and required files are present in sensor folder on each application server.

See the referenced Knowledge Article for full details.

Compliance Job fails with an error:

"Server Config Definition Object Not Found. The reason could be that either the Config Definition Object does not exist OR the user does not have required permissions on the Config Definition Object"

Missing the RBAC permissions on Config File Object definition for the role executing the job.

See the referenced Knowledge Article on the right for full details.

Compliance rules show Indeterminate results.

"Indeterminate" usually means that the compliance rule returned an empty or null value (while expecting a non-empty value), thus unable to process/verify the value.

Sometimes "Indeterminate" compliance result can be displayed due to a timeout, where a compliance command is taking longer than the default value.

See the referenced Knowledge Articles for full details.

Compliance job displays this warning:

"truncated data from job result for rule"

When running a compliance job, the job generates the following warning:

"truncated data from job result for rule Find unauthorized SUID/SGID system executables"

The affected template and rule may vary. The target lists as failed in the run details and there is no server view created for the server.

See the referenced Knowledge Articles for full details.

Compliance job fails with the following message:

"cancelled work-item execution for component"

The "cancelled work-item execution for component" message means that the defined JOB_PART_TIMEOUT on the Compliance Job was reached.

Review the JOB_PART_TIMEOUT defined on the job to ensure it is set high enough to allow for servers where a specific operation will require extra time.

If the JOB_PART_TIMEOUT value seems to low, increase and rerun.

See the referenced documentation on the right for full details on increasing JOB_PART_TIMEOUTs.

Note:

If the JOB_PART_TIMEOUT value already seems high, this should be viewed as a performance or hang issue.

See the corresponding Troubleshooting Guide for assistance in Troubleshooting Compliance Performance.

Compliance job which runs Extended Objects fails with a "Permission Denied" error from scriptutil.

The Compliance Job references an Extended Object which uses scriptutil to run sensor scripts.

The Compliance Job is failing with a "Permission denied" error from scriptutil For example,

% scriptutil -h servername -s /tmp/myscript.sh
/bin/bash: /tmp/_servername -92335-139237268-0-myscript.sh: Permission denied

See the referenced Knowledge Articles for full details.

 

Tip: For faster searching, add an asterisk to the end of your partial query. Example: cert*

TrueSight Server Automation 20.02