Troubleshooting Compliance job performance issues

Understand the problem scope.

• Is this Compliance Job using a custom (customer-created) Component Template or BMC-provided Component Template?
• If the Component Template is BMC-provided content, what is the name and version of the Component Template?(refer to screenshot on the right for details on how to determine the Template name and version) For example,
  CIS Windows 2019 (Version 1.1.0)
  CIS - Red Hat Enterprise Linux 8 (Version 1.0.0)
  PCI Data Security Standard v3 - Windows Server 2016 (Version 3.2.1)
  PCI Data Security Standard v3 - Red Hat Enterprise Linux 7 (Version 3.2.1)
  
• Is the Compliance Job performing Auto-Discovery or was a previous Component Discovery Job run?
• What Compliance Job rule(s) are running slower than expected?
• What is the Operating System vendor and version of the Target Servers? For example, MS Windows 2016, RHEL 7 and so on.
• How many total Target Servers is the Compliance Job running against?
• What is the job parallelism set to?
• Are the performance issues happening against all the Target Servers or a subset of the Target Servers?
  • If a subset, how many affected Target Servers?
  • What are the names of some of the affected Target Servers?
• What is the version of the TSSA Application Server?
• What are the versions of the affected RSCD Agents?
• Is the database Oracle or SQL Server?
• Is the performance problem intermittent, occurring only on some runs, or consistent?
• Does the Compliance Job have a JOB_TIMEOUT and/or JOB_PART_TIMEOUT property value defined? If so, what are the values for each?

Template name and version details:

Review Component Template for unnecessary template parts.

During a Compliance Job run, the following order of events occur:

1) All Template Parts are first collected from the Target.

2) Compliance Rules are evaluated.

A common cause of Compliance performance problems can be that the Component Template has Template parts defined which are not used by Compliance Rules.

This can result in the Template Part being unnecessarily collected from the Target Servers when they are not required for Compliance Rule evaluation.

Review the Template Parts of the Component Template and remove any parts not used by Compliance Rules.

Test the individual Compliance Rule from the TrueSight Server Automation Console

Test the command(s) run by the rule directly on the Target Server

If the Compliance Job performance issues are confined to specific rule(s)on specific Target(s) , can the commands used by the rules be run directly on the Targets?

If so, does this manual run encounter the same performance issues?

This step may not apply if the performance issues are not related to specific rules/Targets but are a general issue with the Compliance Job.

For example, if the Compliance Rule is checking the permissions of a file, this can be validated directly on the Target Server and also via a Live Browse from the TrueSight Server Automation Console. For example,

1. Login to the impacted Target Server OR Live Browse the Target Server from the TrueSight Server Automation Console
2. Go to the file location in question For example, cd /etc
3. Check the permissions on the file
   In this example, it is 000.

See reference section on the right for an example of checking file permissions directly from a Target Server and from the TrueSight Server Automation Console.

Different Compliance Rules will check for other conditions which can similarly be tested directly on the Target. For example,

• Results of a script executed via an Extended Object
• Does a file exist
• Does a registry entry exist
• checksum of a specific file
• Value of a registry entry
• ownership of a file
• Presence of a specific entry in a Configuration file

Enable Debug Logging

The log4j.properties or log4j2.xml file (depending on the version) on an Application Server can be modified to enable debug-level logging for two specific Compliance Level logger classes.

See detailed steps in reference section on the right.

An Application Server restart is not required after enabling this debug.

For simplicity, the debug-level logging can be enabled on just one Application Server and a Job Routing rule can then be added to route the next run of the Compliance Job to this specific Application Server.

Once the additional debug-level logging has been enabled, and the job routing rule has been created, rerun the Compliance Job to generate the additional debug-level logging in the Application Server log.

Generate Compliance Job Log Package

Generate the Compliance Job Log Package for review by BMC Customer Support.

Right-click a failed Compliance Job Run (where debug-level logging had been enabled in step 4) and select "Download Log Package" to capture the required logs. (refer to screenshot on the right for details)

Analyze log files to understand where the performance issues(s) were encountered.

Creating a BMC Support Case.

Provide the following information and log files when creating a case with BMC Customer Support:

• Scope of the issue as identified in step 1 above
• Results of the "Test Rule" performed in step 3
• Results of the tests performed directly on the Target Server and via Live Browse in step 4 (if applicable)
• Job Log Package generated in step 6