Improving performance of reconciliation
You can perform various task to improve the performance of reconciliation. These include managing old records, configuring the reconciliation engine, changes to the database, improving indexing, and so on.
To improve performance by managing old reconciliation configuration related records
- Check the free memory space on the AR System server that runs the reconciliation job.
Space is required to write the logs or just remove the old logs at <ARSystemServerInstallDir>\BMCData\ARSystem\ARServer\Db\re.
- Clear the old records from RE:Job_Runs and RE:Job_Events AR forms.
The URL to open these forms are:
Basically, the RE:Job and RE:Job Events tables store the reconciliation job history records. The following image shows the interface from Developer Studio when you open the Run Status field of RE:Job Runs. In the following screenshot, note the various status values including Started, Success, Failed, and so on.
This table is automatically populated by the reconciliation engine to store the records of reconciliation jobs that were run and the ones that are currently in progress.
This table is populated with huge records and may later cause performance issues. Clear the records from this table with the Status field values, such as Success, Failed, Cancelled, Aborted, Queued, Paused. Delete the records with Paused and SGPaused only if they are very old. Keep the records with status as Started.
Similarly, you can delete the old records from RE:Job Events table as well. Records in RE:JOB_RUNS and RE:JOB_EVENTS forms are related. If you manually delete data, you must ensure data consistency. If archiving has been set up on the RE:JOB_RUNS form the BMC Developer Studio tool, then its association will take care of deleting the related data from RE:JOB_EVENTS after the records are deleted from RE:JOB RUNS.
Another way of maintaining RE:JOB_RUNS and RE:JOB_EVENTS forms is by archiving the data.
To improve performance by archiving RE:JOB_RUNS and RE:JOB_EVENTS form data
Important: To make changes in the Age Qualification field, you must select the overlay type as Overwrite.
- Log in to BMC Developer Studio.
- Open the form RE:JOB_RUNS to configure it for data archiving.
- Click the Definitions tab.
- Expand Other Definitions > Archive and set the following parameters.
- Save the form.
- Open the RE:JOB_EVENTS from and set the following parameters.
- Save the form.
To improve performance by configuring the reconciliation engine
Perform the following steps to increase Definition Check interval, Polling interval, and Continuous job interval. It is recommended to process only normalized CIs if you want to merge only Normalized CIs into the asset dataset.
- Open the Reconciliation Console from the Atrium Core Console.
- Click the Edit Configuration button on the toolbar.
The Server Configuration Editor dialog box appears.
- Increase the value of Definition Check interval, Polling Interval, and Continuous Job Interval.
- Click Save.
Saved changes are applied after a delay of about 10 seconds.
- Select the Disable Progress bar in UI during Job execution.
The options are described in the following table:
Definition Check Interval
|An expiration, in seconds, of the reconciliation engine's cache of job definitions. The default is 300. For example, if you change job, activity, precedence, or merge definitions, and then run a job, you might not see the results after the cache expires. Setting an expiration for the cached definitions helps to improve performance and reduce the log size.|
Increase Polling Interval
|The polling interval in seconds. The reconciliation engine remains inactive until the next job is scheduled to run or for this interval value to elapse, whichever occurs first. Polling interval is used only when the dispatcher does not send a signal to the reconciliation engine. The default is 18000 seconds (5 hours).|
|Increase Continuous Job Interval||The execution interval of a continuous job in seconds. The default value is 1800 seconds (30 minutes). If you set this value to less than 120 seconds (2 minutes), then the reconciliation engine automatically changes that value to 120 seconds (2 minutes).|
To improve performance by making changes in the AR System server configuration
- Increase the number of threads on RPC socket 390698 or the one used by reconciliation engine to the limit that the CPU can handle.
The formula is (2 * Number of CPU on AR System server running reconciliation job). You can do this in the BMC Remedy AR System Administration Console, at System > General > Server Information
- If the CI count is in millions, you may assign a dedicated AR System server for the reconciliation engine, and ensure that no other data management jobs such as Atrium Integrator jobs run on this server.
- Check the ARError.log for any errors that can impact reconciliation and fix them.
- Check CMDB_ENG_DEBUG_LOG.LOG log for any errors that can impact reconciliation and fix them.
- Restart AR System server service.
- Check for any known issues around AR or CMDB for the version in use and resolve the issues.
To improve reconciliation performance by indexing
Indexing can be used for improving reconciliation performance. Indexing a class form can be used to avoid the slow running of queries on the class form’s table.
- If the reconciliation job is taking a lot of time to process CIs, a combined SQL API and filter logs must be generated on the Remedy AR System server and analysed using the Log Analyzer tool.
The log analyzer tool report helps you identify the slow-running APIs and SQLs to determine the attributes to be considered for an index.
- Share the log analyzer tool report with the database administrator and emphasize on the long running queries.
Let the database administrator run the database diagnostic tools to generate a query execution plan on the MS SQL Server or an Automatic Workload Registry (AWR) report on the Oracle server while the reconciliation job is running. Both, the Log Analyzer and database reports, help the database administrator to decide which attributes need indexing and its sequence. Usually, the indexes are based on the attributes used in the identification rule. Add an index using the attributes used in the identification rule. See, Defining class properties in the class manager for more information.
Avoid the LIKE operator if possible.
Identification rules with the LIKE operator increase the time taken to return the results.
To improve reconciliation performance by making changes to the database
- Generate combined AR System server API, SQL, and Filter log at regular intervals while the reconciliation job runs.
Set a log file size as large as 10 MB each and set the count of files to be 10.
In case a particular filter has been identified as running into a loop causing performance issues, then it needs to be corrected. Check if the filter is a custom filter or is out-of-the-box from a Remedy application.
- Analyze the logs using the Log Analyser tool and find out if there is an API and its SQL statement running for a long time.
Introduce indexes on the tables (CMDB class forms) which has long running queries. You may require the assistance of the database administrator to decide on the fields and its order in which an index has to be created.
- Identify the long running queries using the Log Analyser tool report and use respective database tools to fine tune queries.
If you are using an Oracle database, generate an AWR (automatic workload repository report) when running a reconciliation engine job or a long running SQL query.
You must then let the database administrator use Oracle Advisor tools to fine tune queries. Similarly, run a query execution plan on the SQL server when running a long running query.
- Verify the database settings section from the following Knowledge Article.
- Check with the database admin if there has been a recent patch or version upgrade in the database around the time the performance issue started.
- Monitor the CPU and Memory consumption on the database while running reconciliation jobs.