×
 
 
  •  
$helper.renderConfluenceMacro('{bmc-global-announcement:$space.key}')

 

This documentation supports the 19.08 version of BMC CMDB.

To view an earlier version, select the version from the Product version menu.

BMC CMDB 19.08 ... Troubleshooting Reconciliation issues

Improving performance of reconciliation

You can perform various task to improve the performance of reconciliation. These include managing old records, configuring the reconciliation engine, changes to the database, improving indexing, and so on. 

To improve performance by managing old reconciliation configuration related records

  1. Check the free memory space on the Remedy AR System server that runs the reconciliation job.
    Space is required to write the logs or just remove the old logs at <ARSystemServerInstallDir>\BMCData\ARSystem\ARServer\Db\re.
  2. Clear the old records from RE:Job_Runs and RE:Job_Events AR forms. 
    The URL to open these forms are:

http://<midTierServer>:<port>/arsys/forms/<ARServer>/RE:Job_Runs
http://<midTierServer>:<port>/arsys/forms/<ARServer>/RE:Job_Events

Basically, the RE:Job and RE:Job Events tables store the reconciliation job history records. The following image shows the interface from Developer Studio when you open the Run Status field of RE:Job Runs. In the following screenshot, note the various status values including Started, Success, Failed, and so on.

This table is automatically populated by the reconciliation engine to store the records of reconciliation jobs that were run and the ones that are currently in progress. 
This table is populated with huge records and may later cause performance issues. Clear the records from this table with the Status field values, such as Success, Failed, Cancelled, Aborted, Queued, Paused. Delete the records with Paused and SGPaused only if they are very old. Keep the records with status as Started. 

Similarly, you can delete the old records from RE:Job Events table as well. Records in RE:JOB_RUNS and RE:JOB_EVENTS forms are related. If you manually delete data, you must ensure data consistency.  If archiving has been set up on the RE:JOB_RUNS form the BMC Developer Studio tool, then its association will take care of deleting the related data from RE:JOB_EVENTS after the records are deleted from RE:JOB RUNS.

Another way of maintaining RE:JOB_RUNS and RE:JOB_EVENTS forms is by archiving the data.

To improve performance by archiving RE:JOB_RUNS and RE:JOB_EVENTS form data

Important: To make changes in the Age Qualification field, you must select the overlay type as Overwrite.

  1. Log in to Remedy Developer Studio. 
  2. Open the form RE:JOB_RUNS to configure it for data archiving. 
  3. Click the Definitions tab. 
  4. Expand Other Definitions > Archive and set the following parameters.

  5. Save the form.
  6. Open the RE:JOB_EVENTS from and set the following parameters.

  7. Save the form.

To improve performance by configuring the reconciliation engine

Perform the following steps to increase Definition Check interval, Polling interval, and Continuous job interval. If you want to merge only Normalized CIs into the asset dataset, process only normalized CIs.

  1. Open the Reconciliation from Configurations > Core Configurations.
  2. Increase the value of Definition Check interval, Polling Interval, and Continuous Job Interval.
  3. Click Save.
    Saved changes are applied after a delay of about 10 seconds.
  4. In a reconciliation job, select the Disable Progress bar in UI during Job execution.

The options are described in the following table:

OptionDescription

Definition Check Interval

An expiration, in seconds, of the reconciliation engine's cache of job definitions. The default is 300. For example, if you change job, activity, precedence, or merge definitions, and then run a job, you might not see the results after the cache expires. Setting an expiration for the cached definitions helps to improve performance and reduce the log size.

Increase Polling Interval

The polling interval in seconds. The reconciliation engine remains inactive until the next job is scheduled to run or for this interval value to elapse, whichever occurs first. Polling interval is used only when the dispatcher does not send a signal to the reconciliation engine. The default is 18000 seconds (5 hours).
Increase Continuous Job IntervalThe execution interval of a continuous job in seconds. The default value is 1800 seconds (30 minutes). If you set this value to less than 120 seconds (2 minutes), then the reconciliation engine automatically changes that value to 120 seconds (2 minutes).

To improve performance by making changes in the Remedy AR System server configuration

  1. Increase the number of threads on RPC socket 390698 or the one used by reconciliation engine to the limit that the CPU can handle. 
    The formula is (2 * Number of CPU on AR System server running reconciliation job). You can do this in the Remedy Action Request System Administration Console, at System > General > Server Information
  2. If the CI count is in millions, you may assign a dedicated Remedy AR System server for the reconciliation engine, and ensure that no other data management jobs such as Atrium Integrator jobs run on this server.
  3. Check the ARError.log for any errors that can impact reconciliation and fix them.
  4. Check CMDB_ENG_DEBUG_LOG.LOG log for any errors that can impact reconciliation and fix them.
  5. Restart AR System server service. 
  6. Check for any known issues around Remedy AR System or CMDB for the version in use and resolve the issues.

To improve reconciliation performance by indexing

Indexing can be used for improving reconciliation performance. Indexing a class form can be used to avoid the slow running of queries on the class form’s table. 

  • If the reconciliation job is taking a lot of time to process CIs, a combined SQL API and filter logs must be generated on the Remedy Action Request System server and analyzed using the Log Analyzer tool. 
    The log analyzer tool report helps you identify the slow-running APIs and SQLs to determine the attributes to be considered for an index.
  • Share the log analyzer tool report with the database administrator and emphasize on the long running queries. 
    Let the database administrator run the database diagnostic tools to  generate a query execution plan on the MS SQL Server or an Automatic Workload Registry (AWR) report on the Oracle server while the reconciliation job is running. Both, the Log Analyzer and database reports, help the database administrator to decide which attributes need indexing and its sequence. Usually, the indexes are based on the attributes used in the identification rule. Add an index using the attributes used in the identification rule. See, Creating or modifying classes by using Class Manager for more information.

  • Avoid the LIKE operator if possible. 
    Identification rules with the LIKE operator increase the time taken to return the results.

To improve reconciliation performance by making changes to the database

If the Reconciliation Engine takes more time than usual to complete a job, make sure your database setting are correct. This issue can occur because of any of the following reasons:

  • The default cursor_sharing parameter in Oracle 10g is set to exact.
  • The Oracle database instance is allocated only a small amount of memory.
  • SQL Server is allocated insufficient amount of space in the tempdb database.

Perform the following actions to resolve the issues:

  1. Generate combined Remedy AR System server API, SQL, and Filter log at regular intervals while the reconciliation job runs.
    Set a log file size as large as 10 MB each and set the count of files to be 10.
    In case a particular filter has been identified as running into a loop causing performance issues, then it needs to be corrected. Check if the filter is a custom filter or is out-of-the-box from a Remedy application.
  2. Verify the database settings section from the following Knowledge Article. 
    https://bmcsites.force.com/casemgmt/sc_KnowledgeArticle?sfdcid=kA014000000h9kqCAA
  3. Check with the database admin if there has been a recent patch or version upgrade in the database around the time the performance issue started.
  4. Monitor the CPU and Memory consumption on the database while running reconciliation jobs.

To improve reconciliation performance by increasing threads

The Reconciliation Engine log file contains no more than one or two different thread IDs (TID). This indicates that the  Remedy AR System server is not configured to use multi-threads. This problem might occur if you have not configured the Reconciliation Engine to create multiple threads for different activities.

  1. Log in to CMDB Portal as an administrator, and open Reconciliation from the Configurations > Core Configurations.
  2. In the RPC field, specify 390698 or 390699.
  3. Click Save.
    After you make the RPC Socket changes, the ar.cfg (ar.conf ) configuration file is updated with the RE-RPC-Socket: 390698 or 390699 entry.
  4. Log in to Remedy AR System server.
  5. Open the AR System Administration: Server Information form in Search mode and click Search on the toolbar.
    The Server Information for your Remedy AR System server is displayed.
  6. Select the Ports and Queues tab.
    The thread information for the server is displayed in the Server Queue table.
  7. In the Server Queue table, increase the Max threads for any of the types of threads: Fast, List, or Private.

Best practice

The threads that you specify here are used when processing activities. We recommend that you create CPU x 5 for the List queue and CPU x 3 for the Fast queue or CPU x 1.5 for the Private queue. After you increase the threads, the ar.cfg (ar.conf ) file is updated with an entry for each of the thread types, for example, Private-RPC-Socket: 390698 2 4.

Warning

If you assign too many threads, for example 10 or 15, for any of these types, it can cause a system resource issue. The number of threads you can assign for activities depends on the number of CPU cores available in your computer and the number of connections the database can accept. To avoid the Reconciliation Engine from locking out any users, BMC recommends that you create 1.5 threads times the number of CPU cores where maximum number of threads configured is n-1 compared to the max fast or list thread.

Stack trace for thread crashes

In BMC CMDB, stack trace information for thread crashes is logged in the arrecond.log file. You can send the stack trace information to Customer Support to determine the file, function, and line number that caused the crash. This feature requires no configuration and is available by default.

When a thread crash occurs, the arrecond.log file contains an entry, as shown in the following example. When you see the Thread Id <idnumber> crashed entry, provide BMC Support with the Stack Begin line through the Stack End line. These lines provide information for locating the function and line number.

2011/11/22 16:19:45.7414 [ ERROR ] TID: 0000000008 : Thread Id <000008> crashed
2011/11/22 16:19:45.7422 [ ERROR ] TID: 0000000008 : Stack Begin:
/opt/bmc/AtriumCore/cmdb/server/bin/arrecond:_1cGREUtilSDumpStackTraceUnix6Fl_v+0x110
/opt/bmc/AtriumCore/cmdb/server/bin/arrecond:REThreadImplDeath+0xe8
/lib/sparcv9/libc.so.1:0xcd9d4
/opt/bmc/AtriumCore/cmdb/server/bin/arrecond:UnixThreadStartRoutine+0x17c
/lib/sparcv9/libc.so.1:0xcd69c

2011/11/22 16:19:45.7558 [ INFO ] TID: 0000000001 : Reconciliation Engine started
2011/11/22 16:19:45.7561 [ ERROR ] TID: 0000000008 : Stack End
2011/11/22 16:19:45.7564 [ ERROR ] TID: 0000000015 : Thread Id <000015> crashed
2011/11/22 16:19:45.7568 [ ERROR ] TID: 0000000015 : Stack Begin:
/opt/bmc/AtriumCore/cmdb/server/bin/arrecond:_1cGREUtilSDumpStackTraceUnix6Fl_v+0x110
/opt/bmc/AtriumCore/cmdb/server/bin/arrecond:REThreadImplDeath+0xe8
/lib/sparcv9/libc.so.1:0xcd9d4
/opt/bmc/AtriumCore/cmdb/server/bin/arrecond:UnixThreadStartRoutine+0x17c
/lib/sparcv9/libc.so.1:0xcd69c

2011/11/22 16:19:45.7619 [ ERROR ] TID: 0000000015 : Stack End
2011/11/22 16:19:45.8370 DETAILS TID: 0000000001 : + getServerGroupState
Was this page helpful? Yes No Submitting... Thank you
Last modified by Maithili Deshpande on Apr 29, 2021
best_practices troubleshooting reconciliation indexing performance increase_threads

Comments

Log in or register to comment.

Troubleshooting reconciliation based on error messages
Improving reconciliation performance by increasing threads
© Copyright 2005-2007, 2009-2021 BMC Software, Inc. © Copyright 2011-2021 BladeLogic, Inc.
Legal notices

Powered by Atlassian Confluence and Scroll Viewport

© Copyright 2005-2023 BMC Software, Inc. Use of this site signifies your acceptance of BMC’s Terms of Use . BMC, the BMC logo, and other BMC marks are assets of BMC Software, Inc. These trademarks are registered and may be registered in the U.S. and in other countries.