Improving reconciliation performance by increasing threads

The Reconciliation Engine might have performance issues if the number of threads you configured for the Remedy AR System server are insufficient or if the database settings that you specified is incorrect.

• The Reconciliation Engine log file contains no more than one or two different thread IDs (TID). This indicates that the Remedy AR System server is not configured to use multi-threads.
  This problem might occur if you have not configured the Reconciliation Engine to create multiple threads for different activities.

• If the Reconciliation Engine takes more time than usual to complete a job, make sure your database setting are correct. This issue can occur because of any of the following reasons:
  • The default cursor_sharing parameter in Oracle 10g is set to exact.
  • The Oracle database instance is allocated only a small amount of memory.
  • SQL Server is allocated insufficient amount of space in the tempdb database.

The following topics provides solutions for insufficient thread and incorrect database settings

To improve reconciliation performance by increasing threads

1. Log in to CMDB Portal as an administrator, and open Reconciliation from the Configurations > Core Configurations.
2. In the RPC field, specify 390698 or 390699.
3. Click Save.
   After you make the RPC Socket changes, the ar.cfg (ar.conf ) configuration file is updated with the RE-RPC-Socket: 390698 or 390699 entry.
4. Log in to Remedy AR System server.
5. Open the AR System Administration: Server Information form in Search mode and click Search on the toolbar.
   The Server Information for your Remedy AR System server is displayed.
6. Select the Ports and Queues tab.
   The thread information for the server is displayed in the Server Queue table.
7. In the Server Queue table, increase the Max threads for any of the types of threads: Fast, List, or Private.

Stack trace for thread crashes

In BMC CMDB, stack trace information for thread crashes is logged in the arrecond.log file. You can send the stack trace information to Customer Support to determine the file, function, and line number that caused the crash. This feature requires no configuration and is available by default.

When a thread crash occurs, the arrecond.log file contains an entry, as shown in the following example. When you see the Thread Id <idnumber> crashed entry, provide BMC Support with the Stack Begin line through the Stack End line. These lines provide information for locating the function and line number.

2011/11/22 16:19:45.7414 [ ERROR ] TID: 0000000008 : Thread Id <000008> crashed
2011/11/22 16:19:45.7422 [ ERROR ] TID: 0000000008 : Stack Begin:
/opt/bmc/AtriumCore/cmdb/server/bin/arrecond:_1cGREUtilSDumpStackTraceUnix6Fl_v+0x110
/opt/bmc/AtriumCore/cmdb/server/bin/arrecond:REThreadImplDeath+0xe8
/lib/sparcv9/libc.so.1:0xcd9d4
/opt/bmc/AtriumCore/cmdb/server/bin/arrecond:UnixThreadStartRoutine+0x17c
/lib/sparcv9/libc.so.1:0xcd69c

2011/11/22 16:19:45.7558 [ INFO ] TID: 0000000001 : Reconciliation Engine started
2011/11/22 16:19:45.7561 [ ERROR ] TID: 0000000008 : Stack End
2011/11/22 16:19:45.7564 [ ERROR ] TID: 0000000015 : Thread Id <000015> crashed
2011/11/22 16:19:45.7568 [ ERROR ] TID: 0000000015 : Stack Begin:
/opt/bmc/AtriumCore/cmdb/server/bin/arrecond:_1cGREUtilSDumpStackTraceUnix6Fl_v+0x110
/opt/bmc/AtriumCore/cmdb/server/bin/arrecond:REThreadImplDeath+0xe8
/lib/sparcv9/libc.so.1:0xcd9d4
/opt/bmc/AtriumCore/cmdb/server/bin/arrecond:UnixThreadStartRoutine+0x17c
/lib/sparcv9/libc.so.1:0xcd69c

2011/11/22 16:19:45.7619 [ ERROR ] TID: 0000000015 : Stack End
2011/11/22 16:19:45.8370 DETAILS TID: 0000000001 : + getServerGroupState