Troubleshooting CMDB reconciliation issues by collecting diagnostics

BMC CMDB reconciliation performance issues are evident when the reconciliation job is very slow in processing  the configuration items (CIs) and in completing the job.

Collecting issue symptoms

You must check whether any of the following are true:

• More than one instance of the same reconciliation job is running at the same time.
  You can view this on the Reconciliation page in Jobs > Manage Reconciliation.
• The ARRECOND process stops and restarts automatically.
  In Windows, you can view the ARRECOND process in the Windows taskbar.
  In Linux, you can view this process by running the following command:
  grep -i "arrecond".
• Changes were made to any Remedy Action Request System server, CMDB, or the database and the issue of slow reconciliation job started after that change.
  • If any configuration changes pertaining to the running of the Reconciliation engine were made in the Remedy Action Request System server.
  • You see any sudden increase in the count of CIs in CMDB classes, thus causing the slowness in running a Reconciliation job.
  • If any changes were made to the database side, such as addition or removal of any index and a new patch installation.
• CPU and memory consumption on the Remedy AR System server and ARRECOND process are unusually high when running the reconciliation job.
  In Windows, you can view the ARRECOND process in the Windows taskbar.
  In Linux, you can view the ARRECOND process by running the following command:
  grep -i "arrecond"
  Verify if consumption of CPU and memory usage on the Remedy AR System server running ARRECOND process is on a higher side when running the reconciliation job.
• The reconciliation job log indicates that there are duplicate CIs created in large numbers, which can delay the completion of the reconciliation job.
  You can find the reconciliation job log at:
  <ARSystemServerInstallDir>\BMCData\ARSystem\ARServer\Db\

Environment details

You must note the following environment details:

• The operating system of the Remedy AR System server running CMDB reconciliation.
• The version of Remedy AR System and CMDB. Whether any patch or hotfix was applied.
• Whether it is an Remedy AR System server group. If yes, whether CMDB is configured to be failover or a standalone for the reconciliation process.
• If there is any private RPC queue for Reconciliation Engine and if min/max threads are assigned to that queue.
• If private queue is not set for Reconciliation Engine then what is the min/max values for the fast and list queue (390620, 390635).

Issue scope

You must note the following aspects to know the extent of the issue:

• 
  
  • The activity in reconciliation job which is affected by the slow performance.
  • The number of CIs processed and yet to be processed by the reconciliation job.
  • The number of records present in the RE:Job_Runs and RE:Job_Events AR forms and whether there are any excess records in these forms.
    Use the following URLs to access the forms in Remedy AR System:
    http://<midTierServer>:<port>/arsys/forms/<ARServer>/RE:Job_Runs
    http://<midTierServer>:<port>/arsys/forms/<ARServer>/RE:Job_Events

To troubleshoot by gathering logs and configuration files

After you identify the symptoms and scope of the issue, perform the following steps to diagnose and resolve the issue or to create a BMC Support case. 

Save the following log and configuration files:

1. Take a snapshot of the following information:

   Information	Where to find it
   Open the Remedy Action Request System server Group Operation Ranking form through Mid Tier and search the records by selecting Reconciliation engine in the Operation list. Take a snapshot of each Remedy AR System server record from the results list or just export the records to a .csv file by clicking the Report button under the results list.	http://<midTierServer>:<port>/arsys/forms/<aRServer>/AR+System+Server+Group+Operation+Ranking
   Remedy AR System server system details such as RAM, CPU, and so on.	System information.

2. Perform the following tasks to view the job history, overall health, and integration check information.

   Task	Where to look or what do to
   View the health metrics on the dashboard	To view KPIs on the CMDB dashboard
   View the data flow diagram on the dashboard.
   View the reconciliation job details at Jobs > ManageReconciliation
   Run the BMC CMDB maintenance tool	Launch the Maintenance tool
   Run the log zipper	Collecting-diagnostics-in-a-zip-file
   Run arpreparelogs.exe	Running arpreparelogs.exe
   Analyze the logs	Analyzing AR System logs Communities article on running the log analyzer
   Run the data analyzer	Using-CMDB-Data-Analyzer-to-investigate-CMDB-data-issues

Where to go from here

After collecting diagnostics as described in this section, for the next step of troubleshooting based on error messages and optimizing performance, see Troubleshooting-reconciliation-based-on-error-messages.

Related topics

Enabling-reconciliation-job-logs

Analyzing-Reconciliation-job-logs

Improving-performance-of-reconciliation

Accessing-and-navigating-the-CMDB-Portal

Quick-tour-of-the-BMC-CMDB-interface

BMC Communities blog about the new CMDB UI

Configuring access the new CMDB UI in various environments

Running the CMDB maintenance tool

Additional resources