Troubleshooting CMDB reconciliation issues by collecting diagnostics
BMC CMDB reconciliation performance issues are evident when the reconciliation job is very slow in processing the configuration items (CIs) and in completing the job.
Collecting issue symptoms
You must check whether any of the following are true:
- More than one instance of the same reconciliation job is running at the same time.
You can view this on the Reconciliation page in Jobs > Manage Reconciliation. - The ARRECOND process stops and restarts automatically.
In Windows, you can view the ARRECOND process in the Windows taskbar.
In Linux, you can view this process by running the following command:
grep -i "arrecond". - Changes were made to any Remedy Action Request System server, CMDB, or the database and the issue of slow reconciliation job started after that change.
- If any configuration changes pertaining to the running of the Reconciliation engine were made in the Remedy Action Request System server.
- You see any sudden increase in the count of CIs in CMDB classes, thus causing the slowness in running a Reconciliation job.
- If any changes were made to the database side, such as addition or removal of any index and a new patch installation.
- CPU and memory consumption on the Remedy AR System server and ARRECOND process are unusually high when running the reconciliation job.
In Windows, you can view the ARRECOND process in the Windows taskbar.
In Linux, you can view the ARRECOND process by running the following command:
grep -i "arrecond"
Verify if consumption of CPU and memory usage on the Remedy AR System server running ARRECOND process is on a higher side when running the reconciliation job. - The reconciliation job log indicates that there are duplicate CIs created in large numbers, which can delay the completion of the reconciliation job.
You can find the reconciliation job log at:
<ARSystemServerInstallDir>\BMCData\ARSystem\ARServer\Db\
Environment details
You must note the following environment details:
- The operating system of the Remedy AR System server running CMDB reconciliation.
- The version of Remedy AR System and CMDB. Whether any patch or hotfix was applied.
- Whether it is an Remedy AR System server group. If yes, whether CMDB is configured to be failover or a standalone for the reconciliation process.
- If there is any private RPC queue for Reconciliation Engine and if min/max threads are assigned to that queue.
- If private queue is not set for Reconciliation Engine then what is the min/max values for the fast and list queue (390620, 390635).
Issue scope
You must note the following aspects to know the extent of the issue:
- The activity in reconciliation job which is affected by the slow performance.
- The number of CIs processed and yet to be processed by the reconciliation job.
- The number of records present in the RE:Job_Runs and RE:Job_Events AR forms and whether there are any excess records in these forms.
Use the following URLs to access the forms in Remedy AR System:
http://<midTierServer>:<port>/arsys/forms/<ARServer>/RE:Job_Runs
http://<midTierServer>:<port>/arsys/forms/<ARServer>/RE:Job_Events
To troubleshoot by gathering logs and configuration files
After you identify the symptoms and scope of the issue, perform the following steps to diagnose and resolve the issue or to create a BMC Support case.
Save the following log and configuration files:
Log or configuration file | File locations | Reference |
---|---|---|
Reconciliation job log | The default locations for the reconciliation job log is <ARSystemServerInstallDir>\BMCData\ARSystem\ARServer\Db\re | |
SQL+API+Filter logs (in single file) | Windows: <ARSystemServerInstallDir>\BMC Software\ARSystem\Arserver\Db | |
ARserver config files | Windows: <ARSystemServerInstallDir>\Conf\ar.cfg Unix: <ARSystemServerInstallDir>/conf/ar.conf | |
ARRECOND.log | Windows: <ARSystemServerInstallDir>\BMC Software\ARSystem\Arserver\Db | |
ARERROR.log | Windows: <ARSystemServerInstallDir>\BMC Software\ARSystem\Arserver\Db | ARERROR log files are generated by default. |
CMDB_ENG_DEBUG_LOG.log | Windows: <ARSystemServerInstallDir>BMC Software\ARSystem\Arserver\Db | For CMDB Engine, change the log levels from Centralized Configuration Settings from the AR System Home page. Perform the following steps:
|
Take a snapshot of the following information:
Information
Where to find it
Open the Remedy Action Request System server Group Operation Ranking form through Mid Tier and search the records by selecting Reconciliation engine in the Operation list.
Take a snapshot of each Remedy AR System server record from the results list or just export the records to a .csv file by clicking the Report button under the results list.
http://<midTierServer>:<port>/arsys/forms/<aRServer>/AR+System+Server+Group+Operation+Ranking
Remedy AR System server system details such as RAM, CPU, and so on.
System information.
Perform the following tasks to view the job history, overall health, and integration check information.
Task
Where to look or what do to
View the health metrics on the dashboard
View the data flow diagram on the dashboard.
View the reconciliation job details at Jobs > ManageReconciliation
Run the BMC CMDB maintenance tool
Run the log zipper
Run arpreparelogs.exe
Analyze the logs
Run the data analyzer
Where to go from here
After collecting diagnostics as described in this section, for the next step of troubleshooting based on error messages and optimizing performance, see Troubleshooting-reconciliation-based-on-error-messages.
Related topics
Enabling-reconciliation-job-logs
Analyzing-Reconciliation-job-logs
Improving-performance-of-reconciliation
Accessing-and-navigating-the-CMDB-Portal
Quick-tour-of-the-BMC-CMDB-interface
BMC Communities blog about the new CMDB UI
Configuring access the new CMDB UI in various environments
Running the CMDB maintenance tool
Additional resources
Topic | Reference |
---|---|
Atrium Core CMDB Reconciliation not doing what you've configured it for? Here is why | Knowledge Article number 000201184 |
Atrium CMDB – Setup, Configure and Performance fine-tuning | |
CMDB:Troubleshooting Reconciliation Job Activities Webinar | |
Best Practices for handling Reconciliation | |
Remedy 9.x/18xx Release – Configuration Checklist |