This documentation supports the 20.08 version of BMC CMDB, which is available only to BMC Helix subscribers (SaaS).

To view an earlier version, select the version from the Product version menu.

Troubleshooting CMDB reconciliation issues by collecting diagnostics


BMC CMDB reconciliation performance issues are evident when the reconciliation job is very slow in processing  the configuration items (CIs) and in completing the job.

Collecting issue symptoms

You must check whether any of the following are true:

  • More than one instance of the same reconciliation job is running at the same time.
    You can view this on the Reconciliation page in Jobs > Manage Reconciliation.
  • The ARRECOND process stops and restarts automatically.
    In Windows, you can view the ARRECOND process in the Windows taskbar.
    In Linux, you can view this process by running the following command:
    grep -i "arrecond".
  • Changes were made to any Remedy Action Request System server, CMDB, or the database and the issue of slow reconciliation job started after that change.
    • If any configuration changes pertaining to the running of the Reconciliation engine were made in the Remedy Action Request System server.
    • You see any sudden increase in the count of CIs in CMDB classes, thus causing the slowness in running a Reconciliation job.
    • If any changes were made to the database side, such as addition or removal of any index and a new patch installation.
  • CPU and memory consumption on the Remedy AR System server and ARRECOND process are unusually high when running the reconciliation job.
    In Windows, you can view the ARRECOND process in the Windows taskbar.
    In Linux, you can view the ARRECOND process by running the following command:
    grep -i "arrecond"
    Verify if consumption of CPU and memory usage on the Remedy AR System server running ARRECOND process is on a higher side when running the reconciliation job.
  • The reconciliation job log indicates that there are duplicate CIs created in large numbers, which can delay the completion of the reconciliation job.
    You can find the reconciliation job log at:
    <ARSystemServerInstallDir>\BMCData\ARSystem\ARServer\Db\

Environment details

You must note the following environment details:

  • The operating system of the Remedy AR System server running CMDB reconciliation.
  • The version of Remedy AR System and CMDB. Whether any patch or hotfix was applied.
  • Whether it is an Remedy AR System server group. If yes, whether CMDB is configured to be failover or a standalone for the reconciliation process.
  • If there is any private RPC queue for Reconciliation Engine and if min/max threads are assigned to that queue.
  • If private queue is not set for Reconciliation Engine then what is the min/max values for the fast and list queue (390620, 390635).

Issue scope

You must note the following aspects to know the extent of the issue:

    • The activity in reconciliation job which is affected by the slow performance.
    • The number of CIs processed and yet to be processed by the reconciliation job.
    • The number of records present in the RE:Job_Runs and RE:Job_Events AR forms and whether there are any excess records in these forms.
      Use the following URLs to access the forms in Remedy AR System:
      http://<midTierServer>:<port>/arsys/forms/<ARServer>/RE:Job_Runs
      http://<midTierServer>:<port>/arsys/forms/<ARServer>/RE:Job_Events

To troubleshoot by gathering logs and configuration files

After you identify the symptoms and scope of the issue, perform the following steps to diagnose and resolve the issue or to create a BMC Support case. 

Save the following log and configuration files:

Log or configuration fileFile locationsReference
Reconciliation job logThe default locations for the reconciliation job log is <ARSystemServerInstallDir>\BMCData\ARSystem\ARServer\Db\reChanging log settings for core CMDB components
SQL+API+Filter logs (in single file)Windows: <ARSystemServerInstallDir>\BMC Software\ARSystem\Arserver\Db
Unix: ARSystemServerInstallDir>/bmc/arsystem/arserver/db

Enabling logs for performance issues Open link

ARserver config filesWindows: <ARSystemServerInstallDir>\Conf\ar.cfg Unix: <ARSystemServerInstallDir>/conf/ar.conf

ARRECOND.log

Windows: <ARSystemServerInstallDir>\BMC Software\ARSystem\Arserver\Db
Unix: ARSystemServerInstallDir>/bmc/arsystem/arserver/db
Changing log settings for core CMDB components
ARERROR.logWindows: <ARSystemServerInstallDir>\BMC Software\ARSystem\Arserver\Db
Unix: ARSystemServerInstallDir>/bmc/arsystem/arserver/db
ARERROR log files are generated by default.
CMDB_ENG_DEBUG_LOG.logWindows: <ARSystemServerInstallDir>BMC Software\ARSystem\Arserver\Db
Unix:<ARSystemServerInstallDir>/bmc/arsystem/arserver/db

For CMDB Engine, change the log levels from Centralized Configuration Settings from the AR System Home page. Perform the following steps:

  1. Log in to AR System Home page.
  2. Navigate to Applications > AR System Administration > Ar System Administration Console> System> General > Centralized Configuration.
  3. In Component Name, select com.bmc.cmdb.server and select the server name.
  4. To enable debug logging, in CMDB-Logging-Level, double-click the corresponding Setting Value field, and enter DEBUG or TRACE.
  1. Take a snapshot of the following information:

    InformationWhere to find it

    Open the Remedy Action Request System server Group Operation Ranking form through Mid Tier and search the records by selecting Reconciliation engine in the Operation list.

    Take a snapshot of each Remedy AR System server record from the results list or just export the records to a .csv file by clicking the Report button under the results list.

    http://<midTierServer>:<port>/arsys/forms/<aRServer>/AR+System+Server+Group+Operation+Ranking


    Remedy AR System server system details such as RAM, CPU, and so on.

    System information.
  2. Perform the following tasks to view the job history, overall health, and integration check information.

    TaskWhere to look or what do to

    View the health metrics on the dashboard

    View the data flow diagram on the dashboard.

    View the reconciliation job details at Jobs > ManageReconciliation

    Run the BMC CMDB maintenance tool

    Run the log zipper
    Run arpreparelogs.exe

    Running arpreparelogs.exe Open link

    Analyze the logs

    Analyzing AR System logs Open link

    Communities article on running the log analyzer Open link

    Run the data analyzer

Where to go from here

After collecting diagnostics as described in this section, for the next step of troubleshooting based on error messages and optimizing performance, see Troubleshooting reconciliation based on error messages.

Related topics

Enabling reconciliation job logs

Analyzing Reconciliation job logs

Improving performance of reconciliation

Accessing and navigating the CMDB Portal

Quick tour of the BMC CMDB interface

BMC Communities blog about the new CMDB UI

Configuring access the new CMDB UI in various environments

Running the CMDB maintenance tool

Additional resources

TopicReference
Atrium Core CMDB Reconciliation not doing what you've configured it for? Here is whyKnowledge Article number 000201184
Atrium CMDB – Setup, Configure and Performance fine-tuninghttps://community.bmc.com/s/news/aA33n000000ChptCAC/atrium-cmdb-8x-setup-configuration-and-performance-tuning-12-september-20
CMDB:Troubleshooting Reconciliation Job Activities Webinar

Communities page with webinar on reconciliation troubleshooting Open link

Best Practices for handling Reconciliation

Reconciliation best practices Open link

Remedy 9.x/18xx Release – Configuration Checklisthttps://bmcsites.force.com/casemgmt/sc_KnowledgeArticle?sfdcid=kA014000000h9kqCAA


Was this page helpful? Yes No Submitting... Thank you

Comments