This documentation supports the 19.02 version of BMC Atrium Core.

To view the latest version, select the version from the Product version menu.

Troubleshooting CMDB reconciliation issues by collecting diagnostics


BMC CMDB reconciliation performance issues are evident when the reconciliation job is very slow in processing  the configuration items (CIs) and in completing the job.

Collecting issue symptoms

You must check whether any of the following are true:

  • More than one instance of the same reconciliation job is running at the same time.
    You can view this on the Reconciliation page in Jobs > Manage Reconciliation.
  • The ARRECOND process stops and restarts automatically.
    In Windows, you can view the ARRECOND process in the Windows taskbar.
    In Linux, you can view this process by running the following command:
    grep -i "arrecond".
  • Changes were made to any Remedy AR server, CMDB, or the database and the issue of slow reconciliation job started after that change.
    • If any configuration changes pertaining to the running of the Reconciliation engine were made in the Remedy AR server.
    • You see any sudden increase in the count of CIs in CMDB classes, thus causing the slowness in running a Reconciliation job.
    • If any changes were made to the database side, such as addition or removal of any index and a new patch installation.
  • CPU and memory consumption on the AR server and ARRECOND process are unusually high when running the reconciliation job.
    In Windows, you can view the ARRECOND process in the Windows taskbar.
    In Linux, you can view the ARRECOND process by running the following command:
    grep -i "arrecond"
    Verify if consumption of CPU and memory usage on the AR server running ARRECOND process is on a higher side when running the reconciliation job.
  • The reconciliation job log indicates that there are duplicate CIs created in large numbers, which can delay the completion of the reconciliation job.
    You can find the reconciliation job log at:
    <ARSystemServerInstallDir>\BMCData\ARSystem\ARServer\Db\

Environment details

You must note the following environment details:

  • The operating system of the AR server running CMDB reconciliation.
  • The version of AR and CMDB. Whether any patch or hotfix was applied.
  • Whether it is an AR server group. If yes, whether CMDB is configured to be failover or a standalone for the reconciliation process.
  • If there is any private RPC queue for Reconciliation Engine and if min/max threads are assigned to that queue.
  • If private queue is not set for Reconciliation Engine then what is the min/max values for the fast and list queue (390620, 390635).

Issue scope

You must note the following aspects to know the extent of the issue:

    • The activity in reconciliation job which is affected by the slow performance.
    • The number of CIs processed and yet to be processed by the reconciliation job.
    • The number of records present in the RE:Job_Runs and RE:Job_Events AR forms and whether there are any excess records in these forms.
      Use the following URLs to access the forms in AR System:
      http://<midTierServer>:<port>/arsys/forms/<ARServer>/RE:Job_Runs
      http://<midTierServer>:<port>/arsys/forms/<ARServer>/RE:Job_Events

To troubleshoot by gathering logs and configuration files

After you identify the symptoms and scope of the issue, perform the following steps to diagnose and resolve the issue or to create a BMC Support case. 

  1. Save the following log and configuration files:

    Log or configuration fileFile locations
    Reconciliation job logThe default locations for the reconciliation job log is <ARSystemServerInstallDir>\BMCData\ARSystem\ARServer\Db\re
    ARserver SQL+API+Filter (in single file) config filesWindows: <ARSystemServerInstallDir>\Conf\ar.cfg Unix: <ARSystemServerInstallDir>/conf/ar.conf

    ARRECOND.log

    Windows: <ARSystemServerInstallDir>\BMC Software\ARSystem\Arserver\Db
    Unix: ARSystemServerInstallDir>/bmc/arsystem/arserver/db
    ARERROR.logWindows: <ARSystemServerInstallDir>\BMC Software\ARSystem\Arserver\Db
    Unix: ARSystemServerInstallDir>/bmc/arsystem/arserver/db
    CMDB_ENG_DEBUG_LOG.logWindows: <ARSystemServerInstallDir>BMC Software\ARSystem\Arserver\Db
    Unix:<ARSystemServerInstallDir>/bmc/arsystem/arserver/db
  2. Take a snapshot of the following information:

    InformationWhere to find it

    Open the AR System Server Group Operation Ranking form through MidTier and search the records by selecting Reconciliation engine in the Operation list.

    Take a snapshot of each AR server record from the results list or just export the records to a .csv file by clicking the Report button under the results list.

    http://<midTierServer>:<port>/arsys/forms/<aRServer>/AR+System+Server+Group+Operation+Ranking


    AR server system details such as RAM, CPU, and so on.System information.
  3. Perform the following tasks to view the job history, overall health, and integration check information.

    TaskWhere to look or what do to

    View the health metrics on the dashboard

    View the data flow diagram on the dashboard.

    View the reconciliation job details at Jobs > Manage Reconciliation

    Run the BMC Atrium Core maintenance tool

    Running the Atrium Core maintenance tool

    Run the log zipper

    Collecting diagnostics using the log zipper

    Run arpreparelogs.exe

      MultiExcerpt named 'Running_arpreparelogs' was not found
    The page: _ac_NW_LinksLibrary was found, but the multiexcerpt named 'Running_arpreparelogs' was not found. Please check/update the page name used in the 'multiexcerpt-include macro.

    Analyze the logs

    Analyzing AR System logs

    Communities article on running the log analyzer

    Run the data analyzer

    Running the data analyzer

Where to go from here

After collecting diagnostics as described in this section, for the next step of troubleshooting based on error messages and optimizing performance, see Troubleshooting reconciliation based on error messages.

Related topics

Improving reconciliation performance by increasing threads

Accessing and navigating the new CMDB user interface

Quick tour of the BMC CMDB interface

BMC Communities blog about the new CMDB UI

Configuring access the new CMDB UI in various environments

Running the CMDB maintenance tool

Additional resources

TopicReference
Atrium Core CMDB Reconciliation not doing what you've configured it for? Here is whyKA 000029918
Atrium CMDB – Setup, Configure and Performance fine-tuninghttps://communities.bmc.com/docs/DOC-26368
CMDB:Troubleshooting Reconciliation Job Activities Webinar

Communities page with webinar on reconciliation troubleshooting

Best Practices for handling Reconciliation

Reconciliation best practices

Remedy 9.x/18xx Release – Configuration Checklisthttps://bmcsites.force.com/casemgmt/sc_KnowledgeArticle?sfdcid=kA014000000h9kqCAA


Was this page helpful? Yes No Submitting... Thank you

Comments