Troubleshooting Reconciliation Engine jobs that do not start or finish

If you face any issues with the reconciliation jobs that you have created or executed, use the information in this topic to troubleshoot the issue. 

To troubleshoot reconciliation process startup issues

The Reconciliation Engine process is started using armonitor.cfg on Microsoft Windows or armonitor.conf on UNIX. If the Reconciliation Engine is not running, perform the following steps:

  1. Open the armonitor.cfg (armonitor.conf) file.
  2. Verify that the configuration file contains an entry for the arrecond process and that and it is not commented out.
  3. Depending on the platform you have installed, perform the following steps:
    • Microsoft Windows — In the armonitor.cfg file, the value specified for the -i parameter must be the same as the Monitor-directory parameter in the armonitor.cfg file.
    • UNIX — In the arrecond.sh script file, make sure that the -i parameter points to /etc/arsystem/<ARInstanceName>/.
      For more information about the armonitor.cfg file, see armonitor.conf or armonitor.cfg in the BMC Remedy AR System online documentation.
  4. If you modified the armonitor.cfg file, restart the AR System service to run the Reconciliation Engine process.

Related topics

Configuring-core-CMDB-components-and-plug-ins

Analyzing-Reconciliation-job-logs

Reconciliation-job-logging

To troubleshoot reconciliation jobs that are not visible

If there are any reconciliation jobs with missing information, the reconciliation page displays the following message at the bottom of the list of reconciliation jobs:
RE corrupt jobs.png

To fix the issues with such jobs, perform the following steps:

  1. Open the list of jobs with errors by clicking the click here link in the message. 
  2. Open the job and click Edit Job.
  3. Enter the information that is missing or is incorrect. 
  4. Click Save and run the job.

If you are unable to resolve the issue, contact BMC Support.

To troubleshoot Reconciliation Engine scheduled jobs that did not finish

The following procedures might help resolve issues with reconciliation jobs that might be running for a long time without completing them.

To verify if the job is running

At times, the Reconciliation Engine might take several minutes to update the status of your job to Successful or Failed. This might cause you to believe that the job did not finish. 

In such a case, perform the following procedure to verify that the job is still running and that the Reconciliation Engine did not generate any errors.

  1. Check the arerror.log log to determine whether any errors occurred with the AR System server or the arrecond.exe process.

    If the AR System server or the arrecond.exe process stops responding, they are restarted and the status of the Reconciliation Engine job is changed to Aborted. However errors, such as ARERR 93 in the arerror.log log, can point to failures caused by performance or AR System connection issues.

  2. Check the Reconciliation Engine Job log for any information about Reconciliation Engine activities.

    Verify that the last line of the Reconciliation Engine Job log indicates that an activity was completed.

  3. Check the Reconciliation Engine Job logs for any plug-in errors.
    For example, an ARERR 8755 The specified plugin does not exist error message might indicate an issue with the BMC Service Impact Manager extensions to BMC Helix CMDB.

    Tip

    The Reconciliation Engine job logs follow the <jobName>_<fileNumber> naming convention, where the job name refers to the name of the job to be executed and the file number indicates the number of files created for the job + 1. If you specified a file size limit, multiple files for a single job run will be created, for example, MergeComputers_1, MergeComputers_2, and so on.

  4. Check the arrecond.log log for errors.

Important

Changes made to the new job definitions are delayed until the next scheduled Job Definition check interval parameter runs. To load the changes immediately, open and save the job from the UI.

To verify the status of the TSOM filter and plug-in

When a reconciliation job has a TSOM filter associated with it such as SIM: and PN:, the reconciliation job form looks for a TSOM plug-in to run the filter and complete the job. If the plug-in is missing, the TSOM filter in the job form checks for the plug-in, fails to connect, and continues to check again. This causes the job to not be completed.

We recommend that you either enable both, the TSOM filter and the plug-in or disable both to run a reconciliation job.


To Troubleshoot Reconciliation Engine jobs that do not start

Perform the following procedure when you notice that Reconciliation Engine jobs are not running even though they were scheduled to run or triggered manually.

Parameter or component

Troubleshooting information

Reconciliation job

Verify that the Job status for the specific job is set to Active.

Reconciliation job

If a previously failed job is run again because of an updated rule, you get the following error:

Failed to save Standard Rules-com.bmc.atrium.lcds.common.AtriumException:Failed to apply Standard Rules.

This error occurs as the job fails with a unique index violation error due to existence of orphan entries from the previous run. This might occur because of an AR System timeout issue, due to conditions such as AR System server termination, a network connectivity issue, the database unavailability, and so on.

To fix this issue, enable AUI logs from the Mid Tier and then based on the error log, delete the orphan entries, and resume activity of a standard job update. For more information, see Knowledge Article number DOC-62155 on BMC Communities (Support ID required).

arrecond.exe

Verify that the arrecond.exe process (Reconciliation Engine) is running and that its process ID (PID) is not changing every minute.
The Reconciliation Engine process ID must remain the same, which denotes that the dispatcher is not abruptly restarting.

armonitor.log file

Check whether there is an entry indicating that the arrecond.exe process is restarting every few seconds.

If the armonitor.log file shows frequent entries indicating that thearrecond.exe process is restarting every few seconds, then there is an issue with the reconciliation engine process. Check the arrecond.log file for details.

Reconciliation Engine

Verify that the Reconciliation Engine is connected to the correct  AR System server instance.
For information about how to verify that the Reconciliation Engine is connected to the correct AR System server instance see Reconciliation-job-logging.

Tip

To verify that a specific process is running on a Microsoft Windows computer, view the Processes tab on the Task Manager. On UNIX, use the ps - ef | grep <processName> command.

arerror.log, and armonitor.cfg (armonitor.conf ) files

If the Reconciliation Engine is not running or if it is not connecting to the AR System server, search the arerror.log, and armonitor.cfg (armonitor.conf ) files for errors.

Application Pending form

Wait for a few seconds for the record in the Application Pending form to be processed and the job to start.
The job usually starts within a few seconds after it is initiated. A CCS updater utility instructs the reconciliation engine to perform tasks based on the entry in the Application Pending form.

Tip

Search for the Application Pending form and open it in Search mode to view the job record.

In the case of server group configuration, confirm if the reconciliation process is running on the server where it is ranked in the application ranking form.

Dispatcher


  1. Perform the following steps to verify that the dispatcher is running:
    1. To verify that the dispatcher is configured to run, search for the arsvcdsp.exe process in the armonitor.cfg file.
    2. From the Microsoft Windows Task Bar or UNIX console, verify that the dispatcher process is running and that its process ID is not changing every minute.
      The dispatcher process ID must remain the same, which denotes that the dispatcher is not abruptly restarting.

  2. If the dispatcher is not running, wait for the amount of time specified in the Polling Interval field on the Server Configuration Editor dialog box.

    Tip

    You can also reduce the Polling Interval time. This ensures that the arrecond.exe process polls the Reconciliation Engine job queue more often.

arrecond and arsvcdsp processes

If the arrecond and arsvcdsp processes are running, but the Application Pending entry is not processed, perform the following steps to enable dispatcher logging.

  1. In the ar.cfg (ar.conf ) configuration file, set the Dispatch-Log-File entry to <fullPath>/dispatch.log.
  2. Restart the AR System server.

 Execute Job activities

For jobs that are started by way of workflow or Execute Job activities, verify that the job name is correct and that the status of the specified job is Active on the Job History Console.

arerror.log

Examine arerror.log to verify that the server was down for maintenance during a scheduled run of the Reconciliation Engine Job.



 

Tip: For faster searching, add an asterisk to the end of your partial query. Example: cert*