Troubleshooting reconciliation based on error messages

If you are experiencing slow performance with reconciliation jobs, and if you see a high frequency of CI data related error messages, you must fix those issues first. The following are a few of such error messages and their solutions.

Before you begin

To troubleshoot CI data related error messages, you must be a CMDB administrator and you may also have to work with your database administrator.

To troubleshoot error messages related to identification of CIs

One of the major reasons of slow performance during identification of CIs is due to a large number of duplicate configuration items (CIs) identified. 

The following is a sample error message of duplicate CI message found.

Example error message

Found multiple matches (instances) for class <BMC.CORE:BMC_ApplicationService> in look-up dataset <BMC Asset> for qualification <'Name' = $Name$> of group <BMC.ADDM.Group> with values < Name = Apache HTTP Web server >

Error message analysis

From the error message we can know that the class form is BMC.CORE:BMC_ApplicationService. The qualification used is <'Name' = $Name$> and value of the Name attribute is <Name = Apache HTTP Web server>.

Solution

The solution to this error involves identifying the duplicate CIs that use the attributes used in identification rules. For the example error message, you can perform the following steps to troubleshoot this issue:

1. Open the BMC.CORE:BMC_ApplicationService form in a browser.
   http://<midTierServer>:<port>/arsys/forms/<ARServer>/BMC.CORE:BMC_ApplicationService/
   The qualification used is <'Name' = $Name$> and value of the Name attribute is <Name = Apache HTTP Web server>
2. Search the BMC.CORE:BMC_ApplicationService form for the Name attribute with value equal to Apache HTTP Web server.
   The search may probably return more than one record.
3. Identify the attributes other than those existing in the identification rules to differentiate CIs and then add those attributes to the identification rule.
   You may also use the Data Analyzer tool to find the duplicate records. 
4. If there are genuine duplicate CIs which need to be removed, you must perform the following steps:
   1. Take a backup of the Remedy database or the class form data before purging the duplicate CIs.
      Once the duplicate data is identified with the help of the data or asset owner, the CMDB administrator can mark these CIs as deleted and then use the reconciliation engine job to purge them.
   2. From the multiple matching records, find the ones you want to retain.
      This can be based on, for example, the CreateDate, ModfiedDate, Submitter or any other attribute value that differentiates duplicate data from the original one. 
   3. Update the MarkAsDeleted attribute of these duplicate CIs to yes.
      You can perform this step manually, using Atrium Integrator, or a database query. 
   4. Use the reconciliation purge activity to remove the CIs that are marked as deleted from the dataset . 
      Before this step, you must verify the CIs which are to be purged.

Example error message

DatasetID: BMC.ADDM ReconciliationId : OI-7D96B4223FEB432793BBDF25B2BC224F

Associating reconciliation identity to <OI-A31AA20DD2424EC08092DF0CA5011E06> for instance id <OIGAA5V0A4BP5APK98VEPK98VE7F73> for class <BMC.CORE:BMC_ComputerSystem> in dataset <BMC.ADDM>

ARERR[120092] The dataset ID and Reconciliation Identity combination is not unique.

Error message analysis

This error "ARERR [120092] The dataset ID and Reconciliation Identity combination is not unique" means that the same reconciliation identity cannot be assigned to another CI in the same dataset. This is expected behavior. To resolve the issue, you must find out why the reconciliation job was trying to assign the same reconciliation ID to another CI in the same dataset.

The class mentioned in the above error message is BMC.CORE:BMC_ComputerSystem. There are out of the box and custom identification rules for each class. 

The following example shows the BMC_ComputerSystem class identification rules.

The priorities shown in the example show the sequence in which the rules check for CI duplication in the target dataset (BMC.ASSET). The priority proceeds to the next available rule until it reaches the final one. 

Let’s assume that the following CI records are for BMC.CORE:BMC_ComputerSystem.

If we compare the above record table with the Identification rule for BMC_ComputerSystem class, the serial number is common for Dell and HCL laptops. Let us further assume that the fields for the Virtual and PartiionID attributes are NULL for Dell and HCL Laptop CI records. In this case, only the attribute Serial Number carries a value and this attribute shares the same number between these two CI records.

When the reconciliation job runs, the CI passes the first and second priority rule. After that, the job identifies the CI on the third rule based on the value of Serial Number considering that the isVirtual and PartiionID had NULL values. The job tries to assign the reconciliation ID OI-E7315F17CAC04E8AB89C2D5E8E924595 to the CI representing the HCL 200 Series Laptop in BMC.ADDM. This results in the error being logged.

Solution

Considering the above example, we must populate the correct values for the attributes such as isVirtual and PartitionID.

In other scenarios, you must identify the identification rule that may be causing the duplicate CI data in the source dataset or fix the source application configuration which is sending the CI data to the source dataset (BMC.ADDM in this example).

Reconciliation performance issues may also be due to merge related activities.

To troubleshoot the reconciliation activity related to merge activities

If your reconciliation job performance is slow due to a merge activity, you may want to verify and set the merge order to By class in separate transactions for faster processing using the following steps.

1. Set the merge order.
   1. Open the reconciliation job.
   2. In the Merge activity, make sure that the Merge Order is set to By Class in separate transactions.
      This setting helps improve the reconciliation performance.
2. Check the job log and troubleshoot merge related issues based on specific errors.

Example of merge related issues or errors

If you see a lot of merge related error messages in the RE job log, you must fix them as those UPDATE SQL queries will fail while consuming time and resources. The following is one such error message which must be fixed. You may see other merge related error messages which can also be addressed in a similar way.

Example error message

ARERR[120141] Cannot set MarkAsDeleted to "No" on the relationship instance because one or both of the relationship endpoints are MarkAsDeleted. Rel(clsId:BMC.CORE:BMC_HOSTEDACCESSPOINT instId: OI-3195059586FB4BACB5D582D5CE9EC490 endpoint inst is soft deleted -- InstanceId: OI-7B6226B15ECF47BC88339A4563BD1C77, ClassId: BMC_IPENDPOINT

Merging of record failed and rolling back the transaction Class: BMC.CORE:BMC_HOSTEDACCESSPOINT Reconciliation IdentityId: OI-AC802CB8B2304BCAA532E3A6BEC44388

Error message analysis

From the example error message, you can know that the Relationship Class record in BMC.CORE:BMC_HOSTEDACCESSPOINT could not be merged for the attribute MarkAsDeleted to No. The reason for the error is that you were attempting to set MarkAsDeleted to No for a relationship instance for which one or both endpoints are already soft deleted, that is, set to MarkAsDeleted. 

Example:

Relationship Endpoint Class records for BMC_IPEndpoint and BMC_ComputerSystem.

Relationship CI record for BMC_HOSTEDACCESSPOINT joining BMC_IPEndpoint and BMC_ComputerSystem.

If you try to set the attribute MarkAsDeleted to No for the previously mentioned relationship CI record, it will result in the following error message. 

ARERR[120141] Cannot set MarkAsDeleted to "No" on the relationship instance because one or both of the relationship endpoints are MarkAsDeleted.

This error occurs because you cannot set the value of MarkAsDeleted to No for the Relationship Class record in case the endpoints of relationship record MarkAsDeleted are set to YES.

Solution

Set MarkAsDeleted to No on both endpoints of the relationship before setting MarkAsDeleted to No on the relationship instance.

 To resolve missing endpoints during the Merge activity

The Reconciliation Engine fails to merge CIs during a Merge activity because of missing endpoint for a relationship. The error message displayed in the Reconciliation Engine log includes Cannot find the endpoint of relationship::<className>. This error occurs if you manually modify a CI and fail to modify its relationship, causing data integrity issues.

1. Reset reconciliation ID to 0 for all instances in the specific dataset.
2. Start the Identification and Merge activity.

To resolve CIs merged multiple times

During a Merge activity, the same CI seems to be merged more than once. The error message displayed on the Job History Console includes Algorithm: Including child CIs and committing together. This error likely occurred because you selected Commit Together or Commit Separately for the Include Child CIs? option of the Merge activity but did not use a Qualification group to restrict the CIs to be merged. In this case, a child CI in a composite object is merged once on its own and once as part of the composite object.

• Select the Stand Alone option for the Include Child CIs? option of the Merge activity.
• Alternatively, attach a Qualification group to the activity to exclude CIs that will also be merged as part of a composite object.

To troubleshoot the reconciliation Purge activity 

Performance issues rarely happen during the purge activity. If you see any issues, turn on the AR API SQL, and Filter logs and run the Log Analyzer tool to determine the expensive API SQL calls for appropriate actions.

Where to go from here

Improving performance of reconciliation