Troubleshooting reconciliation based on error messages
If you are experiencing slow performance with reconciliation jobs, and if you see a high frequency of CI data related error messages, you must fix those issues first. The following are a few of such error messages and their solutions.
Before you begin
To troubleshoot CI data related error messages, you must be a CMDB administrator and you may also have to work with your database administrator.
To troubleshoot error messages related to identification of CIs
One of the major reasons of slow performance during identification of CIs is due to a large number of duplicate configuration items (CIs) identified.
The following is a sample error message of duplicate CI message found.
Example error message
Found multiple matches (instances) for class <BMC.CORE:BMC_ApplicationService> in look-up dataset <BMC Asset> for qualification <'Name' = $Name$> of group <BMC.ADDM.Group> with values < Name = Apache HTTP Web server >
Error message analysis
From the error message we can know that the class form is BMC.CORE:BMC_ApplicationService. The qualification used is <'Name' = $Name$>
and value of the Name attribute is <Name = Apache HTTP Web server>
.
Solution
The solution to this error involves identifying the duplicate CIs that use the attributes used in identification rules. For the example error message, you can perform the following steps to troubleshoot this issue:
- Open the BMC.CORE:BMC_ApplicationService form in a browser.
http://<midTierServer>:<port>/arsys/forms/<ARServer>/BMC.CORE:BMC_ApplicationService/
The qualification used is<'Name' = $Name$>
and value of the Name attribute is<Name = Apache HTTP Web server>
- Search the BMC.CORE:BMC_ApplicationService form for the
Name
attribute with value equal toApache HTTP Web server
.
The search may probably return more than one record. - Identify the attributes other than those existing in the identification rules to differentiate CIs and then add those attributes to the identification rule.
You may also use the Data Analyzer tool to find the duplicate records. - If there are genuine duplicate CIs which need to be removed, you must perform the following steps:
- Take a backup of the Remedy database or the class form data before purging the duplicate CIs.
Once the duplicate data is identified with the help of the data or asset owner, the CMDB administrator can mark these CIs as deleted and then use the reconciliation engine job to purge them. - From the multiple matching records, find the ones you want to retain.
This can be based on, for example, theCreateDate
,ModfiedDate
,Submitter
or any other attribute value that differentiates duplicate data from the original one. - Update the
MarkAsDeleted
attribute of these duplicate CIs toyes
.
You can perform this step manually, using Atrium Integrator, or a database query. - Use the reconciliation purge activity to remove the CIs that are marked as deleted from the dataset .
Before this step, you must verify the CIs which are to be purged.
- Take a backup of the Remedy database or the class form data before purging the duplicate CIs.
Example error message
DatasetID: BMC.ADDM ReconciliationId : OI-7D96B4223FEB432793BBDF25B2BC224F
Associating reconciliation identity to <OI-A31AA20DD2424EC08092DF0CA5011E06> for instance id <OIGAA5V0A4BP5APK98VEPK98VE7F73> for class <BMC.CORE:BMC_ComputerSystem> in dataset <BMC.ADDM>
ARERR[120092] The dataset ID and Reconciliation Identity combination is not unique.
Error message analysis
This error "ARERR [120092] The dataset ID and Reconciliation Identity combination is not unique"
means that the same reconciliation identity cannot be assigned to another CI in the same dataset. This is expected behavior. To resolve the issue, you must find out why the reconciliation job was trying to assign the same reconciliation ID to another CI in the same dataset.
The class mentioned in the above error message is BMC.CORE:BMC_ComputerSystem
. There are out of the box and custom identification rules for each class.
The following example shows the BMC_ComputerSystem
class identification rules.
The priorities shown in the example show the sequence in which the rules check for CI duplication in the target dataset (BMC.ASSET). The priority proceeds to the next available rule until it reaches the final one.
Let’s assume that the following CI records are for BMC.CORE:BMC_ComputerSystem
.
Name | Dataset | Reconciliation ID | Serial Number |
---|---|---|---|
Dell 2500 series laptop | BMC.ADDM |
| 686868 |
Dell 2500 series laptop | BMC.ASSET | OI-E7315F17CAC04E8AB89C2D5E8E924595 | 686868 |
HCL 200 Series Laptop | BMC.ADDM | 0 | 686868 |
If we compare the above record table with the Identification rule for BMC_ComputerSystem
class, the serial number is common for Dell and HCL laptops. Let us further assume that the fields for the Virtual
and PartiionID
attributes are NULL for Dell and HCL Laptop CI records. In this case, only the attribute Serial Number
carries a value and this attribute shares the same number between these two CI records.
When the reconciliation job runs, the CI passes the first and second priority rule. After that, the job identifies the CI on the third rule based on the value of Serial Number
considering that the isVirtual
and PartiionID
had NULL values. The job tries to assign the reconciliation ID OI-E7315F17CAC04E8AB89C2D5E8E924595
to the CI representing the HCL 200 Series Laptop in BMC.ADDM. This results in the error being logged.
Solution
Considering the above example, we must populate the correct values for the attributes such as isVirtual
and PartitionID
.
In other scenarios, you must identify the identification rule that may be causing the duplicate CI data in the source dataset or fix the source application configuration which is sending the CI data to the source dataset (BMC.ADDM in this example).
Reconciliation performance issues may also be due to merge related activities.
To troubleshoot the reconciliation activity related to merge activities
If your reconciliation job performance is slow due to a merge activity, you may want to verify and set the merge order to By class in separate transactions for faster processing using the following steps.
- Set the merge order.
- Open the reconciliation job.
- In the Merge activity, make sure that the Merge Order is set to By Class in separate transactions.
This setting helps improve the reconciliation performance.
- Check the job log and troubleshoot merge related issues based on specific errors.
Example of merge related issues or errors
If you see a lot of merge related error messages in the RE job log, you must fix them as those UPDATE
SQL queries will fail while consuming time and resources. The following is one such error message which must be fixed. You may see other merge related error messages which can also be addressed in a similar way.
Example error message
ARERR[120141] Cannot set MarkAsDeleted to "No" on the relationship instance because one or both of the relationship endpoints are MarkAsDeleted. Rel(clsId:BMC.CORE:BMC_HOSTEDACCESSPOINT instId: OI-3195059586FB4BACB5D582D5CE9EC490 endpoint inst is soft deleted -- InstanceId: OI-7B6226B15ECF47BC88339A4563BD1C77, ClassId: BMC_IPENDPOINT
Merging of record failed and rolling back the transaction Class: BMC.CORE:BMC_HOSTEDACCESSPOINT Reconciliation IdentityId: OI-AC802CB8B2304BCAA532E3A6BEC44388
Error message analysis
From the example error message, you can know that the Relationship Class record in BMC.CORE:BMC_HOSTEDACCESSPOINT could not be merged for the attribute MarkAsDeleted
to No
. The reason for the error is that you were attempting to set MarkAsDeleted
to No
for a relationship instance for which one or both endpoints are already soft deleted, that is, set to MarkAsDeleted
.
Example:
Relationship Endpoint Class records for BMC_IPEndpoint
and BMC_ComputerSystem
.
Name | Class Name | Reconciliation ID | MarkAsDeleted |
---|---|---|---|
IPv4_11 | BMC_IPEndPoint | OI123123123123 | YES |
Dell_15 | BMC_ComputerSystem |
| YES |
Relationship CI record for BMC_HOSTEDACCESSPOINT
joining BMC_IPEndpoint
and BMC_ComputerSystem
.
Name | Source reconciliation ID | Destination reconciliation ID | MarkAsDeleted |
---|---|---|---|
Dell_IpEndPoint | OI789789789789 | OI123123123123 | YES |
If you try to set the attribute MarkAsDeleted
to No for the previously mentioned relationship CI record, it will result in the following error message.
ARERR[120141] Cannot set MarkAsDeleted to "No" on the relationship instance because one or both of the relationship endpoints are MarkAsDeleted.
This error occurs because you cannot set the value of MarkAsDeleted
to No for the Relationship Class
record in case the endpoints of relationship record MarkAsDeleted
are set to YES.
Solution
Set MarkAsDeleted
to No on both endpoints of the relationship before setting MarkAsDeleted
to No on the relationship instance.
To resolve missing endpoints during the Merge activity
The Reconciliation Engine fails to merge CIs during a Merge activity because of missing endpoint for a relationship. The error message displayed in the Reconciliation Engine log includes Cannot find the endpoint of relationship::<className>
. This error occurs if you manually modify a CI and fail to modify its relationship, causing data integrity issues.
- Reset reconciliation ID to 0 for all instances in the specific dataset.
- Start the Identification and Merge activity.
To resolve CIs merged multiple times
During a Merge activity, the same CI seems to be merged more than once. The error message displayed on the Job History Console includes Algorithm: Including child CIs and committing together
. This error likely occurred because you selected Commit Together or Commit Separately for the Include Child CIs? option of the Merge activity but did not use a Qualification group to restrict the CIs to be merged. In this case, a child CI in a composite object is merged once on its own and once as part of the composite object.
- Select the Stand Alone option for the Include Child CIs? option of the Merge activity.
- Alternatively, attach a Qualification group to the activity to exclude CIs that will also be merged as part of a composite object.
To troubleshoot the reconciliation Purge activity
Performance issues rarely happen during the purge activity. If you see any issues, turn on the AR API SQL, and Filter logs and run the Log Analyzer tool to determine the expensive API SQL calls for appropriate actions.
To troubleshoot reconciliation jobs that are not visible
A reconciliation job that has been created in the Atrium Core console is not visible in CMDB Portal. This issue might occur if RE job definition is corrupted. For example, RE job definition might be corrupted if the association of the dataset with the job is broken.
If the job definition is corrupted, the reconciliation job:
- is not visible in the list of reconciliation jobs in the CMDB Portal.
- is visible in Atrium Core console but cannot be run.
Solution
- From the Dashboard, go to Manage reconciliation.
- To view corrupted jobs, click <placeholder>
- Select the job that you want to fix and click edit
- Enter the correct values for the job definition and save the job.
You must edit the job by reselecting the dataset in the RE Activity. If this does not fix the issue, you must create a new dataset and associate it in a new RE job.
- Fix the issues that are causing the dataset to be corrupted or recreate the dataset.
- Delete the existing reconciliation job with the corrupted dataset.
- Create a new reconciliation job and use the error-free dataset.
Where to go from here
Improving performance of reconciliation
Comments
Log in or register to comment.