Planning data reconciliation

This section describes how to plan for the reconciliation of data from source datasets to a production dataset. With multiple data providers loading data into multiple datasets of BMC CMDB, you need a reconciliation process to compare expected data against discovered data and create one complete and correct production dataset. The BMC CMDB Reconciliation Engine performs the main reconciliation tasks of identifying, merging, and comparing datasets and also gives you other tools for working with datasets.

Instance identification across datasets

Before you can compare or merge different versions of an entity, you must determine that they indeed represent the same entity. You must identify each instance.

The Identify activity accomplishes this matching by applying rules that you specify against instances of the same class in different datasets. For example, a rule to identify computer system instances might specify that the IP addresses of the instances be equal.

When the rules find a match, both instances are tagged with the same reconciliation identity, an extra attribute that shows that they each represent the same item in their respective datasets.

You can also manually identify instances that were not identified by rules in an Identify activity.

To identify instances, you must create an Identify group for each participating dataset. In this group you must create Identify rules that attempt to match instances of a particular class in that dataset against instances in all other participating datasets. For example, to compare datasets A, B, and C you need the following groups: one each to match A against B and C, B against A and C, and C against A and B.

To identify data in different classes based on different criteria, you must create more Identify groups. Because the subclasses of the specified class inherit the groups, if your data is sufficiently normalized, you could specify groups only for the base class BMC_BaseElement.

You then create an Identify activity and associate the Identify groups to it. Designate one of the participating datasets as the master dataset, meaning that the reconciliation identity of its instances is applied to matching instances in the other datasets, which are known as auto-identify datasets. If the instance in the master dataset does not have an identity, one is automatically generated.

For more information about identifying, see Configuring reconciliation custom identification rules.

Dataset comparison with reconciliation

The Compare activity operates against instances in two datasets and produces a report based on the Compare results. The report shows those instances that appear in only one of the datasets and details the differences between instances that appear in both.

Compare lets you compare an expected configuration against an actual one, which you could use for more than one purpose. You might use Compare to alert you that something has changed in a configuration that you expected to remain static. Alternatively, if you have a change request in progress, you might use Compare to verify that the configuration reaches its expected new state.

Only instances that have been given a reconciliation identity can be compared, and they are compared only against other instances with the same identity. 

To compare instances, you must create a Compare activity and specify the two datasets that you want to compare. From that activity you can choose to exclude particular attributes from the comparison by creating Exclusion rules for them.

For more information about comparing, see Identifying changes to CIs by using the Compare activity.

Dataset merge with reconciliation

Merging takes data from multiple source datasets and creates a composite by copying that data to a single target dataset according to precedence values that you specify.

Merging is essential to produce a single, valid configuration when different discovery applications provide overlapping data about the same items, or when you need to commit changes that were made in an overlay dataset. Only instances that have been given an identity can participate in a Merge. To take advantage of the areas of strength in each dataset, you create precedence values that favor those strengths. Merging the highest-precedence attribute values gives you one configuration item (CI) instance with the best of all discovered data.

An overall precedence value is given to each dataset, with the ability to override it for particular classes and attributes in each dataset. Whichever dataset has the highest precedence value for a given attribute has its value for that attribute placed in the target dataset. A precedence value specified for a class also applies to its subclasses unless they override it with precedence values of their own.

You can merge data from multiple source datasets either by creating one Merge activity that includes all the source datasets or by creating independent Merge activities that each merge only the data from one source dataset.

Dataset merge with reconciliation using a single merge activity

When you use one Merge activity, the precedence values of all source datasets are compared to each other at once, and the data from the dataset with the highest precedence value is written to the target dataset. The following figure provides an example of precedence values being applied when two datasets are merged with a single Merge activity.

 Single Merge activity with two source datasets 

In this example, source Datasets A and B are merged into target Dataset C. Though Dataset A has a higher precedence value (500) than Dataset B (300), Dataset A has class and attribute precedence values for Application System and the IPAddress attribute of Computer System (both 200) that are lower than Dataset B. Dataset C has a precedence value (100) lower than either source, and as a result, none of the data it contained in step 1 survives the merge.

In the Merge activity represented by step 2, Dataset C receives the Monitor and the SystemType attribute of the Computer System from Dataset A, with a precedence value that trumps Dataset B's. But because the Application System and the IPAddress attribute of the Computer System have lower precedence values in Dataset A, Dataset C receives these from Dataset B.

Dataset merge with reconciliation using independent merge activities

Using independent Merge activities is the recommended method of merging datasets, using only one source dataset per Merge activity. After each Merge activity runs, the target dataset retains – for each attribute that was merged — the precedence value of the dataset that supplied the data for that attribute. When you use independent Merge activities, each activity compares the precedence values of its source dataset to the precedence values of those last victorious datasets.

Because the source dataset in any merge is always compared against the highest precedence value from previous merges, it is as though precedence values from all source datasets are compared in each merge. This frees you from having to design a Merge activity for every combination of source datasets that might be merged together, and enables you to add new source datasets in the future without reworking all your Merge activities. The following figure provides an example of precedence values being applied when two datasets are merged with independent Merge activities.

 Independent Merge activities, each with one source dataset

This example uses the same source and target datasets as the example in Planning data reconciliation, and achieves the same end result. Step 1 again shows the data in the target dataset before the merge.

In the Merge activity represented by step 2, Dataset A is merged into Dataset C. Dataset A's precedence values at every level are higher than Dataset C's, so after this step Dataset C contains all the data from Dataset A. You can also see that though Dataset C's precedence value is still 100, the precedence values of the data in it have been adopted from Dataset A.

In the Merge activity represented by step 3, Dataset B is merged into Dataset C. Dataset B's precedence value of 300 is enough to beat the precedence values stored for all attributes of the Application System and the IPAddress attribute of the Computer System, so its data replaces the data written from Dataset A in step 1. But Dataset B's data for all attributes of the Monitor and the SystemType attribute of the Computer System is not written to the target because the data placed there from Dataset A has higher precedence values.

For more information about merging, see Planning data reconciliation.

Combining activities as a reconciliation job

You can create and execute reconciliation activities only as part of a job, which can contain multiple activities and performs them in the sequence that you specify. When you remove an activity from a job, it is deleted and cannot be used in other jobs. You can run a job:

• With schedules defined for the job
• Manually
• From another job
• From a BMC CMDB API program
• From Remedy AR System workflow

When you use Remedy AR System workflow or a BMC CMDB API program to execute a job, you can dynamically specify datasets and qualifications for the job to operate against, replacing those defined for the job. This enables you to reuse reconciliation definitions with multiple overlay datasets and with subsets of data.

For more information, see Creating a reconciliation job.