Understanding the export process
This section provides a detailed description of the export process and the format and function of mapping files.
Export process overview
When performing an export, the exporter needs to read data from BMC Discovery's datastore, restructure it so that it matches the schema of the system it is being exported to and export it to the remote system. If any errors occur during the export of the data then the exporter needs to decide how much of the data to roll back.
To accomplish this, the exporter goes through the following steps:
- Determines which system to send the data to, and with which connection parameters. This information comes from the adapter configuration specified by the user.
- Runs a connection test to the specified remote system.
- Determines which data is to be exported to the remote system, and how that data is to be restructured during the export.
This information comes from the mapping set specified by the user. Each mapping file in the mapping set describes a dataset in BMC Discovery's datastore. For each mapping file, the specified data is retrieved, restructured and exported.
- After all the mapping files have been run, performs any final tasks (logs the export statistics, writes any required manifests) and closes the connection.
The most complicated part of the export process is the restructuring of data. Each record in the BMC Discovery dataset specified in the mapping file is converted by the exporter into a set of configuration items (CIs). One CI is conceptually similar to a record in a table. A set of CIs can be thought of as a set of records in various tables that are linked by foreign keys. For example, a set of CIs could contain one host CI, four IPAddress CIs and a CPU CI.
Each set of CIs has one Main CI. All of the others are sub CIs. In the previous example, the Host would be the main CI, and the IPAddress and CPU items would be sub CIs.
During the export, each record of the BMC Discovery data set is converted into one set of CIs. Each set of CIs is exported together. If the export of any CI in the set fails, then the whole set is rolled back. For example, if the CPU CI in our example cannot be inserted because a required field is left blank, then the Host and IP Address items will not be inserted either.
An overview of the process for retrieving, restructuring and exporting the data for each mapping file is shown in the following figure.
There are four key steps in the process:
- The query string is run. You can find more information about the Query section in a mapping file in The Mapping File Format.
- The result of running that query is produced.
- The records are transformed into a set of CIs. This is described in more detail in Transforming a BMC Discovery Dataset using a Mapping File.
- The set of CIs are inserted into the database.
Export is not synchronization
When BMC Discovery data is exported, the appliance does not delete any previously exported data on the target system. For example, when exporting to an RDBMS, you need to perform a manual table truncate procedure to remove the data.
Mapping file format
This section provides an introduction to the mapping file format. It corresponds to step 1 in the Key Steps section. For further details about the sections of the mapping files, see a Closer look at Mapping Files.
Each mapping file is made up of two sections, the query section and the transformation section.
- The query is a standard search service query interpreted by BMC Discovery. BMC Discovery uses the query to retrieve information from the datastore and then returns the result to the exporter. Further information about search service can be found in the Using the Search and Reporting service.
- The transformation section specifies how the results of the query will be transformed into the appropriate format for publication by an adapter, for example, CSV files.
The following diagram shows the Query and Transform sections of a default mapping file. The diagram also shows the way the transformation section is divided into Main and Sub CIs.
To transform a BMC Discovery dataset by using a mapping file
This section describes step 3 in the Key Steps section.
A mapping file contains a BMC Discovery datastore query. When the mapping file is run, the query is executed on the datastore and the query result from this is used as the source data to the transformation specified in the mapping file.
Consider the following query:
This query returns the following result set:
The payroll application
Our company website
The employee expenses application
The first two fields (Name and Description) have returned one value per record. The next two fields, on the other hand, are the result of key expression traversals over a relationship. They each return a sequence of values: one value per relationship that was traversed. They are the result of traversing all relationships of the type
RunningSoftware:HostedSoftware:Host from the Business Application Instance (BAI). (There is no need to use explode to cause the key expressions to be treated as separate rows in the output.)
The first BAI (Payroll) had three such relationships, and so the
host_fqdn fields returned three values each for that BAI's record. The second BAI (Website) had four such relationships, while the last BAI (Employee Expenses) had two.
Both of the fields that returned sequences (
host_fqdn) returned sequences that correspond. The first entry in the Payroll's
host_hostname field (webserv01) corresponds to the first entry in Payroll's
host_fqdn field. The second and third entries in each field also match.
Using these corresponding sequences, we can compile a list of Hosts that are related to each application. In our example, the Payroll application could be described as follows:
We have taken one record from the result set and pivoted it, generating a Business Application Instance CI and three Host CIs from the record. This is how the transformation process works. Consider the following CI declarations from a mapping file (this is described in more detail in A Closer look at Mapping Files).
The first CI (the one declared "main") is the principal CI that this mapping file is concerned with. It is typically the node from which the various traversals start.
The sub-CI ("host") is generated from other fields in the result set. If its fields return sequences then you will need to set "collection='true'"; if you only expect one value per field then you can leave that declaration out.
The "relationship" element in the sub-CI tells the exporter how your main CI and sub CI are related. It is used when exporting to systems where the relationship has a name, such as Atrium CMDB. For the simpler adapters (such as CSV and RDB) it is ignored. If you intend to use the mapping file for these adapters only, you still need to specify the relationship, its name and direction but you can specify any values.
In order for the Exporter to validate mapping files, at least one field in each CI must be given the attribute "identity='true'".
A closer look at mapping files
Query section and the use of timestamp
A sample of the Query section of the mapping file is shown in the following section:
The Query section is built up of search service functions. For more information on how to build search queries, see the Using the Search and Reporting service.
The following search service functions are not supported by BMC Discovery Export:
In the query section, the exporter makes a variable available that contains the time at which the exporter was last run. This variable is called "lastExportFinished" and is used with the function parseTime as follows:
This generates a timestamp that the datastore can recognize.
When this variable is encountered, the exporter substitutes the variable with the date that it was last run. The exporter then sends the search query to the datastore.
By unchecking the "Export changed items only" check box, the exporter will set the lastExportFinished to 1 Jan 1980. This will result in a full export.
Example: Using the variable as part of a where clause
This variable can be used as part of a where clause.The following example will return items that have changed since the last time this exporter was run:
This variable can also be used with search services functions inside mapping file queries. For example, it can be used to filter on changes in dependencies between BAI and software collection.
If you have other conditions to place in the query's where clause, it is generally best to put the other conditions before the modified check, to avoid comparing modified times of many nodes that do not match the condition. For example:
The transformation section is made up of a number of CIs. Each CI has a name (cmdb-name) and a number of field elements. There is one main CI and zero or more sub CIs. There can only be one main element (it has the attribute "main" set to true).
Main CI transformation
In this section of the mapping file, the main attribute is set to "true", indicating that this is the main CI.
The name of the CI on the remote computer is BMC_Application.
The set of fields with
identity = 'true' together uniquely identify this CI. Identity tags can be set on one or more fields.
If more than one field has "identity='true'" set then the exporter will only overwrite an existing item if it has identical values in all of the identity fields. In other words, multiple identity fields cause an AND operation, not an OR.
Errors during the mapping validation phase
Errors might be raised during the mapping validation phase. The following table describes these possible errors.
A field cannot have both a 'const' and a 'src' attribute. Field: X
Field X has both a "const" and a "src" attribute specified. These attributes specify where the data for the field will come from; only use one of them.
Every field must have either a 'const' or a 'src' attribute. Field: X
The specified field did not have either a "src" or "const" attribute. These attributes specify where the data for the field will come from; one of them is required.
The sub CI X has no relationship configured.
A sub-CI (ie. one without a "main='true'" attribute) needs to have a "Relationship" element. This relationship element is ignored for simple adapters such as the RDB or CSV adapters; it can be specified as:
The CI has no identity fields and no node reference field.
All CIs need to have at least one field marked with "identity='true'". The exporter uses these attributes to reconcile CIs with those in the destination system.
The first field of a collection CI cannot be const.
The first field of a CI with "collection='true'" cannot use "const" as a source. If you only need one data field in your CI, and it is const then export at least one other field of data with the CI.
The main CI cannot be a collection.
The main CI (ie. the CI with "main='true'") cannot be a collection CI. Remove the "collection='true'" attribute.
The main CI cannot have a relationship configured.
The main CI (ie. the CI with "main='true'") is related to all the other CIs in the set by the relationships configured with those CIs. It cannot have a relationship of its own configured. Remove the "Relationship" element.