Understanding the export process

This section provides a detailed description of the export process and the format and function of mapping files.

Export process overview

When performing an export, the exporter needs to read data from BMC Discovery's datastore, restructure it so that it matches the schema of the system it is being exported to and export it to the remote system. If any errors occur during the export of the data then the exporter needs to decide how much of the data to roll back.
To accomplish this, the exporter goes through the following steps:

1. Determines which system to send the data to, and with which connection parameters. This information comes from the adapter configuration specified by the user.
2. Runs a connection test to the specified remote system.
3. Determines which data is to be exported to the remote system, and how that data is to be restructured during the export.
   This information comes from the mapping set specified by the user. Each mapping file in the mapping set describes a dataset in BMC Discovery's datastore. For each mapping file, the specified data is retrieved, restructured and exported.
4. After all the mapping files have been run, performs any final tasks (logs the export statistics, writes any required manifests) and closes the connection.

The most complicated part of the export process is the restructuring of data. Each record in the BMC Discovery dataset specified in the mapping file is converted by the exporter into a set of configuration items (CIs). One CI is conceptually similar to a record in a table. A set of CIs can be thought of as a set of records in various tables that are linked by foreign keys. For example, a set of CIs could contain one host CI, four IPAddress CIs and a CPU CI.

Each set of CIs has one Main CI. All of the others are sub CIs. In the previous  example, the Host would be the main CI, and the IPAddress and CPU items would be sub CIs.

During the export, each record of the BMC Discovery data set is converted into one set of CIs. Each set of CIs is exported together. If the export of any CI in the set fails, then the whole set is rolled back. For example, if the CPU CI in our example cannot be inserted because a required field is left blank, then the Host and IP Address items will not be inserted either.

An overview of the process for retrieving, restructuring and exporting the data for each mapping file is shown in the following figure.

Key steps

There are four key steps in the process:

1. The query string is run. You can find more information about the Query section in a mapping file in The Mapping File Format.
2. The result of running that query is produced.
3. The records are transformed into a set of CIs. This is described in more detail in Transforming a BMC Discovery Dataset using a Mapping File.
4. The set of CIs are inserted into the database.

Mapping file format

This section provides an introduction to the mapping file format. It corresponds to step 1 in the Key Steps section. For further details about the sections of the mapping files, see a Closer look at Mapping Files.
Each mapping file is made up of two sections, the query section and the transformation section.

• The query is a standard search service query interpreted by BMC Discovery. BMC Discovery uses the query to retrieve information from the datastore and then returns the result to the exporter. Further information about search service can be found in the Using the Search and Reporting service.
• The transformation section specifies how the results of the query will be transformed into the appropriate format for publication by an adapter, for example, CSV files.

The following diagram shows the Query and Transform sections of a default mapping file. The diagram also shows the way the transformation section is divided into Main and Sub CIs.

To transform a BMC Discovery dataset by using a mapping file

This section describes step 3 in the Key Steps section.

A mapping file contains a BMC Discovery datastore query. When the mapping file is run, the query is executed on the datastore and the query result from this is used as the source data to the transformation specified in the mapping file.
Consider the following query:

search BusinessApplicationInstance
    where parseTime("${lastExportFinished}") < modified(#)
    show name, description,
      #RunningSoftware:HostedSoftware:Host:Host.hostname as host_hostname,
      #RunningSoftware:HostedSoftware:Host:Host.local_fqdn as host_fqdn

This query returns the following result set:

The first two fields (Name and Description) have returned one value per record. The next two fields, on the other hand, are the result of key expression traversals over a relationship. They each return a sequence of values: one value per relationship that was traversed. They are the result of traversing all relationships of the type RunningSoftware:HostedSoftware:Host from the Business Application Instance (BAI). (There is no need to use explode to cause the key expressions to be treated as separate rows in the output.)

The first BAI (Payroll) had three such relationships, and so the host_hostname and host_fqdn fields returned three values each for that BAI's record. The second BAI (Website) had four such relationships, while the last BAI (Employee Expenses) had two.

Both of the fields that returned sequences (host_hostname and host_fqdn) returned sequences that correspond. The first entry in the Payroll's host_hostname field (webserv01) corresponds to the first entry in Payroll's host_fqdn field. The second and third entries in each field also match.

Using these corresponding sequences, we can compile a list of Hosts that are related to each application. In our example, the Payroll application could be described as follows:

Name: Payroll
Description: The payroll application
  Hosts:
  Host 1
  Hostname: webserv01
  FQDN: webserv01.mycompany.com

  Host 2
  Hostname: London_orcl
  FQDN: london_orcl.mycompany.com

  Host 3
  Hostname: sap_01
  FQDN: sap_01.mycompany.com

We have taken one record from the result set and pivoted it, generating a Business Application Instance CI and three Host CIs from the record. This is how the transformation process works. Consider the following CI declarations from a mapping file (this is described in more detail in A Closer look at Mapping Files).

<ci cmdb-name="bai" main="true">
    <field src="name" dest="Name" identity="true"/>
    <field src="description" dest="Description"/>
  </ci>

<ci cmdb-name="host" collection="true">
    <field src="host_hostname" dest="HostHostName" identity="true"/>
    <field src="host_fqdn" dest="HostFQDN" identity="true"/>
    <relationship cmdb-name="hostedsoftware" direction="main-to-sub"/>
  </ci>

The first CI (the one declared "main") is the principal CI that this mapping file is concerned with. It is typically the node from which the various traversals start.

The sub-CI ("host") is generated from other fields in the result set. If its fields return sequences then you will need to set "collection='true'"; if you only expect one value per field then you can leave that declaration out.

The "relationship" element in the sub-CI tells the exporter how your main CI and sub CI are related. It is used when exporting to systems where the relationship has a name, such as Atrium CMDB. For the simpler adapters (such as CSV and RDB) it is ignored. If you intend to use the mapping file for these adapters only, you still need to specify the relationship, its name and direction but you can specify any values.

In order for the Exporter to validate mapping files, at least one field in each CI must be given the attribute "identity='true'".

A closer look at mapping files

Query section and the use of timestamp

A sample of the Query section of the mapping file is shown in the following section:

search BusinessApplicationInstance
    where parseTime("${lastExportFinished}") < modified(#)
    show name, description, #id as noderef,
      #RunningSoftware:HostedSoftware:Host:Host.#id as host_noderef,
      #RunningSoftware:HostedSoftware:Host:Host.hostname as hostname,
      #RunningSoftware:HostedSoftware:Host:Host.name as hosts_name

The Query section is built up of search service functions. For more information on how to build search queries, see the Using the Search and Reporting service.

In the query section, the exporter makes a variable available that contains the time at which the exporter was last run. This variable is called "lastExportFinished" and is used with the function parseTime as follows:

parseTime("${lastExportFinished}")

This generates a timestamp that the datastore can recognize.

When this variable is encountered, the exporter substitutes the variable with the date that it was last run. The exporter then sends the search query to the datastore.

By unchecking the "Export changed items only" check box, the exporter will set the lastExportFinished to 1 Jan 1980. This will result in a full export.

Example: Using the variable as part of a where clause
This variable can be used as part of a where clause.The following example will return items that have changed since the last time this exporter was run:

search Host
where parseTime("${lastExportFinished}") < modified(#)
show hostname

This variable can also be used with search services functions inside mapping file queries. For example, it can be used to filter on changes in dependencies between BAI and software collection.

If you have other conditions to place in the query's where clause, it is generally best to put the other conditions before the modified check, to avoid comparing modified times of many nodes that do not match the condition. For example:

search Host
where os_type = "Windows" and
parseTime("${lastExportFinished}") < modified(#)
show hostname

Transformation section

The transformation section is made up of a number of CIs. Each CI has a name (cmdb-name) and a number of field elements. There is one main CI and zero or more sub CIs. There can only be one main element (it has the attribute "main" set to true).

Main CI transformation

In this section of the mapping file, the main attribute is set to "true", indicating that this is the main CI.

<ci cmdb-name="BMC_Application" main="true">
    <field src="name" dest="Name" identity="true"/>
    <field src="description" dest="shortDescription"/>
</ci>

The name of the CI on the remote computer is BMC_Application.

The set of fields with identity = 'true' together uniquely identify this CI. Identity tags can be set on one or more fields.

Errors during the mapping validation phase

Errors might be raised during the mapping validation phase. The following table describes these possible errors.

The mapping file XSD

<?xml version="1.0" encoding="utf-8"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">

  <!-- The direction of a relationship between the main
       and a sub CI -->
  <xs:simpleType name="relationshipDirection">
    <xs:restriction base="xs:string">
      <xs:enumeration value="main-to-sub"/>
      <xs:enumeration value="sub-to-main"/>
    </xs:restriction>
  </xs:simpleType>

  <!-- A field in the CMDB, ie. a CI attribute. -->
  <xs:complexType name="Field">
  <!--
      Set a field's "const" attribute to specify a set value to write
      to a field in the CMDB. A Field can have either "const" or 
      "src" set, not both.
  -->
    <xs:attribute name="const" type="xs:string"/>
    <!--
    The field in the query's result set to take this field's value
    from. You can use aliases for long fields. 
    For example, use "name" as the src attribute in the following
    query:
          query host show name

    Or use the alias name "ip_address" as the src attribute in
    the following:
          query host 
          show
          #HostWithAddress:HostAddress:AddressOfHost:
          IpAddress.address as ip_address
    (The traversal is on two lines for readability, in reality
    it is on a single line.)
    -->
    <xs:attribute name="src" type="xs:string"/>

    <!--
    The CMDB class' attribute to write the value to.
    -->
    <xs:attribute name="dest" type="xs:string" use="required"/>

    <!--
    If this field is required in the CMDB, set this value to "true".
    If you set this to "true" and a null value is returned from the 
    data store, the sub-CI will be ignored, that is, the exporter 
    will not attempt to insert it into the CMDB.

    Failure to do this could cause the CMDB to throw an exception
    when 
 committing the transaction for the main CI. You would lose
    the main CI and all of its sub-CIs instead of just the sub-CI
    with the null field.
    -->
    <xs:attribute name="required" type="xs:boolean" default="false"/>

    <!--
    Does this field form part of the CI's identity? "false" by
    default. If any field is set to true then the transform engine
    will do a lookup in this CMDB class to see if an object with the
    same values for the identity fields exists, and use that if
    one is found. If none is found, this item will be inserted.
    If no fields are marked as identity fields, all items will be
    inserted as new, even if they're identical.
    -->
    <xs:attribute name="identity" type="xs:boolean"          default="false"/>
    </xs:complexType>

  <!-- A relationship between the main CI and a sub-CI. -->
    <xs:complexType name="Relationship">
      <xs:sequence>
      <!--
      A relationship can have fields in the same way as a CI can.
      -->
      <xs:element name="field" type="Field" minOccurs="0"        
           maxOccurs="unbounded"/>
      </xs:sequence>
    <!--
    The cmdb class name for this CI.
    -->
    <xs:attribute name="cmdb-name" type="xs:string" use="required"/>
    <!--
    The direction the relationship goes in. For example, if this is
    the dependency relationship from an application to a host, then
    the application is dependent on the host. The direction is thus
    "main-to-sub" if the main CI is the application.
    -->
    <xs:attribute name="direction" type="relationshipDirection"
         use="required"/>
    </xs:complexType>


  <!-- A mapping for one CI class -->
  <xs:complexType name="CI">
    <xs:sequence>
      <!--
      The fields for this CI
      -->
      <xs:element name="field" type="Field" minOccurs="1"
           maxOccurs="unbounded"/>

    <!--
    How this CI relates to the main CI
    -->
      <xs:element name="relationship" type="Relationship"
           minOccurs="0" maxOccurs="1"/>
      </xs:sequence>

    <!--
    CI class name in the CMDB
    -->
    <xs:attribute name="cmdb-name" type="xs:string" use="required"/>

    <!--
    Is this the "main" CI? There is one main CI and (optionally)
    many sub-CIs.
    -->
    <xs:attribute name="main" type="xs:boolean" default="false"/>

    <!--
    If true (the default), any existing item found in the CMDB based
    on the identity fields will have its non-identity fields
    overwritten by the fields from the imported item. If false, the
    existing object will be left untouched.
    This value should possibly be set per-attribute.
    -->
    <xs:attribute name="overwrite-non-id-fields" type="xs:boolean"
         default="true"/>

    <!--
    If true, then all of the fields that make up this CI have to be 
    collection fields, and all the collections have to have the same
    length. The transform engine will generate one sub-CI per set of
    values in the collections.

    If this is false and any of the fields for this CI are returned
    as a collection, an error will occur.

    This is only applicable to sub-CIs. Defaults to "false".
    -->
    <xs:attribute name="collection" type="xs:boolean"
         default="false"/>

    <!--
    Some items exist only in the context of their main CIs. A good
    example is an IP address - the same IP address may exist many
    times on the network, but will only exist once per Host. Thus,
    the IP address is identified by its address and its relationship
    to its main CI, namely the Host.
      
    Set this attribute to true to tell the lookup to treat the
    relationship to the main CI as part of the identity.
    -->
    <xs:attribute name="parent-is-identifier" type="xs:boolean"
         default="false"/>

    <!--
    Specify which field contains a reference to the BMC Discovery node
    that this CI is an export of. The exporter uses this reference
    to store and retrieve the ID given to this CI in the remote
    system. This ID is then used to reconcile the CI against the
    remote system before resorting to lookups based on identity
    fields.


    In BMC Discovery QL, you specify a node reference by "#". For
    example, you retrieve some fields and the node reference from
    a Host by:
            "search Host show name, fqdn, # as host_ref"

    In this example, you would set the "node-reference-field" to
    "host_ref".
    -->
    <xs:attribute name="node-reference-field" type="xs:string"/>
  </xs:complexType>

  <!-- The main document element. -->
  <xs:element name="mapping">
    <xs:complexType>
      <xs:sequence>
        <!-- The query to run against the datastore -->
        <xs:element name="query" type="xs:string" minOccurs="1"
           maxOccurs="1"/>
        <!-- The CIs in this mapping-->
        <xs:element name="ci" type="CI" minOccurs="0"
          maxOccurs="unbounded"/>
      </xs:sequence>
      <xs:attribute name="description" type="xs:string" 
         default="No description."/>    
      <xs:attribute name="delete-kind" type="xs:string" />
    </xs:complexType>
  </xs:element>
</xs:schema>