This documentation supports the 20.02 (12.0) version of BMC Discovery.To view an earlier version of the product, select the version from the Product version menu.

Understanding the export process

The BMC Discovery export framework is deprecated

The BMC Discovery export framework is deprecated in the 20.02 (12.0) release of BMC Discovery and will be removed in a future release. For new exports or integrations, you should consider Using-the-REST-API.

Related topics

Using

Setting-up-and-configuring-an-export

BMC Discovery Export enables you to publish data from BMC Discovery directly into relational databases and into all systems that can accept CSV files. BMC Discovery data is exported by creating an exporter, which is a combination of a mapping set with an adapter configuration. These concepts are explained later in this section.

Export does not delete any BMC Discovery data from BMC Discovery, or from the export target.

Exporters

An exporter is the combination of a mapping set and an adapter configuration.

  • The mapping set is a set of mapping files. These allow you to determine which data to extract from BMC Discovery and how to transform it according to the target data model.
  • The adapter configuration is used to determine where to publish the data.

Mapping files and sets

When performing an export, the exporter needs to know which data to read from the BMC Discovery datastore and where to place it. It does this with mapping files. A mapping file is an XML document that contains the following information:

  1. A BMC Discovery datastore query specifying which data to export
  2. Information on how BMC Discovery Export must restructure (or transform) the query result in order to make it usable by the system being exported to (that is, to comply with the remote system's schema)

A mapping set is a set of mapping files that are used to carry out the export. In order to perform an export using a mapping set, the exporter will run the query from each one of the mapping files in the set. A typical export requires more than one query of the datastore. For example, one query to get hosts, another to get Business Application Instances (BAIs), and so on.

For further information about mapping file formats, see The Mapping File Format.

Adapters and adapter configurations

BMC Discovery Export supports the following adapters:

  • RDB — populates relational databases
  • CSV — generates CSV files

An adapter configuration typically contains the information needed to connect to the target destination:

  • Target host
  • Port
  • User name
  • Password

You can create multiple adapter configurations. For example, if you needed to export data from BMC Discovery to two relational databases ("development" and "production") then you would create two adapter configurations for the RDB adapter, one for each destination.

To permit users to edit the adapter configuration

The cmdb-export-administrator user group gives permissions for users to edit the adapter's configuration. To add a user to this group:

  1. From the BMC Discovery appliance UI, log in to BMC Discovery.
  2. From the main menu, click the Administration icon. The Administration page displays. 
  3. From the Security section, click Users.
  4. Edit the profile of any user that you want to administer BMC Discovery Export. Add the user to the cmdb-export-administrator group. The user must be a member of the public group.


This section provides a detailed description of the export process and the format and function of mapping files.

Export process overview

When performing an export, the exporter needs to read data from BMC Discovery's datastore, restructure it so that it matches the schema of the system it is being exported to and export it to the remote system. If any errors occur during the export of the data then the exporter needs to decide how much of the data to roll back.
To accomplish this, the exporter goes through the following steps:

  1. Determines which system to send the data to, and with which connection parameters. This information comes from the adapter configuration specified by the user.
  2. Runs a connection test to the specified remote system.
  3. Determines which data is to be exported to the remote system, and how that data is to be restructured during the export.
    This information comes from the mapping set specified by the user. Each mapping file in the mapping set describes a dataset in BMC Discovery's datastore. For each mapping file, the specified data is retrieved, restructured and exported.
  4. After all the mapping files have been run, performs any final tasks (logs the export statistics, writes any required manifests) and closes the connection.

The most complicated part of the export process is the restructuring of data. Each record in the BMC Discovery dataset specified in the mapping file is converted by the exporter into a set of configuration items (CIs). One CI is conceptually similar to a record in a table. A set of CIs can be thought of as a set of records in various tables that are linked by foreign keys. For example, a set of CIs could contain one host CI, four IPAddress CIs and a CPU CI.

Each set of CIs has one Main CI. All of the others are sub CIs. In the previous  example, the Host would be the main CI, and the IPAddress and CPU items would be sub CIs.

During the export, each record of the BMC Discovery data set is converted into one set of CIs. Each set of CIs is exported together. If the export of any CI in the set fails, then the whole set is rolled back. For example, if the CPU CI in our example cannot be inserted because a required field is left blank, then the Host and IP Address items will not be inserted either.

An overview of the process for retrieving, restructuring and exporting the data for each mapping file is shown in the following figure.
TheExportProcess.png

Key steps

There are four key steps in the process:

  1. The query string is run. You can find more information about the Query section in a mapping file in The Mapping File Format.
  2. The result of running that query is produced.
  3. The records are transformed into a set of CIs. This is described in more detail in Transforming a BMC Discovery Dataset using a Mapping File.
  4. The set of CIs are inserted into the database.

Export is not synchronization

When BMC Discovery data is exported, the appliance does not delete any previously exported data on the target system. For example, when exporting to an RDBMS, you need to perform a manual table truncate procedure to remove the data.

Mapping file format

This section provides an introduction to the mapping file format. It corresponds to step 1 in the Key Steps section. For further details about the sections of the mapping files, see a Closer look at Mapping Files.
Each mapping file is made up of two sections, the query section and the transformation section.

  • The query is a standard search service query interpreted by BMC Discovery. BMC Discovery uses the query to retrieve information from the datastore and then returns the result to the exporter. Further information about search service can be found in the Using-the-Search-and-Reporting-service.
  • The transformation section specifies how the results of the query will be transformed into the appropriate format for publication by an adapter, for example, CSV files.

The following diagram shows the Query and Transform sections of a default mapping file. The diagram also shows the way the transformation section is divided into Main and Sub CIs.
QueryAndTransforms.png

To transform a BMC Discovery dataset by using a mapping file

This section describes step 3 in the Key Steps section.

A mapping file contains a BMC Discovery datastore query. When the mapping file is run, the query is executed on the datastore and the query result from this is used as the source data to the transformation specified in the mapping file.
Consider the following query:

search BusinessApplicationInstance
    where parseTime("${lastExportFinished}") < modified(#)
    show name, description,
      #RunningSoftware:HostedSoftware:Host:Host.hostname as host_hostname,
      #RunningSoftware:HostedSoftware:Host:Host.local_fqdn as host_fqdn

This query returns the following result set:

 

name

description

host_hostname

host_fqdn

1

Payroll

The payroll application

websrv01

Webserv01.mycompany.com

 

 

 

london_orcl

London_orcl.mycompany.com

 

 

 

sap_01

Sap_01.mycompany.com

2

Website

Our company website

Webserv01

Webserv01.mycompany.com

 

 

 

Webserv02

Webserv02.mycompany.com

 

 

 

Webserv03

Webserv03.mycompany.com

 

 

 

London_orcl

London_orcl.mycompany.com

3

Employee Expenses

The employee expenses application

websrv01

Webserv01.mycompany.com

 

 

 

london_orcl

London_orcl.mycompany.com

The first two fields (Name and Description) have returned one value per record. The next two fields, on the other hand, are the result of key expression traversals over a relationship. They each return a sequence of values: one value per relationship that was traversed. They are the result of traversing all relationships of the type RunningSoftware:HostedSoftware:Host from the Business Application Instance (BAI). (There is no need to use explode to cause the key expressions to be treated as separate rows in the output.)

The first BAI (Payroll) had three such relationships, and so the host_hostname and host_fqdn fields returned three values each for that BAI's record. The second BAI (Website) had four such relationships, while the last BAI (Employee Expenses) had two.

Both of the fields that returned sequences (host_hostname and host_fqdn) returned sequences that correspond. The first entry in the Payroll's host_hostname field (webserv01) corresponds to the first entry in Payroll's host_fqdn field. The second and third entries in each field also match.

Using these corresponding sequences, we can compile a list of Hosts that are related to each application. In our example, the Payroll application could be described as follows:

Name: Payroll
Description: The payroll application
  Hosts:
  Host 1
  Hostname: webserv01
  FQDN: webserv01.mycompany.com

  Host 2
  Hostname: London_orcl
  FQDN: london_orcl.mycompany.com

  Host 3
  Hostname: sap_01
  FQDN: sap_01.mycompany.com

We have taken one record from the result set and pivoted it, generating a Business Application Instance CI and three Host CIs from the record. This is how the transformation process works. Consider the following CI declarations from a mapping file (this is described in more detail in A Closer look at Mapping Files).

<ci cmdb-name="bai" main="true">
    <field src="name" dest="Name" identity="true"/>
    <field src="description" dest="Description"/>
  </ci>

<ci cmdb-name="host" collection="true">
    <field src="host_hostname" dest="HostHostName" identity="true"/>
    <field src="host_fqdn" dest="HostFQDN" identity="true"/>
    <relationship cmdb-name="hostedsoftware" direction="main-to-sub"/>
  </ci>

The first CI (the one declared "main") is the principal CI that this mapping file is concerned with. It is typically the node from which the various traversals start.

The sub-CI ("host") is generated from other fields in the result set. If its fields return sequences then you will need to set "collection='true'"; if you only expect one value per field then you can leave that declaration out.

The "relationship" element in the sub-CI tells the exporter how your main CI and sub CI are related. It is used when exporting to systems where the relationship has a name, such as Atrium CMDB. For the simpler adapters (such as CSV and RDB) it is ignored. If you intend to use the mapping file for these adapters only, you still need to specify the relationship, its name and direction but you can specify any values.

In order for the Exporter to validate mapping files, at least one field in each CI must be given the attribute "identity='true'".

A closer look at mapping files

Query section and the use of timestamp

A sample of the Query section of the mapping file is shown in the following section:

search BusinessApplicationInstance
    where parseTime("${lastExportFinished}") < modified(#)
    show name, description, #id as noderef,
      #RunningSoftware:HostedSoftware:Host:Host.#id as host_noderef,
      #RunningSoftware:HostedSoftware:Host:Host.hostname as hostname,
      #RunningSoftware:HostedSoftware:Host:Host.name as hosts_name

The Query section is built up of search service functions. For more information on how to build search queries, see the Using-the-Search-and-Reporting-service.

Note

The following search service functions are not supported by BMC Discovery Export:

  • dq
  • dq_band
  • dq_metric

In the query section, the exporter makes a variable available that contains the time at which the exporter was last run. This variable is called "lastExportFinished" and is used with the function parseTime as follows:

parseTime("${lastExportFinished}")

This generates a timestamp that the datastore can recognize.

When this variable is encountered, the exporter substitutes the variable with the date that it was last run. The exporter then sends the search query to the datastore.

By unchecking the "Export changed items only" check box, the exporter will set the lastExportFinished to 1 Jan 1980. This will result in a full export.

Example: Using the variable as part of a where clause
This variable can be used as part of a where clause.The following example will return items that have changed since the last time this exporter was run:

search Host
where parseTime("${lastExportFinished}") < modified(#)
show hostname

This variable can also be used with search services functions inside mapping file queries. For example, it can be used to filter on changes in dependencies between BAI and software collection.

If you have other conditions to place in the query's where clause, it is generally best to put the other conditions before the modified check, to avoid comparing modified times of many nodes that do not match the condition. For example:

search Host
where os_type = "Windows" and
parseTime("${lastExportFinished}") < modified(#)
show hostname

Transformation section

The transformation section is made up of a number of CIs. Each CI has a name (cmdb-name) and a number of field elements. There is one main CI and zero or more sub CIs. There can only be one main element (it has the attribute "main" set to true).

Main CI transformation

In this section of the mapping file, the main attribute is set to "true", indicating that this is the main CI.

<ci cmdb-name="BMC_Application" main="true">
    <field src="name" dest="Name" identity="true"/>
    <field src="description" dest="shortDescription"/>
</ci>

The name of the CI on the remote computer is BMC_Application.

The set of fields with identity = 'true' together uniquely identify this CI. Identity tags can be set on one or more fields.

Note

If more than one field has "identity='true'" set then the exporter will only overwrite an existing item if it has identical values in all of the identity fields. In other words, multiple identity fields cause an AND operation, not an OR.

Errors during the mapping validation phase

Errors might be raised during the mapping validation phase. The following table describes these possible errors.

Error

Description

A field cannot have both a 'const' and a 'src' attribute. Field: X

Field X has both a "const" and a "src" attribute specified. These attributes specify where the data for the field will come from; only use one of them.

Every field must have either a 'const' or a 'src' attribute. Field: X

The specified field did not have either a "src" or "const" attribute. These attributes specify where the data for the field will come from; one of them is required.

The sub CI X has no relationship configured.

A sub-CI (ie. one without a "main='true'" attribute) needs to have a "Relationship" element. This relationship element is ignored for simple adapters such as the RDB or CSV adapters; it can be specified as:<relationship cmdb-name="ignored" direction="main-to-sub"/>

The CI has no identity fields and no node reference field.

All CIs need to have at least one field marked with "identity='true'". The exporter uses these attributes to reconcile CIs with those in the destination system.
For the RDB and CSV adapters the "identity" attributes on sub-CIs are ignored, but at least one must still be provided — just specify the first field in the CI as "identity='true'".

The first field of a collection CI cannot be const.

The first field of a CI with "collection='true'" cannot use "const" as a source. If you only need one data field in your CI, and it is const then export at least one other field of data with the CI.

The main CI cannot be a collection.

The main CI (ie. the CI with "main='true'") cannot be a collection CI. Remove the "collection='true'" attribute.

The main CI cannot have a relationship configured.

The main CI (ie. the CI with "main='true'") is related to all the other CIs in the set by the relationships configured with those CIs. It cannot have a relationship of its own configured. Remove the "Relationship" element.

The mapping file XSD

 

Click here to show an example of the XSD that describes the format of the mapping files.
<?xml version="1.0" encoding="utf-8"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">

  <!-- The direction of a relationship between the main
       and a sub CI -->
  <xs:simpleType name="relationshipDirection">
    <xs:restriction base="xs:string">
      <xs:enumeration value="main-to-sub"/>
      <xs:enumeration value="sub-to-main"/>
    </xs:restriction>
  </xs:simpleType>

  <!-- A field in the CMDB, ie. a CI attribute. -->
  <xs:complexType name="Field">
  <!--
      Set a field's "const" attribute to specify a set value to write
      to a field in the CMDB. A Field can have either "const" or
      "src" set, not both.
  -->
    <xs:attribute name="const" type="xs:string"/>
    <!--
    The field in the query's result set to take this field's value
    from. You can use aliases for long fields.
    For example, use "name" as the src attribute in the following
    query:
          query host show name

    Or use the alias name "ip_address" as the src attribute in
    the following:
          query host
          show
          #HostWithAddress:HostAddress:AddressOfHost:
          IpAddress.address as ip_address
    (The traversal is on two lines for readability, in reality
    it is on a single line.)
    -->
    <xs:attribute name="src" type="xs:string"/>

    <!--
    The CMDB class' attribute to write the value to.
    -->
    <xs:attribute name="dest" type="xs:string" use="required"/>

    <!--
    If this field is required in the CMDB, set this value to "true".
    If you set this to "true" and a null value is returned from the
    data store, the sub-CI will be ignored, that is, the exporter
    will not attempt to insert it into the CMDB.

    Failure to do this could cause the CMDB to throw an exception
    when
 committing the transaction for the main CI. You would lose
    the main CI and all of its sub-CIs instead of just the sub-CI
    with the null field.
    -->
    <xs:attribute name="required" type="xs:boolean" default="false"/>

    <!--
    Does this field form part of the CI's identity? "false" by
    default. If any field is set to true then the transform engine
    will do a lookup in this CMDB class to see if an object with the
    same values for the identity fields exists, and use that if
    one is found. If none is found, this item will be inserted.
    If no fields are marked as identity fields, all items will be
    inserted as new, even if they're identical.
    -->
    <xs:attribute name="identity" type="xs:boolean"          default="false"/>
    </xs:complexType>

  <!-- A relationship between the main CI and a sub-CI. -->
    <xs:complexType name="Relationship">
      <xs:sequence>
      <!--
      A relationship can have fields in the same way as a CI can.
      -->
      <xs:element name="field" type="Field" minOccurs="0"        
           maxOccurs="unbounded"/>
      </xs:sequence>
    <!--
    The cmdb class name for this CI.
    -->
    <xs:attribute name="cmdb-name" type="xs:string" use="required"/>
    <!--
    The direction the relationship goes in. For example, if this is
    the dependency relationship from an application to a host, then
    the application is dependent on the host. The direction is thus
    "main-to-sub" if the main CI is the application.
    -->
    <xs:attribute name="direction" type="relationshipDirection"
         use="required"/>
    </xs:complexType>


  <!-- A mapping for one CI class -->
  <xs:complexType name="CI">
    <xs:sequence>
      <!--
      The fields for this CI
      -->
      <xs:element name="field" type="Field" minOccurs="1"
           maxOccurs="unbounded"/>

    <!--
    How this CI relates to the main CI
    -->
      <xs:element name="relationship" type="Relationship"
           minOccurs="0" maxOccurs="1"/>
      </xs:sequence>

    <!--
    CI class name in the CMDB
    -->
    <xs:attribute name="cmdb-name" type="xs:string" use="required"/>

    <!--
    Is this the "main" CI? There is one main CI and (optionally)
    many sub-CIs.
    -->
    <xs:attribute name="main" type="xs:boolean" default="false"/>

    <!--
    If true (the default), any existing item found in the CMDB based
    on the identity fields will have its non-identity fields
    overwritten by the fields from the imported item. If false, the
    existing object will be left untouched.
    This value should possibly be set per-attribute.
    -->
    <xs:attribute name="overwrite-non-id-fields" type="xs:boolean"
         default="true"/>

    <!--
    If true, then all of the fields that make up this CI have to be
    collection fields, and all the collections have to have the same
    length. The transform engine will generate one sub-CI per set of
    values in the collections.

    If this is false and any of the fields for this CI are returned
    as a collection, an error will occur.

    This is only applicable to sub-CIs. Defaults to "false".
    -->
    <xs:attribute name="collection" type="xs:boolean"
         default="false"/>

    <!--
    Some items exist only in the context of their main CIs. A good
    example is an IP address - the same IP address may exist many
    times on the network, but will only exist once per Host. Thus,
    the IP address is identified by its address and its relationship
    to its main CI, namely the Host.
     
    Set this attribute to true to tell the lookup to treat the
    relationship to the main CI as part of the identity.
    -->
    <xs:attribute name="parent-is-identifier" type="xs:boolean"
         default="false"/>

    <!--
    Specify which field contains a reference to the BMC Discovery node
    that this CI is an export of. The exporter uses this reference
    to store and retrieve the ID given to this CI in the remote
    system. This ID is then used to reconcile the CI against the
    remote system before resorting to lookups based on identity
    fields.


    In BMC Discovery QL, you specify a node reference by "#". For
    example, you retrieve some fields and the node reference from
    a Host by:
            "search Host show name, fqdn, # as host_ref"

    In this example, you would set the "node-reference-field" to
    "host_ref".
    -->
    <xs:attribute name="node-reference-field" type="xs:string"/>
  </xs:complexType>

  <!-- The main document element. -->
  <xs:element name="mapping">
    <xs:complexType>
      <xs:sequence>
        <!-- The query to run against the datastore -->
        <xs:element name="query" type="xs:string" minOccurs="1"
           maxOccurs="1"/>
        <!-- The CIs in this mapping-->
        <xs:element name="ci" type="CI" minOccurs="0"
          maxOccurs="unbounded"/>
      </xs:sequence>
      <xs:attribute name="description" type="xs:string"
         default="No description."/>    
      <xs:attribute name="delete-kind" type="xs:string" />
    </xs:complexType>
  </xs:element>
</xs:schema>