Importing data from BMC Discovery
Note
Starting with the 20.18.01 release, Remedyforce has replaced the older BMC Discovery integration, which required Pentaho, with an improved (API-based, no Pentaho required) integration. This improved integration offers many enhancements including additional classes, relationships and synchronization. It is recommended that you move to the new integration to take advantage of the many new features. To learn more about the differences between the old (Pentaho) integration and the new (API-based) integration, please refer to the
Remedyforce and BMC Discovery FAQ
documentation. For more information about how the new integration works, see Integrating Remedyforce with BMC Discovery.
You can import data from BMC Discovery (known before version 11.0 as BMC Atrium Discovery and Dependency Mapping) into BMC Remedyforce CMDB 2.0 by using the Pentaho Data Integration tool. The Pentaho packages for importing data from BMC Discovery are available on the BMC Communities website. For information about how configuration items (CIs) are imported and relationships are created in CMDB 2.0, see Overview of how data is imported into BMC Remedyforce.
The following topics provide information about importing data from BMC Discovery into BMC Remedyforce CMDB:
Note
For information about the BMC Discovery versions that BMC Remedyforce supports, see Supported browsers, mobile devices, and integrations.
Before you begin
Before you can import data from BMC Discovery into BMC Remedyforce CMDB, you must perform the following tasks:
- Install Java Runtime Environment (JRE).
- Download the Pentaho Data Integration tool.
Note: BMC Discovery 11.0 works with the existing Pentaho packages that are available on BMC Communities. However, there is a known issue with the Edit Mapping feature. For more information about the issue and the workaround, see issue 79517 in the Known and corrected issues topic. - Create the KETTLE_HOME variable to store the location of the Kitchen.bat file.
- Register at the BMC Communities website.
- (Optional) Configure BMC Remedyforce to retain the original Instance Name value of CIs imported from an external source.
- If you have enabled the setting to access your Salesforce organization from limited IP addresses, ensure that you have the security token to log on to your Salesforce organization.
For more information about the security token, see Salesforce Help. Download the appropriate Pentaho package from the BMC Communities website. The following table helps you to decide which package to download:
Package to download Details Imports CIs from BMC Discovery and then creates relationships between the imported CIs in BMC Remedyforce CMDB based on the Assembly ID field. Imports CIs and relationships of Computer System CIs with IPEndpoint, LANEndpoint, OperatingSystem, Processor, and Software Server CIs from BMC Discovery.
This package also sets the HasImpact field value of the relationship records imported in BMC Remedyforce CMDB to Yes.
For information about the KJB (job) and KTR (transformation) files that are included in the Pentaho package, see KJB and KTR files for importing data from BMC Discovery.
Best practice
BMC recommends that you do not change the folder structure because doing so can interfere with the running of the batch files provided in the CombinedJobs folder.
To import data from BMC Discovery
- Navigate to the location where you downloaded and unzipped the Pentaho Data Integration tool, navigate to the data-integration folder, and make a backup copy of the Kitchen.bat file.
- In the folder that you downloaded from the BMC Communities website, double-click the Common folder, copy the Kitchen.bat file, and paste it into the data-integration folder.
- To launch the Pentaho Data Integration tool, navigate to the data-integration folder and double-click the Spoon.bat file.
- In Pentaho Spoon, select File > Open, navigate to the folder where you downloaded the Pentaho package, and open the appropriate KJB file.
For example, to import LAN Endpoint CIs into BMC Remedyforce CMDB, open the TransferADDMLANEndpointInfotoCMDB.kjb file in the LANEndpoint folder. In the Transformation step (transformation), provide your BMC Discovery server access details.
Navigate to the KJB file that you opened in step 4.
In the Update BE Table With ADDM <CI type name> Information step (transformation), provide your Salesforce organization access details and, if required, customize the Pentaho package.
Important
If you have enabled the setting to access your Salesforce organization from limited IP addresses, you must append the security token to your Salesforce organization password.
For example, if the password for your Salesforce organization is mypassword and your security token is XXXXXXXXX, specify mypasswordXXXXXXXXX in the Password fields.
To save the KTR and KJB files, click Save.
- Close the Pentaho Data Integration tool.
For the imported records, the Source field is set to BMC Discovery. You can use the Source field to generate reports and also when you enable and configure reconciliation. From the folder of the CI type for which you are importing data, double-click the .bat file provided in the folder.
For example, if you are importing data for Computer System, open the SchedulingComputerSystem.bat file.Important
You must use the .bat file to avoid any errors while importing.
To schedule jobs to import data
On the BMC Communities website, the Pentaho packages for importing data from external data sources into BMC Remedyforce page contain a Scheduling Files folder. This folder contains a sample batch file that you can use as a base to create your batch files for scheduling data import. For more information about scheduling a job, see http://wiki.Pentaho.com/display/EAI/Kitchen+User+Documentation.
- Navigate to the folder where you have extracted the Pentaho package for which you want to schedule import jobs.
- Open the Scheduling Files folder and create a copy of the sample batch file, such as SchedulingComputerSystem.bat.
- Open the batch file that you created in step 2, perform the following actions, and save the file:
- Update the path of the Kitchen.bat file based on your version of the Pentaho Data Integration tool:
KETTLE_HOME="<path>\<PentahoFolder>\data-integration"
- Update the path of the Kitchen.bat file and the KJB file:
<path>\
<PentahoFolder>
\data-integration\kitchen.bat /file:"<JobFile>\<JobFileName>.kjb"/level:Detailed
- <path> is the location where you have extracted the Pentaho installer ZIP file.
- <PentahoFolder> is the extracted root folder for your version of the Pentaho Data Integration tool.
For example, for version 6.1 of the Pentaho Data Integration Tool, the extracted root folder ispdi-ce-6.1.0.1-196.
- <JobFile> is the location where you have extracted the job files.
- <JobFileName> is the KJB file that you want to run at the scheduled interval.
- The level option specifies the level of logging for the job. The values that you can specify for the level option are Minimal, Basic, Detailed, Debugging, and Row Level. The log files are saved in the same folder where KJB files are saved.
- Update the path of the Kitchen.bat file based on your version of the Pentaho Data Integration tool:
- Go to Start > Control Panel > Administrative Tools > TaskScheduler.
- In the Task Scheduler window, in the Actions area, click Create Basic Task.
- In the Create Basic Task Wizard, type the name and description of the task, and click Next.
- Select the appropriate option for starting the task, and click Next.
For example, you can select Daily or When the computer starts as the option for starting the task. - Configure the additional options for starting the task, and click Next.
The additional options are displayed based on the option that you selected in step 7. For example, if you select the Daily option for starting the task, you must configure the start date and frequency at which you want to run the batch file. - Select the Start a program radio button, and click Next.
- Click Browse to locate the batch file that you have created to schedule the job, and click Next.
- Select the Open the Properties dialog for this task when I click Finish check box, and click Finish.
User scenario for importing data from BMC Discovery
David is a member of the Infrastructure team for Downtown Bank, responsible for managing the laptops for ABC project. He signs up for BMC Remedyforce and accepts to use the preconfigured, ITIL based incident and problem management processes for managing CIs. Also, David wants to use the CMDB to represent the physical, logical, and conceptual items and to track the relationships between the different CIs.
Charlotte, a BMC Remedyforce administrator, helps David to import details about ABC team member laptops from BMC Discovery to BMC Remedyforce CMDB. Charlotte imports the laptop data into BMC Remedyforce CMDB by running multiple jobs from the Pentaho package in the following sequence:
- To import the operating system, processor, IP EndPoint, and LAN Endpoint data, Charlotte navigates to the CombinedJobs folder and runs the TransferAdvancedADDMinfotoCMDB.bat file.
After running the job, she opens the Operating System, Processor, IP EndPoint, and LAN Endpoint classes in the Instance Editor and verifies that the entries are correct on the Relationships tab. - To import the laptop OS details, Charlotte navigates to the ComputerSystem folder and runs the SchedulingComputerSystem.bat file. Next, Charlotte navigates to the OperatingSystem folder and runs the SchedulingOperatingSystem.bat file.
After running the jobs, she opens the Operating System class in the Instance Editor and verifies that the entries are correct on the Relationships tab. - To import the laptop software-server details, Charlotte navigates to the CombinedJobs folder and runs the TransferComputerSystemAndSoftwareADDMinfotoCMDB.bat file.
After running the job, she opens the Software Server class in the Instance Editor and verifies that the entries are correct on the Relationships tab.
KJB and KTR files for importing data from BMC Discovery
The Pentaho packages contain job (KJB) and transformation (KTR) files that are created in the Pentaho Data Integration tool. The KJB files (jobs) and KTR files (transformations) store metadata in XML format. A KJB file contains a series of transformations that are run in a sequence. A KTR file contains a single transformation. The KJB (job) files retrieve data from external sources into BMC Remedyforce CMDB.
For information about the KJB and KTR files in the Pentaho packages for BMC Discovery, see KJB files in the Pentaho packages and KTR files in the Pentaho packages.
KJB files in the Pentaho packages
Each Pentaho package includes different KJB (job) files for importing different types of configuration items (CIs), such as computer systems and LAN endpoints. These job files are bundled with related files and provided to you in folders. The folder names correspond to CI types in BMC Remedyforce CMDB.
The following table provides information about the job files that you must run to import specific CI types into BMC Remedyforce CMDB:
Folder name (CI type) | Job file | Job description |
---|---|---|
ComputerSystem | TransferADDM(CS)Infoto CMDB.kjb | Transfers information about the BMC_ComputerSystem CI type |
OperatingSystem | TransferADDM(OS)Infoto CMDB.kjb | Transfers information about the BMC_OperatingSystem CI type |
Processor | TransferADDM(CPU)Infoto CMDB.kjb | Transfers information about the BMC_Processor CI type |
IPEndpoint | TransferADDMIPEndpoint InfotoCMDB.kjb | Transfers information about the BMC_IPEndPoint CI type (both IPv4 and IPv6) |
LANEndpoint | TransferADDMLANEndpoint InfotoCMDB.kjb | Transfers information about the BMC_LANEndpoint CI type |
SoftwareServer | TransferADDMSoftwareServer InfotoCMDB.kjb | Transfers information about the Note: Depending on the organization size, running the Software Server job transfers large amounts of data to the Salesforce organization, which might lead to the organization storage limit being exceeded. |
CombinedJobs |
|
|
KTR files in the Pentaho packages
All job (KJB) files for importing data from specific CI types contain a series of transformations that are run in a sequence. Each transformation maps to a KTR file that is available, along with the KJB file, in the folder for each CI type.
The following table provides information about the KTR files and the corresponding transformations that the KJB file for each CI type contains:
KTR file | Transformation in the KJB file | Description |
---|---|---|
None | Check if Delta timestamp File exists | The Pentaho package utilizes a time stamp file to determine which records were added or modified since the last time the job was run. This step is used for incremental import. If a time stamp file does not exist, the Create the initial timestamp file step is run; otherwise, the Transformation step is run. |
CreateInitialTimeStamped FileForADDM<CIType>.ktr | Create the initial Timestamp file | Records the time of import. This transformation runs only if you are importing for the first time or you have deleted the existing time stamp file. |
GenerateCsvApiFile.ktr | Transformation | Reads the saved time stamp and accepts the credentials for the BMC Discovery server. This step creates a query to fetch data for the corresponding CI type from BMC Discovery and saves that data in a file. |
TransferADDM<CIType> InfoToCMDB.ktr | Update BE Table With ADDM <CI type name> Information | Transfers data from BMC Discovery to the Base Element object of BMC Remedyforce CMDB 2.0. |
StoreADDM<CIType>Time stamp.ktr | Store the current timestamp | If the data import is successful, the time of import is saved. |
For example, the ComputerSystem folder contains the CreateInitialTimeStampedFileForADDM(CS).ktr, TransferADDM(CS)InfoToCMDB.ktr, and StoreADDM(CS)Timestamp.ktr files.
The following table provides information about the steps that are included in the TransferADDM<CIType>InfoToCMDB.ktr file (Update BE Table With ADDM <CI type name> Information transformation). The Pentaho package runs these steps to transfer data from BMC Discovery into BMC Remedyforce CMDB. You can view these steps only when you open the KTR file in the Pentaho Data Integration tool.
Step | Description |
---|---|
ADDM input | Uses the file that is created in the Transformation step (GenerateCsvApiFile.ktr) to list the BMC Discovery fields that are queried. |
Transform ADDM variables | Facilitates JavaScript transformations such as appending or trimming fields. |
Salesforce Input[CMDB_Class] | Fetches details of the BMC Remedyforce CMDB class from your Salesforce organization and accepts the credentials for the Salesforce organization. |
Get Class Name | Changes the BMC Remedyforce CMDB class name into uppercase characters to compare it with the class name that is fetched from BMC Discovery. |
Get CMDB Class Id | Maps data imported from BMC Discovery to the appropriate CI class in BMC Remedyforce CMDB. |
Sort | Sorts the records that are imported from BMC Discovery. |
Unique rows | Checks that imported records are unique so that duplicate records are not created in BMC Remedyforce CMDB |
Upsert into class <CI type name> | Stores the imported data into the specified CI class and accepts the credentials for the Salesforce organization in which you want to save the imported data. |
Success rows | Stores the rows that are imported successfully. |
Failure rows | Stores the rows that are not imported along with error code, error descriptions, and error fields. |
The following steps in this transformation are available only in the Pentaho package to import CIs and relationships into CMDB 2.0. Also, these steps are available in all jobs in the package, other than the job to import Computer System CIs (TransferADDM(CS)InfotoCMDB.kjb). | |
Salesforce Input [Base Relationship] | Fetches details of the Base Relationship class in BMC Remedyforce CMDB from your Salesforce organization and accepts the credentials for the Salesforce organization. |
Stream lookup for <CI type name> | Checks if the Base Relationship class stores any relationships with the specified CI type as the destination. |
Salesforce Input [BaseElement] | Fetches details of the Base Element class in BMC Remedyforce CMDB from your Salesforce organization and accepts the credentials for the Salesforce organization. |
Stream lookup for ComputerSystem | Identifies the computer system CIs that are the source of the relationships with the specified CI type. |
Filter rows | Identifies and filters relationships that already exist in BMC Remedyforce CMDB so that duplicate relationships are not created. |
Duplicate BaseRelationship | Records the relationships that already exist in BMC Remedyforce CMDB. |
Upsert into Base Relationships | Stores the relationship records in the This step also sets the HasImpact field value of all relationship records imported in the |
Base Relationship Upsert Failure Rows | Stores the rows that are not imported, error code, error descriptions, and error fields. |
Base Relationship Upsert Success Rows | Stores the rows that are imported successfully. |
Troubleshooting
The following table describes the troubleshooting tips that you can use to resolve common issues that you might face when importing data.
Error or issue | Applies to the Pentaho package for | Description or procedure |
---|---|---|
Viewing logs of the import jobs | All | Log files are created in the folder where you have saved the KJB files. Success and failure rows files are also created in the same folder. If a failure occurs, the error code and its description are provided in the failure row file. You can also use the failure row file to import data to Salesforce. |
Not all records are imported | All | Delete the delta time stamp file of the job that you ran, and run the job again. |
Import fails | All | If you have upgraded to BMC Remedyforce Winter 17 (20.17.01), either you use the latest Pentaho packages updated on the BMC Communities or map a value to the Source field of the Base Element object. Also, ensure that you have Edit permission on the Source field of the Base Element object. |
An "out of memory" error occurs | All | While importing a large number of records, if you get the
|
A "too many script statements" error occurs | All | When you run an import, if you receive "Too many script statements" as an onscreen Apex error message or in an email, reduce the batch size by 10 in the Batch Size field in the Settings section on the Salesforce Upsert window. |
Records are not being to your CMDB. | All | Raise a case with Salesforce to create a custom index on the Assembly ID field of the Base Element object. |
Importing users by using the failure rows file for LDAP | LDAP server | A failure row file is a text file that is saved in the folder where you have saved your KJB files. Perform the necessary steps to remove the error provided for the failure rows in the failure rows file, and then import the data by using the following steps. The following Pentaho packages provide a transformation file in the FailureRowsInput folder. You can use this transformation file for importing users from the failure rows file to the Salesforce organization.
To import data from the failure rows file
|
Failed in writeToSalesForce: java.lang.IllegalArgumentException: The char '0x0' after 'Print' is not a valid XML character | All | Some unicode characters that cannot be parsed by the XML parser are present in any of the mapped fields. Either delete these characters in data or delete the mapping of such fields. |
Unable to query Salesforce | All | Check your Salesforce organization credentials in the Salesforce Upsert and Salesforce Input(CMDB_Classes) steps. |
The job does not appear in Atrium Integrator. | BMC Atrium CMDB | Ensure that you have saved your KTR and KJB files in a folder in Atrium Integrator. |
Error setting value #2 [ARDelta_1 Integer] on prepared statement (Integer) | BMC Atrium CMDB | This error is generated if multiple records are created in the BMC Remedy AR System NGIE:Delta form for the running transformation. Delete the additional ARDelta entries that are created for the error transformation file by deleting the records. To find the duplicate records for a transformation file, open the NGIE:Delta form in BMC Remedy AR System. In the TransName field, enter the transformation file name, and click Search. |
Did not find Remedy Application Service password for server<server name> in UDM:RAppPassword Form on server <server name> | BMC Atrium CMDB | Check your Atrium Server Connection credentials in the ARInput step. |
'Oracle Database Server 10g Release 2' is not valid for the type xsd: <data type > | BMC Atrium CMDB | Check the data type of the mapped fields in the Salesforce Upsert step. |
Duplicate LAN Endpoint entries are created in BMC Remedyforce CMDB. | BMC Client Management | In BMC Remedyforce Summer 15 Patch 1 and earlier versions, when you imported LAN Endpoint data from BMC Client Management, duplicate LAN endpoint entries might have been created in BMC Remedyforce CMDB. This duplicate data was created because of the following factors:
In BMC Remedyforce Summer 15 Patch 1 and later versions, instead of the Network Interface IP address, the MAC address is used as the unique source identifier for the LAN endpoints imported from BMC Client Management, and new duplicate LAN endpoint entries are not created in BMC Remedyforce CMDB. To resolve this issue, you must install the latest Pentaho package for BMC Client Management released with BMC Remedyforce Summer 15 Patch 1 or later. Also, you must manually delete any existing duplicate LAN endpoint entry on the Remedyforce CMDB. For more information, see Deleting configuration items. |
The HTTP Status 404 error is displayed when trying to connect to Salesforce. | All | Ensure that your Pentaho transformations connect to https://login.salesforce.com/services/Soap/u/<API version> instead of https://www.salesforce.com/services/Soap/u/<API version>. Starting from January 1, 2016, Salesforce retired www.salesforce.com as an API endpoint. For more information, see the announcement on BMC Communities website. To view a video demonstration of how to update your Pentaho transformations, see Salesforce API Endpoint Retirement. BMC Remedyforce has updated the API endpoint in the Pentaho packages that are currently available on the BMC Communities website. |
Error connecting to your Salesforce organization when using version 5.4 of the Pentaho Data Integration Tool. | All | Check whether the Require TLS 1.1 or higher for HTTPS connections Salesforce critical update is enabled in your organization. Pentaho Data Integration Tool 5.4 does not support TLS 1.1 and cannot connect to your Salesforce organization if this Salesforce critical update is enabled in the organization. To resolve this issue, perform one of the following actions:
|
Error running the Pentaho packages. In some cases, you might also not be able to open the Enter Mapping window by clicking Edit Mapping in the Salesforce Upsert window. | All | In the Pentaho packages provided by BMC Remedyforce, ensure that the API version in the Salesforce Webservice URL field in steps, such as Salesforce Upsert, is supported.
|
In the Salesforce Upsert window, when you click Edit Mapping, the following error message is displayed:
| All | Perform the following steps to resolve this issue: |
Related topics
Applying models while importing CIs and assets
Scheduling jobs to import data
Comments
Log in or register to comment.