Importing data from BMC Client Management by using the REST Web API

You can import data from BMC Client Management (known before version 12.0 as BMC FootPrints Asset Core) into CMDB 2.0 by using the Pentaho Data Integration tool. The Pentaho packages for importing data from BMC Client Management are available on the BMC Communities website. For information about how configuration items (CIs) are imported and relationships are created in CMDB 2.0, see Overview-of-how-data-is-imported-into-BMC-Remedyforce.

The following topics provide information about the Pentaho package for importing data by using the Representational State Transfer (REST) web API. For information about the REST web API, see the BMC Client Management Help.

For information about importing data by accessing the BMC Client Management database, see Importing-data-from-BMC-Client-Management-by-accessing-the-database.

Note

The Pentaho package for importing data by using the REST web API supports BMC Client Management and BMC Client Management On-demand versions 12.0 and later. If you integrate Remedyforce with BMC Client Management On-demand 12.0 or later, you must import data by using the FootPrints Asset Core WebServices Integration Package CMDB 2.0 only.

However, if you integrate by using Remedyforce agentless discovery and/or own the Premium and Premium Plus license, you are provided with a new inbuilt integration method that does not use Pentaho packages. Note that when you use any one of the two integration methods, the other method will be disabled.

For more information about the integration methods and BMC Client Management versions that BMC Helix Remedyforce supports, see Remedyforce and Client Management - Original vs. Newer Integrations and Supported-browsers-mobile-devices-and-integrations.

Before you begin

Before you can import data from BMC Client Management by using the REST web API, you must perform the following tasks:

Install the JRE.
Download the Pentaho Data Integration tool.
Warning
Note
If you are using the FootPrints Asset Core WebServices Integration Package CMDB 2.0, use version 6.0 of the Pentaho Data Integration tool.
Register at the BMC Communities website.
Enable the REST web API on the BMC Client Management server.
Configure the BMC Remedyforce external integration on the BMC Client Management server.
Install an SSL certificate on the BMC Client Management server.
(Optional)Configure BMC Remedyforce to retain the original Instance Name value of CIs imported from an external source.
If you have enabled the setting to access your Salesforce organization from limited IP addresses, ensure that you have the security token to log on to your Salesforce organization.
For more information about the security token, see Salesforce Help.
Download the Pentaho package from the BMC Communities website (https://communities.bmc.com/docs/DOC-32929).
For information about the KJB (job) and KTR (transformation) files that are included in the Pentaho package, see KJB files for importing data from BMC Client Management.
Success
Best practice
We recommend that you do not change the folder structure because doing so can interfere with the running of the batch files provided in the CombinedJobs folder.

To import data from BMC Client Management

Important

The job (KJB) files in the Pentaho package send a REST web API request to the BMC Client Management server to initiate the export of data to a .csv file. Each job file contains the list of BMC Client Management fields that are available by default in the .csv file.

If you add, delete, or change the order of fields in the .csv file, you must update the Pentaho jobs and map the fields and the order in which they are listed to the .csv file. Otherwise, the data that you import might be corrupted or the import might not complete successfully. For more information, see Data that is imported from BMC Client Management is corrupted.

To launch the Pentaho Data Integration tool, perform the following actions:
1. Navigate to the location where you downloaded and unzipped the Pentaho Data Integration tool.
2. Navigate to the data-integration folder and double-click the Spoon.bat file.
In Pentaho Spoon, select File > Open, navigate to the folder where you downloaded the Pentaho package, and open the KJB (job) file for the CI type that you want to import.
For example, to import computer system CIs, open the TransferAssetCoreCSinfotoCMDB.kjb file.
From the Edit menu, select Settings.
In the Job properties window, click the Parameters tab.
Modify the Default value column for the different parameters.

Click here to see the parameters.

The various transformations and steps in the job use the values that you specify on the Parameters tab. If you do not specify values for the SFDC_USER and SFDC_PASSWORD parameters, you must provide the user name and password of your Salesforce organization in the Salesforce Input[CMDB_Class] and Upsert Into Class <CI type name> steps.

To access these steps, in the job (KJB) file, right-click the Import Records in Remedyforce CMDB step, and select Open Referenced Object> Transformation.
Warning
Note
If you are running a combined job, you can specify the parameter values only in the combined job (for example, TransferAssetCoreComputerSystemAndProductinfotoCMDB.kjb). Each job that is included in the combined job uses the parameter values that you specify in the combined job. However, if you specify parameter values in the combined and individual jobs, values specified in the individual jobs take precedence.
(Optional) Update the mapping between the data from BMC Client Management and CMDB (fields in the Base Element object).
Click here to see the detailed steps.
1. Double-click the Upsert Into Class <CI type name> step.
2. (Optional) In the Salesforce Upsert window, enter the user name and password of your Salesforce organization.
  Based on your requirements, you can replace the Salesforce user name and password with the SFDC_USER and SFDC_PASSWORD parameters.
3. In the Fields area, click Edit Mapping.
4. In the Enter Mapping window, update mappings for fields based on your requirements.
  For information about updating the out-of-the-box mapping, see Field mapping in CMDB 2.0.
5. To save your settings and close the Enter Mapping window, click OK.
6. To save your changes and close the Salesforce Upsert window, click OK.
To save the KTR and KJB files, click Save.
In the KJB file tab, click Run this job.
Perform one of the following actions based on the Pentaho version that you are using:
Pentaho version
Action
6.1
In the Run Options window, click Run.
6.0
In the Execute a job window, click Launch.
Data is imported into the appropriate CMDB class. The first time you run the job, all CIs are imported into the appropriate CMDB class. When you rerun the job, only the new and modified CIs are imported into the CMDB class.

Transformation status is depicted by using the following icons:
- — Complete
- — Running
- — Unsuccessful
(Optional) To view logs, in the Execution results section, click the Logging tab.
All the errors are displayed in red.
For the imported records, the Source field is set to BCM. You can use the Source field to generate reports and also when you enable and configure reconciliation.

Pentaho version	Action
6.1	In the Run Options window, click Run.
6.0	In the Execute a job window, click Launch.

To schedule jobs to import data

On the BMC Communities website, the Pentaho packages for importing data from external data sources into page contain a SchedulingFiles folder. This folder contains a sample batch file that you can use as a base to create your batch files for scheduling data import. For more information about scheduling a job, see http://wiki.Pentaho.com/display/EAI/Kitchen+User+Documentation.

Note

To understand the job scheduling process, refer to the following procedure. This procedure is an example to be used as a reference for all other packages.

To schedule the Remedyforce LDAP Pentaho package to import users, perform the following steps:

(For all other required packages, perform these steps by using the respective batch files.)

Navigate to the Remedyforce Package folder.
Rename your extracted package folder to SchedulingFiles.
Open the folder where you have installed the Pentaho Data Integration tool and minimize this folder.
Open the SchedulingFiles folder.
The folder contains two .bat files.
To create a copy of the batch file, copy SchedulingTransferLDAPInfo.bat file and paste it in the same folder.
Right-click the SchedulingTransferLDAPInfo.bat file and click Edit.
The SchedulingTransferLDAPInfo.bat notepad opens.
Minimize the notepad and open the Pentaho data-integration folder as mentioned in Step 3.
Copy the data-integration folder address and navigate to the notepad as mentioned in Step 6.
The following figure shows an example of the updated SchedulingTransferLDAPInfo.bat file.
Before the first line of the notepad, type cd and paste the copied data-integration folder address . (Make sure to add a space betweeen cd and the folder address).
Paste the data-integration folder address in the Kettle Home folder address and remove the double quotes ("").
Paste the data-integration folder address in the kitchen.bat file address by replacing only the part before \kitchen.bat.
Navigate to the Remedyforce Package folder and copy the address for TransferLDAPInfo.kjb.
Paste the TransferLDAPInfo.kjb address in between \kitchen.bat /file: and /level:Detailed. Remove the double quotes ("") and any additional spaces between the address.
Save the notepad.

To prepare for scheduling

Open the Command Prompt as an administrator.
Type cd and then paste the SchedulingFiles folder address with a space in between the two.
Copy and paste the .bat file name (SchedulingTransferLDAPInfo.bat).
The job starts running.

To schedule the job

Go to Start > Control Panel > Administrative Tools > Task Scheduler.
In the Actions section, click Create Task.
In the General tab, enter the name and description.
Select the Run with highest privileges check box, and click OK.
On the Triggers tab, click New, select the appropriate option for starting the task, and click OK.
For example, you can select Daily or When the computer starts as the option for starting the task.
On the Actions tab, click New.
Click Browse, select SchedulingTransferLDAPInfo.bat, and click OK.

KJB files for importing data from BMC Client Management

The Pentaho package contains job (KJB) and transformation (KTR) files that are created in the Pentaho Data Integration tool. The KJB files (jobs) and KTR files (transformations) store metadata in XML format. A KJB file contains a series of transformations that are run in a sequence. A KTR file contains a single transformation. The KJB (job) files retrieve data from external sources into BMC Remedyforce CMDB.

For more information, see KJB files in the Pentaho package and Steps in the KJB files for importing data from BMC Client Management.

KJB files in the Pentaho package

Each Pentaho package includes different KJB (job) files for importing different types of configuration items (CIs), such as computer systems and LAN endpoints. These job files are bundled with related files and provided to you in folders. The folder names correspond to CI types in BMC Remedyforce CMDB.

The following table provides information about the job files that you must run to import specific CI types into BMC Remedyforce CMDB:

Steps in the KJB files for importing data from BMC Client Management

All job (KJB) files for importing data from specific CI types include the series of steps and transformations that are listed in the following table.

Troubleshooting

The following topics provide troubleshooting tips to resolve issues that might occur when you import data from BMC Client Management:

Mappings cannot be updated from the Salesforce Upsert window

In the Salesforce Upsert window, the drop-down fields listed in the Module field column in the Fields area might not display any values. In this case, you must update the mapping between the data from BMC Client Management and BMC Remedyforce CMDB by using the Edit Mapping option.

Data that is imported from BMC Client Management is corrupted

If the data imported from BMC Client Management is corrupted or the import does not complete successfully, the .csv file that contains the data exported by BMC Client Management might have been modified. You might have added, deleted, or changed the order of fields in the .csv file. To resolve this issue, you must update the Pentaho jobs and map the fields and the order in which they are listed to the .csv file.

Perform the following actions to fix the corrupted data that is imported from BMC Client Management:

Navigate to the location where you downloaded and unzipped the Pentaho Data Integration tool.
To launch the Pentaho Data Integration tool, navigate to the data-integration folder.
Double-click the Spoon.bat file and open the job (KJB) file that corresponds to the .csv file that you have updated.
For example, if you have updated fields in the .csv file for computer system CIs, open the TransferAssetCoreCSinfotoCMDB.kjb file.
In the job (KJB) file, right-click the Import Records in Remedyforce CMDB step, and select Open Referenced Object > Transformation.
In the transformation (KTR) file that is opened, double-click the CSV file input step.
In the CSV Input window, map the fields and the order in which they are listed to the .csv file that contains data from the BMC Client Management server.
You can right-click a field and select an option such as Move up and Insert before this row.
To save your changes, click OK.
To save the KTR and KJB files, click .

Troubleshooting common issues when importing data

The following table describes the troubleshooting tips that you can use to resolve common issues that you might face when importing data.

Error or issue	Applies to the Pentaho package for	Description or procedure
While importing LAN endpoint records from BMC Client Management, the import process throws an exception.	BMC Client Management	If the network interface information is available for LAN endpoint class records, they are imported from BMC Client Management. LAN endpoint class records without the network interface information are filtered out during the import process and are not inserted into Remedyforce. If you install Remedyforce 20.22.02 or upgrade to it, this feature is available by default. Administrators need not configure any settings. If you use Pentaho packages for importing data, you must download the latest version of the following packages from BMC Communities: BMC FootPrints Asset Core CMDB 2.0 for BMC Client Management 12.0: click here to download. BMC FootPrints Asset Core WebServices CMDB 2.0: click here to download. For out-of-the-box Pentaho packages, you do not have to configure any settings. For customized Pentaho packages, you must add the following filter: Job: TransferAssetCoreLANEndPointinfotoCMDB.kjb Transformation: TransferAssetCoreLANEndPointinfotoCMDB.ktr Step: Filter rows For details about the filter, download and refer to the Pentaho packages.
Viewing logs of the import jobs	All	Log files are created in the folder where you have saved the KJB files. Success and failure rows files are also created in the same folder. If a failure occurs, the error code and its description are provided in the failure row file. You can also use the failure row file to import data to Salesforce.
Not all records are imported	All	Delete the delta time stamp file of the job that you ran, and run the job again.
Import fails	All	If you have upgraded to Remedyforce Winter 17 (20.17.01), either you use the latest Pentaho packages updated on the BMC Communities or map a value to the Source field of the Base Element object. Also, ensure that you have Edit permission on the Source field of the Base Element object.
An "out of memory" error occurs	All	While importing a large number of records, if you get the OutofMemoryError or Java heap size error message, increase the heap size in the Spoon.bat file. Navigate to the location where you downloaded and unzipped the Pentaho Data Integration tool. Navigate to the data-integration folder. Right-click the Spoon.bat file, and select Edit. Locate the following line and replace 512 with a higher value: If "%PENTAHO_DI_JAVA_OPTIONS%"=="" set PENTAHO_DI_JAVA_OPTIONS= "-Xmx512m" "-XX:MaxPermSize=256m" Click Save. Relaunch the Spoon.bat file and rerun the job file.
A "too many script statements" error occurs	All	When you run an import, if you receive "Too many script statements" as an onscreen Apex error message or in an email, reduce the batch size by 10 in the Batch Size field in the Settings section on the Salesforce Upsert window.
Records are not being to your CMDB.	All	Raise a case with Salesforce to create a custom index on the Assembly ID field of the Base Element object.
Importing users by using the failure rows file for LDAP	LDAP server	A failure row file is a text file that is saved in the folder where you have saved your KJB files. Perform the necessary steps to remove the error provided for the failure rows in the failure rows file, and then import the data by using the following steps. The following Pentaho package provides a transformation file in the FailureRowsInput folder. You can use this transformation file for importing users from the failure rows file to the Salesforce organization: LDAP integration with Remedyforce (assign permission sets and Remedyforce managed package license): https://community.bmc.com/s/news/aA33n000000TPBRCA4/importing-users-from-an-ldap-server To import data from the failure rows file Double-click and open the FailureRowsInput folder. Double-click and open the LDAP_FAILURE_ROWS_ <YYYYMMDDHHMMSS> file. Where *YYYYMMDDHHMMSS* is the time stamp of the file. The first line in the file displays the header row and from the second line onward details of the failed import are displayed. For each failed import record: Read the information under the headings Error Description, Error Fields, and Error Codes. Resolve the error. Open the FailureLDAP transformation file with the Spoon batch file of the Pentaho Data Integration tool. Double-click the Salesforce Upsert step, and enter your Salesforce organization username and password in the Connection section. (Optional) To verify the connection, click Test connection. Click OK. To save the FailureLDAP transformation file, click . Click . In the Execute a transformation window, click Launch.
Failed in writeToSalesForce: java.lang.IllegalArgumentException: The char '0x0' after 'Print' is not a valid XML character	All	Some unicode characters that cannot be parsed by the XML parser are present in any of the mapped fields. Either delete these characters in data or delete the mapping of such fields.
Unable to query Salesforce	All	Check your Salesforce organization credentials in the Salesforce Upsert and Salesforce Input(CMDB_Classes) steps.
The job does not appear in Atrium Integrator.	BMC Atrium CMDB	Ensure that you have saved your KTR and KJB files in a folder in Atrium Integrator.
Error setting value #2 [ARDelta_1 Integer] on prepared statement (Integer)	BMC Atrium CMDB	This error is generated if multiple records are created in the BMC Remedy AR System NGIE:Delta form for the running transformation. Delete the additional ARDelta entries that are created for the error transformation file by deleting the records. To find the duplicate records for a transformation file, open the NGIE:Delta form in BMC Remedy AR System. In the TransName field, enter the transformation file name, and click Search.
Did not find Remedy Application Service password for server<server name> in UDM:RAppPassword Form on server <server name>	BMC Atrium CMDB	Check your Atrium Server Connection credentials in the ARInput step.
'Oracle Database Server 10g Release 2' is not valid for the type xsd: <data type >	BMC Atrium CMDB	Check the data type of the mapped fields in the Salesforce Upsert step.
Duplicate LAN Endpoint entries are created in Remedyforce CMDB.	BMC Client Management	In Remedyforce Summer 15 Patch 1 and earlier versions, when you imported LAN Endpoint data from BMC Client Management, duplicate LAN endpoint entries might have been created in Remedyforce CMDB. This duplicate data was created because of the following factors: In BMC Client Management, a LAN endpoint could periodically obtain multiple IP addresses using DHCP. In Remedyforce Summer 15 and earlier, the Network Interface IP address was used as the unique source identifier for LAN endpoints imported from BMC Client Management. In Remedyforce Summer 15 Patch 1 and later versions, instead of the Network Interface IP address, the MAC address is used as the unique source identifier for the LAN endpoints imported from BMC Client Management, and new duplicate LAN endpoint entries are not created in BMC Remedyforce CMDB. To resolve this issue, you must install the latest Pentaho package for BMC Client Management released with Remedyforce Summer 15 Patch 1 or later. Also, you must manually delete any existing duplicate LAN endpoint entry on the Remedyforce CMDB. For more information, see Deleting-configuration-items.
The HTTP Status 404 error is displayed when trying to connect to Salesforce.	All	Ensure that your Pentaho transformations connect to *https://login.salesforce.com/services/Soap/u/<API version>* instead of *https://www.salesforce.com/services/Soap/u/<API version>. Starting from January 1, 2016, Salesforce retired www.salesforce.com* as an API endpoint. For more information, see the announcement on BMC Communities website. To view a video demonstration of how to update your Pentaho transformations, see Salesforce API Endpoint Retirement. Remedyforce has updated the API endpoint in the Pentaho packages that are currently available on the BMC Communities website.
Error connecting to your Salesforce organization when using version 5.4 of the Pentaho Data Integration Tool.	All	Check whether the Require TLS 1.1 or higher for HTTPS connections Salesforce critical update is enabled in your organization. Pentaho Data Integration Tool 5.4 does not support TLS 1.1 and cannot connect to your Salesforce organization if this Salesforce critical update is enabled in the organization. To resolve this issue, perform one of the following actions: Use version 6.1 of the Pentaho Data Integration Tool to import data from various data sources to Remedyforce. Deactivate the Require TLS 1.1 or higher for HTTPS connections Salesforce critical update in your Salesforce organization. For information about when the critical update will be automatically activated, see Salesforce Knowledge Article Number 000232871.
Error running the Pentaho packages. In some cases, you might also not be able to open the Enter Mapping window by clicking Edit Mapping in the Salesforce Upsert window.	All	In the Pentaho packages provided by Remedyforce, ensure that the API version in the Salesforce Webservice URL field in steps, such as Salesforce Upsert, is supported. In the LDAP Pentaho packages, the default API version in the Salesforce Webservice URL field is 51 (https://login.salesforce.com/services/Soap/u/51). In all other Pentaho packages, the default API version in the Salesforce Webservice URL field is 51 (https://login.salesforce.com/services/Soap/u/51).
In the Salesforce Upsert window, when you click Edit Mapping, the following error message is displayed: Certain referenced fields were not found!	All	Perform the following steps to resolve this issue: In the Salesforce Upsert window, click Edit Mapping. The Certain referenced fields were not found! error message is displayed. To open the Enter Mapping window, click OK. The tool removes all referenced fields from the existing mappings. The Pentaho package provided by Remedyforce includes only one referenced field, BMCServiceDesk__CMDB_Class__c. The CMDB_Class field is no longer available in the Mappings column. In the Source fields column, select CMDB_Class and click Add. When you move CMDB_Class from the Source field column to the Mappings column, the c in the field name is replaced with r. For example, BMCServiceDesk__CMDB_Class__c is replaced with BMCServiceDesk__CMDB_Class__r. (Optional) Update mappings for other fields based on your requirements. For information about updating the out-of-the-box mapping, see Field mapping in CMDB 2.0. To save your settings and close the Enter Mapping window, click OK. In the Salesforce Upsert window, in the Module field column in the Fields area, click BMCServiceDesk__CMDB_Class__r, and replace r with c. To save your changes and close the Salesforce Upsert window, click OK.