Importing data from BMC Client Management by using the REST Web API
You can import data from BMC Client Management (known before version 12.0 as BMC FootPrints Asset Core) into BMC Remedyforce CMDB 2.0 by using the Pentaho Data Integration tool. The Pentaho packages for importing data from BMC Client Management are available on the BMC Communities website. For information about how configuration items (CIs) are imported and relationships are created in BMC Remedyforce CMDB 2.0, see Overview of how data is imported into BMC Remedyforce.
The following topics provide information about the Pentaho package for importing data by using the Representational State Transfer (REST) web API. For information about the REST web API, see the BMC Client Management Help.
For information about importing data by accessing the BMC Client Management database, see Importing data from BMC Client Management by accessing the database.
Important
The Pentaho package for importing data by using the REST web API supports BMC Client Management and BMC Client Management OnDemand versions 12.0 and later. If you have configured the integration with BMC Client Management OnDemand 12.0 or later, you must import data by using this Pentaho package only.
For information about the BMC Client Management versions that BMC Remedyforce supports, see Supported browsers, mobile devices, and integrations.
Before you begin
Before you can import data from BMC Client Management by using the REST web API, you must perform the following tasks:
- Install the JRE.
Download the Pentaho Data Integration tool.
Note
If you are using the FootPrints Asset Core WebServices Integration Package CMDB 2.0, use version 6.0 of the Pentaho Data Integration tool.
- Register at the BMC Communities website.
- Enable the REST web API on the BMC Client Management server.
- Configure the BMC Remedyforce external integration on the BMC Client Management server.
- Install an SSL certificate on the BMC Client Management server.
- (Optional)Configure BMC Remedyforce to retain the original Instance Name value of CIs imported from an external source.
- If you have enabled the setting to access your Salesforce organization from limited IP addresses, ensure that you have the security token to log on to your Salesforce organization.
For more information about the security token, see Salesforce Help. Download the Pentaho package from the BMC Communities website (https://communities.bmc.com/docs/DOC-32929).
For information about the KJB (job) and KTR (transformation) files that are included in the Pentaho package, see KJB files for importing data from BMC Client Management.Best practice
BMC recommends that you do not change the folder structure because doing so can interfere with the running of the batch files provided in the CombinedJobs folder.
To import data from BMC Client Management
Important
The job (KJB) files in the Pentaho package send a REST web API request to the BMC Client Management server to initiate the export of data to a .csv file. Each job file contains the list of BMC Client Management fields that are available by default in the .csv file.
If you add, delete, or change the order of fields in the .csv file, you must update the Pentaho jobs and map the fields and the order in which they are listed to the .csv file. Otherwise, the data that you import might be corrupted or the import might not complete successfully. For more information, see Data that is imported from BMC Client Management is corrupted.
- To launch the Pentaho Data Integration tool, perform the following actions:
Navigate to the location where you downloaded and unzipped the Pentaho Data Integration tool.
Navigate to the data-integration folder and double-click the Spoon.bat file.
- In Pentaho Spoon, select File > Open, navigate to the folder where you downloaded the Pentaho package, and open the KJB (job) file for the CI type that you want to import.
For example, to import computer system CIs, open the TransferAssetCoreCSinfotoCMDB.kjb file. - From the Edit menu, select Settings.
In the Job properties window, click the Parameters tab.
Modify the Default value column for the different parameters.
(Optional) Update the mapping between the data from BMC Client Management and BMC Remedyforce CMDB (fields in the
Base Element
object).- To save the KTR and KJB files, click Save .
- In the KJB file tab, click Run this job .
Perform one of the following actions based on the Pentaho version that you are using:
Pentaho version Action 6.1 In the Run Options window, click Run. 6.0 In the Execute a job window, click Launch. Data is imported into the appropriate CMDB class. The first time you run the job, all CIs are imported into the appropriate CMDB class. When you rerun the job, only the new and modified CIs are imported into the CMDB class.
Transformation status is depicted by using the following icons:- — Complete
- — Running
- — Unsuccessful
- (Optional) To view logs, in the Execution results section, click the Logging tab.
All the errors are displayed in red.
For the imported records, the Source field is set to BCM. You can use the Source field to generate reports and also when you enable and configure reconciliation.
To schedule jobs to import data
On the BMC Communities website, the Pentaho packages for importing data from external data sources into BMC Remedyforce page contain a Scheduling Files folder. This folder contains a sample batch file that you can use as a base to create your batch files for scheduling data import. For more information about scheduling a job, see http://wiki.Pentaho.com/display/EAI/Kitchen+User+Documentation.
- Navigate to the folder where you have extracted the Pentaho package for which you want to schedule import jobs.
- Open the Scheduling Files folder and create a copy of the sample batch file, such as SchedulingComputerSystem.bat.
- Open the batch file that you created in step 2, perform the following actions, and save the file:
- Update the path of the Kitchen.bat file based on your version of the Pentaho Data Integration tool:
KETTLE_HOME="<path>\<PentahoFolder>\data-integration"
- Update the path of the Kitchen.bat file and the KJB file:
<path>\
<PentahoFolder>
\data-integration\kitchen.bat /file:"<JobFile>\<JobFileName>.kjb"/level:Detailed
- <path> is the location where you have extracted the Pentaho installer ZIP file.
- <PentahoFolder> is the extracted root folder for your version of the Pentaho Data Integration tool.
For example, for version 6.1 of the Pentaho Data Integration Tool, the extracted root folder ispdi-ce-6.1.0.1-196.
- <JobFile> is the location where you have extracted the job files.
- <JobFileName> is the KJB file that you want to run at the scheduled interval.
- The level option specifies the level of logging for the job. The values that you can specify for the level option are Minimal, Basic, Detailed, Debugging, and Row Level. The log files are saved in the same folder where KJB files are saved.
- Update the path of the Kitchen.bat file based on your version of the Pentaho Data Integration tool:
- Go to Start > Control Panel > Administrative Tools > TaskScheduler.
- In the Task Scheduler window, in the Actions area, click Create Basic Task.
- In the Create Basic Task Wizard, type the name and description of the task, and click Next.
- Select the appropriate option for starting the task, and click Next.
For example, you can select Daily or When the computer starts as the option for starting the task. - Configure the additional options for starting the task, and click Next.
The additional options are displayed based on the option that you selected in step 7. For example, if you select the Daily option for starting the task, you must configure the start date and frequency at which you want to run the batch file. - Select the Start a program radio button, and click Next.
- Click Browse to locate the batch file that you have created to schedule the job, and click Next.
- Select the Open the Properties dialog for this task when I click Finish check box, and click Finish.
KJB files for importing data from BMC Client Management
The Pentaho package contains job (KJB) and transformation (KTR) files that are created in the Pentaho Data Integration tool. The KJB files (jobs) and KTR files (transformations) store metadata in XML format. A KJB file contains a series of transformations that are run in a sequence. A KTR file contains a single transformation. The KJB (job) files retrieve data from external sources into BMC Remedyforce CMDB.
For more information, see KJB files in the Pentaho package and Steps in the KJB files for importing data from BMC Client Management.
KJB files in the Pentaho package
Each Pentaho package includes different KJB (job) files for importing different types of configuration items (CIs), such as computer systems and LAN endpoints. These job files are bundled with related files and provided to you in folders. The folder names correspond to CI types in BMC Remedyforce CMDB.
The following table provides information about the job files that you must run to import specific CI types into BMC Remedyforce CMDB:
Folder name | Job file | Description |
---|---|---|
Computer System | TransferAssetCoreCSinfotoCMDB.kjb | Transfers information about the |
LANEndPoints | TransferAssetCoreLANEndPointInfoto | Transfers information about the |
OS | TransferAssetCoreOSinfotoCMDB.kjb | Transfers information about the |
Processor | TransferAssetCoreProcessorInfoto | Transfers information about the |
Product | TransferAssetCoreProductInfotoCMDB.kjb | Transfers information about the Note: Depending on the organization size, |
Combined Jobs |
| The Combined Jobs folder contains the following job files:
|
Steps in the KJB files for importing data from BMC Client Management
All job (KJB) files for importing data from specific CI types include the series of steps and transformations that are listed in the following table.
Step | Description |
---|---|
Start | Starts the job to import data from BMC Client Management by using the REST web API. |
Check If Delta Timestamp File exists | Checks for a time stamp file. The Pentaho package utilizes a time stamp file to determine which records have been added or modified since the last time the job was run. This step is used for incremental import. If a time stamp file does not exist, the Create the initial timestamp file step is run. Otherwise, the Send Request for Export step is run. |
Create the initial timestamp file | Creates the time stamp file to record the time of import. This step is run only if you are importing for the first time or have deleted the existing time stamp file. |
Send Request For Export | Sends a REST web API request to the BMC Client Management server to initiate the export of data to a .zip file. The response from the BMC Client Management server provides either the location and name of the .zip file or an error code. |
Check Request REST Response | Checks whether the request for export was successful. If the request for export was successful, the Download File step is run. Otherwise, the Log Request Error from AC step is run. |
Log Request Error From AC | Logs an error if the response from the BMC Client Management server includes an error code. |
Download File | Downloads the .zip file that is stored on the BMC Client Management server. This file contains the data to be imported in BMC Remedyforce and is downloaded to the folder where you have saved the KJB files. If the .zip file is not available, this step is run again after the specified time interval, which is 10 seconds by default. The default number of times this step is rerun is 3. You can change the default values based on your requirements. |
Is File Downloaded | Checks whether the .zip file is downloaded to the computer on which the Pentaho package is running. |
Write File Not Available | Creates an error message file if the .zip file is not available for download even after the final run of the Download File step. |
Check Download REST Response | Checks for an error in downloading the file from the BMC Client Management server. If the file is successfully downloaded, the Import Records in Remedyforce CMDB step is run. Otherwise, the Log Download Error From AC step is run. |
Log Download Error From AC | Logs an error if the file is not downloaded successfully from the BMC Client Management server. |
Import Records In Remedyforce CMDB | Extracts records from the .zip file and imports data in BMC Remedyforce CMDB based on the specified mapping. |
Delete File From Server | Sends a request to the BMC Client Management server to delete the .zip file. |
Check Delete REST Response | Checks whether the file has been deleted successfully from the BMC Client Management server. If the file is not deleted successfully, the Log Download Error From AC step is run. |
Log Download Error From AC | Logs an error if the file is not deleted successfully from the BMC Client Management server. |
Store the current time stamp | If the data is successfully imported, stores the time at which the data was imported to BMC Remedyforce. |
Troubleshooting
The following topics provide troubleshooting tips to resolve issues that might occur when you import data from BMC Client Management:
Mappings cannot be updated from the Salesforce Upsert window
In the Salesforce Upsert window, the drop-down fields listed in the Module field column in the Fields area might not display any values. In this case, you must update the mapping between the data from BMC Client Management and BMC Remedyforce CMDB by using the Edit Mapping option.
Data that is imported from BMC Client Management is corrupted
The job (KJB) files in the Pentaho package send a REST web API request to the BMC Client Management server to initiate the export of data to a .csv file. Each job file contains the list of BMC Client Management fields that are available by default in the .csv file.
If the data imported from BMC Client Management is corrupted or the import does not complete successfully, the .csv file that contains the data exported by BMC Client Management might have been modified. You might have added, deleted, or changed the order of fields in the .csv file. To resolve this issue, you must update the Pentaho jobs and map the fields and the order in which they are listed to the .csv file.
Perform the following actions to fix the corrupted data that is imported from BMC Client Management:
- Navigate to the location where you downloaded and unzipped the Pentaho Data Integration tool.
- To launch the Pentaho Data Integration tool, navigate to the data-integration folder.
- Double-click the Spoon.bat file and open the job (KJB) file that corresponds to the .csv file that you have updated.
For example, if you have updated fields in the .csv file for computer system CIs, open the TransferAssetCoreCSinfotoCMDB.kjb file. - In the job (KJB) file, right-click the Import Records in Remedyforce CMDB step, and select Open Referenced Object > Transformation.
- In the transformation (KTR) file that is opened, double-click the CSV file input step.
- In the CSV Input window, map the fields and the order in which they are listed to the .csv file that contains data from the BMC Client Management server.
- You can right-click a field and select an option such as Move up and Insert before this row.
- To save your changes, click OK.
- To save the KTR and KJB files, click .
Troubleshooting common issues when importing data
The following table describes the troubleshooting tips that you can use to resolve common issues that you might face when importing data.
Error or issue | Applies to the Pentaho package for | Description or procedure |
---|---|---|
Viewing logs of the import jobs | All | Log files are created in the folder where you have saved the KJB files. Success and failure rows files are also created in the same folder. If a failure occurs, the error code and its description are provided in the failure row file. You can also use the failure row file to import data to Salesforce. |
Not all records are imported | All | Delete the delta time stamp file of the job that you ran, and run the job again. |
Import fails | All | If you have upgraded to BMC Remedyforce Winter 17 (20.17.01), either you use the latest Pentaho packages updated on the BMC Communities or map a value to the Source field of the Base Element object. Also, ensure that you have Edit permission on the Source field of the Base Element object. |
An "out of memory" error occurs | All | While importing a large number of records, if you get the
|
A "too many script statements" error occurs | All | When you run an import, if you receive "Too many script statements" as an onscreen Apex error message or in an email, reduce the batch size by 10 in the Batch Size field in the Settings section on the Salesforce Upsert window. |
Records are not being to your CMDB. | All | Raise a case with Salesforce to create a custom index on the Assembly ID field of the Base Element object. |
Importing users by using the failure rows file for LDAP | LDAP server | A failure row file is a text file that is saved in the folder where you have saved your KJB files. Perform the necessary steps to remove the error provided for the failure rows in the failure rows file, and then import the data by using the following steps. The following Pentaho packages provide a transformation file in the FailureRowsInput folder. You can use this transformation file for importing users from the failure rows file to the Salesforce organization.
To import data from the failure rows file
|
Failed in writeToSalesForce: java.lang.IllegalArgumentException: The char '0x0' after 'Print' is not a valid XML character | All | Some unicode characters that cannot be parsed by the XML parser are present in any of the mapped fields. Either delete these characters in data or delete the mapping of such fields. |
Unable to query Salesforce | All | Check your Salesforce organization credentials in the Salesforce Upsert and Salesforce Input(CMDB_Classes) steps. |
The job does not appear in Atrium Integrator. | BMC Atrium CMDB | Ensure that you have saved your KTR and KJB files in a folder in Atrium Integrator. |
Error setting value #2 [ARDelta_1 Integer] on prepared statement (Integer) | BMC Atrium CMDB | This error is generated if multiple records are created in the BMC Remedy AR System NGIE:Delta form for the running transformation. Delete the additional ARDelta entries that are created for the error transformation file by deleting the records. To find the duplicate records for a transformation file, open the NGIE:Delta form in BMC Remedy AR System. In the TransName field, enter the transformation file name, and click Search. |
Did not find Remedy Application Service password for server<server name> in UDM:RAppPassword Form on server <server name> | BMC Atrium CMDB | Check your Atrium Server Connection credentials in the ARInput step. |
'Oracle Database Server 10g Release 2' is not valid for the type xsd: <data type > | BMC Atrium CMDB | Check the data type of the mapped fields in the Salesforce Upsert step. |
Duplicate LAN Endpoint entries are created in BMC Remedyforce CMDB. | BMC Client Management | In BMC Remedyforce Summer 15 Patch 1 and earlier versions, when you imported LAN Endpoint data from BMC Client Management, duplicate LAN endpoint entries might have been created in BMC Remedyforce CMDB. This duplicate data was created because of the following factors:
In BMC Remedyforce Summer 15 Patch 1 and later versions, instead of the Network Interface IP address, the MAC address is used as the unique source identifier for the LAN endpoints imported from BMC Client Management, and new duplicate LAN endpoint entries are not created in BMC Remedyforce CMDB. To resolve this issue, you must install the latest Pentaho package for BMC Client Management released with BMC Remedyforce Summer 15 Patch 1 or later. Also, you must manually delete any existing duplicate LAN endpoint entry on the Remedyforce CMDB. For more information, see Deleting configuration items. |
The HTTP Status 404 error is displayed when trying to connect to Salesforce. | All | Ensure that your Pentaho transformations connect to https://login.salesforce.com/services/Soap/u/<API version> instead of https://www.salesforce.com/services/Soap/u/<API version>. Starting from January 1, 2016, Salesforce retired www.salesforce.com as an API endpoint. For more information, see the announcement on BMC Communities website. To view a video demonstration of how to update your Pentaho transformations, see Salesforce API Endpoint Retirement. BMC Remedyforce has updated the API endpoint in the Pentaho packages that are currently available on the BMC Communities website. |
Error connecting to your Salesforce organization when using version 5.4 of the Pentaho Data Integration Tool. | All | Check whether the Require TLS 1.1 or higher for HTTPS connections Salesforce critical update is enabled in your organization. Pentaho Data Integration Tool 5.4 does not support TLS 1.1 and cannot connect to your Salesforce organization if this Salesforce critical update is enabled in the organization. To resolve this issue, perform one of the following actions:
|
Error running the Pentaho packages. In some cases, you might also not be able to open the Enter Mapping window by clicking Edit Mapping in the Salesforce Upsert window. | All | In the Pentaho packages provided by BMC Remedyforce, ensure that the API version in the Salesforce Webservice URL field in steps, such as Salesforce Upsert, is supported.
|
In the Salesforce Upsert window, when you click Edit Mapping, the following error message is displayed:
| All | Perform the following steps to resolve this issue: |
Related topics
Applying models while importing CIs and assets
Scheduling jobs to import data
Troubleshooting common issues when importing data
Troubleshooting issues when importing data from BMC Client Management
Comments
Log in or register to comment.