This documentation supports the 22.1 version of Action Request System.
To view an earlier version, select the version from the Product version menu.

Overview of the data load process

You can use the Data Management tool to import data into your BMC Remedy ITSM applications. The section walks you through the activities that you need to perform for loading data. For details, see Configuring Data Management.

Planning activities

Before you start to populate the data management spreadsheets, run through the planning activities that are outlined in this section. Performing these activities up front, ensures that you thoroughly understand the nature of the data load jobs you will be running and their specific needs.

The planning activities discussed in this section includes:

Identifying forms

Data for BMC Remedy ITSM applications is stored on three types of forms, depending on the type of data they contain: 

  • Foundation forms
  • Process setup forms
  • Transactional forms

Before you create a data load job, you need to identify which data form types are involved in the data load. Familiarizing yourself with the types of data that you are loading helps you to better understand the nature of the data load and its need. This kind of familiarity with the data load job is important in later steps. 

Identifying data creation and update use cases

Ensure that you understand the forms and spreadsheets that your data load job will use to:

  • add information to the database, if any.  
  • update information in the database, if any.

You cannot update data from every spreadsheet tab. Some spreadsheet tabs allow you to only add new data. See Data creation and updates using a dataload job for a list of the spreadsheet tabs from which you can only add new data.

Preparing for bulk data loads

Bulk data loads can present troubleshooting and performance challenges when the data quality cannot be guaranteed or if your system resources are not sufficient to run the data load processes. If you are running bulk data loads, keep the following points in mind when planning the data load jobs, to reduce the impact of any issues associated with data quality or performance. 

  • If you are unsure of the quality of the data you are loading, BMC recommends that you load the data in smaller, manageable chunks. Loading bulk data in chunks makes it easier to troubleshoot the data load job if it encounters issues with data quality or formatting during the validate step. You cannot move on to the promote step step until all of the data that you are importing is valid. 

Identifying customizations

The data load spreadsheets that BMC ships with the Data Management feature are designed to work with an out-of-the-box BMC Remedy ITSM system. If you have customized the database for the specific needs of your organization by adding fields to the data forms, you must account for these custom fields when planning the data load jobs. If you don't, the data load job will not pick up the data from the customized fields.   

Important

When using spreadsheets to run a data load job, always place any customized data fields in the columns that come after the out-of-the-box data columns. Do not intermix the customized data columns with the out-of-the-box data columns. For example, if the form you are loading has 10 out-of-the-box fields and two customized fields, on the data load spreadsheet the out-of-the-box forms will appear in columns A through J and the customized data fields must appear in column K and L.

Tip

If you have custom data, always run a test data load job to ensure that the custom data is picked up by the data load job.

Creating a test plan

Always test your dataload jobs before you run them on live data. Create a test plan that takes into account the following considerations:

  • The forms into which you will be loading data 
  • The volume of data you are loading
  • Your system performance and resources (memory, processing speed, network bandwidth, database capabilities, and so on)
  • The number of threads configured for CAI
  • UDM performance on the server

When you finish drafting the test plan, review it with stakeholders and experts on the various system areas that your plan touches on.

Populating the data spreadsheets

When all of the prerequisites are in place and you have completed your test plan, you can start to work with the data load spreadsheets.

Important

When providing source data using, for example, dataload spreadsheets, LDAP source, ARX files, or XML files, BMC Remedy ITSM requires that you provide the following data at a minimum:

Foundation data

Out-of-the-box spreadsheets

Out-of-the-box job templates

Company

Company.xlsx

Company

Organization

PeopleOrg.xlsx

People Organization

Location

Location.xlsx

Location

Support Group

SupportGroup.xlsx

Support Group

People

People.xlsx

People

For instructions about how to work with spreadsheets, see Downloading and creating data load spreadsheets.

Identifying spreadsheets

  • Identifying the spreadsheets for your use case Mapping between Foudation data types and target forms.
  • Using CTM_LoadPeopleModification tab on the People.xlsx spreadsheet: 
    • You can create a template with all support group associations and permissions that you would like to give a user or a login id. Specify the newly created template name and login id in the spreadsheet to load those support group associations or user permissions groups. Update the Group list in the BMC Remedy AR System User form with the  support group associations and permissions.
    • deadlock in the database happens if same person or login ID is repeated in a spreadsheet and the duplicate records are processed in different threads or processes in the database. To prevent the deadlock in the database, BMC recommends you to have one template with all support group associations, people permissions, and functional roles to apply to all users in one job. If multiple templates need to be added to a single person, divide the job per template rather than dividing a job per user. 

Identifying templates

Using the templates that BMC provides with the Data Management utility, you can quickly create data load jobs. You can use the data load templates as provided, or you can create custom templates for each Data Management user that based on the type of data loads the user will be performing.

The following example describes how to create a custom template based on the People template, which is one of the BMC template types.

  1. From the Application menu on the BMC Remedy ITSM Home page, select Data Management > Job console.
  2. In the navigation pane of Job console, choose Other Functions > Search template.
  3. In the Template Name field, type People, then click Search. The system shows the People template details.
  4. Click Copy Template.
  5. Provide a name for the new template that you are creating and select a template category from the Category menu. 
  6. In the Steps section of the window, remove any steps that you do not need in the template. To do this, highlight the steps that you do not need and from the  top of the Steps section, click Delete

    The job dependency and sequencing within the steps that remain are adjusted by the built-in UDM sequencing and dependency logic.

  7. Click Save

Ensuring  data  quality

If the validate step of the data load job encounters missing required or conditionally required fields, unexpected null values, data formatting issues, and so on, it will not run successfully. Before running that data load job, you need to manually review the data in the spreadsheets to ensure that the data that they contain is free from these types of errors.

If you find errors, you must address them in the spreadsheet before you create the data load job. In some cases, you might need to address the data errors in the database and then generate a new set of spreadsheets.

Data quality application preferences

If you are unsure of your data quality, consider setting the Job Wait preference to Manual Validate - Manual Promote. This preference causes the the job to wait at the validate step so that you can review the data before the validation runs and gives you the opportunity proactively troubleshoot data issues. BMC recommends setting this user preference especially when loading bulk data in which data quality is uncertain.

You also have the option of setting the Wait flag on individual steps in any given data load job directly on the Job console, by highlighting the job step and clicking Enable Wait. Use this option if you don't want every job step to wait for manual verification, only the ones that you choose.

Data creation and updates using a dataload job

BMC provides out-of-the-box dataload spreadsheets that are used as a source of data to run a dataload job. The data from these spreadsheets is used to create new records and update the pre-existing records on the target forms. Each spreadsheet is organized into tabs and each tab is mapped to an intermediate form, also known as staging form. You can access and download these spreadsheets from the Spreadsheet Management window. For more information about dataload spreadsheets, see Content in dataload spreadsheets and to locate and download spreadsheets, see Managing spreadsheets.

Creating and running data load jobs

When you have ensured that the pre-requisites are met, completed the planning activities, and populated the spreadsheets, you are ready to work with the data load jobs

When importing Company and Support Group data, Data Management generates new group IDs for imported companies and for imported support groups. This happens in the background and is transparent to you. The generation of new group IDs does not alter or affect the row level permissions of these companies and support groups.

Each data load job has the following steps:

  • The load step imports your data from your source files to the dataload staging forms. This step leverages Atrium Integrator jobs, which integrate source and target data stores with data transformation based on specific rules.
  • The validate step ensures that the data that was loaded during the load step does not contain fatal errors. Each Validate step represents the validation of data for one staging form. Note that during this step, if errors are found you will need to address the data errors.
  • The promote step either automatically or manually pushes all staging form records that are valid to their respective target forms within the BMC Remedy ITSM applications, based on the option selected for each Promote step within the job.

You might need to run these steps iteratively until all of the data passes the Validate step and can be promoted.

Data load job overview

You can use the dataload tool to import data into your BMC Remedy ITSM Suite of applications.

The main procedures of the dataload process are described in the following list:

  1. Create a job using the Data Management Job Console. For more information, see Creating a data load job.

  2. Select a job template that includes a set of steps.
    For instructions on using a template supplied by BMC, see Creating a job using out-of-the-box templates. For information about customizing templates, see Creating customized job templates.
  3. Run the job. Choose one of the following methods to run your job:
    • Schedule a job by using the Job Scheduler Console to run at a future date and time. For more information, see Scheduling data load jobs.
      Running a job includes loading, validating, and promoting data. For more information, see Completing steps.
    • Run the job manually and immediately instead of scheduling the job. For more information, see Running a job.
    If you have customized BMC Remedy ITSM Suite, see Using Data Management tools for custom Foundation data.

Updating data

The Data Management supports data updates only for CTM:LoadPeople form. 

  If you try to update the data on the forms for which data update is not supported:

  • For certain forms, the system overwrites the data and replaces the earlier field values with the values provided in the update. If you want to update data on such forms, you must provide values for all the form fields irrespective of whether you want to update those fields.
  • For certain forms listed in Data creation and updates using a dataload job, the data is only created and not updated. These forms are marked by a Create Only  flag, which  is displayed in the Steps panel of a job record when the Validate and Promote steps are created for a job.
    Forms marked by a Create Only flag

Best Practices

  • BMC recommends that you use the UDM utility to create new records and not update records. 
  • Even though creating and updating data depends on how you configure your application preferences for all jobs, BMC recommends that you do not change the settings for the Create Only field in the Application Preferences form when you are using the job templates.
  • If you want to customize your job templates using the existing template, BMC recommends that you ensure that you go through the list of Create Only forms and the settings for existing template.

Troubleshooting and Best Practices

The following links provide valuable information about troubleshooting the Data Management feature and Data Management Best Practices.

List of Create Only forms 

The following forms are marked by the Create Only flag for which data is only created and not updated:

Staging formJob templateStep name
CTM:LoadPeopleTemplateSFRPeople and ALL Foundation

Validate People Template Functional Role

Promote People Template Functional Role

CTM:LoadPeoplePermissionGroups

People and ALL Foundation

Validate Permission Group

Promote Permission Group

CTM:LoadSGPFunctionalRole

People and ALL Foundation

Validate Functional Role

Promote Functional Role

CTM:LoadPeopleWorkLog

People and ALL Foundation

Validate People Worklog

Promote People Worklog

CTM:LoadPeopleAttributes

People and ALL Foundation

Validate People Attribute

Validate People Attribute

CTM:LoadSGPOnCALLSupport Group and ALL Foundation

Validate On CALL

Promote On CALL

CFG:LoadBusinessTimeHolidaysSupport Group and ALL Foundation

Validate Business Time Holiday

Promote Business Time Holiday

CTM:LoadSGPAssignmentsSupport Group and ALL Foundation

Validate Favorite Groups

Promote Favorite Groups

PCT:LoadProdCatAliasMappingProduct Catalog and ALL Foundation

Validate Alias Mapping

Promote Alias Mapping

Staging formJob templateStep

CFG:LoadGroupEventMapping

Incident Process Setup and ALL Process Setup

Validate Script Association

Promote Script Association

CHG:LoadTemplateSPGAssoc

Change Process Setup and ALL Process Setup

Validate Change Template Support Group Assoc

Promote Change Template Support Group Assoc

HPD:LoadTemplateAssociations

Incident Process Setup and ALL Process Setup

Validate Incident Template CI Assoc

Promote Incident Template CI Assoc

HPD:LoadTemplateSPGAssocIncident Process Setup and ALL Process Setup

Validate Incident Template Support Group Assoc

Promote Incident Template Support Group Assoc

TMS:LoadVariableMappingTask Process Setup and ALL Process Setup

Validate Task Variable Mapping

Promote Task Variable Mapping

TMS:LoadAssignmentConfig

Task Process Setup and ALL Process Setup

Validate Task Assignment

Promote Task Assignment

TMS:LoadFlowTemplateTask Process Setup and ALL Process Setup

Validate Task Flow Template

Promote Task Flow Template

TMS:LoadAssociationTemplateTask Process Setup and ALL Process Setup

Validate Task Association Template

Promote Task Association Template

Staging formJob templateStep
AAS:LoadWorkInfo

Transactional Activity and

Transactional Release

Validate Activity Work Info

Promote Activity Work Info

APR:LoadSignature

Transactional Change and ALL Transactional

Validate Approval Signatures

Promote Approval Signatures

Transactional Release

Validate Release Signature

Promote Release Signature

AST:LoadBMC_BaseRelationship

Transactional Asset and ALL Transactional

Validate Asset Relationship

Promote Asset Relationship

AST:LoadWorkLogTransactional Asset and ALL Transactional

Validate Asset Work Info

Promote Asset Work Info

AST:LoadLicCertProdAssocTransactional Contract and ALL Transactional

Validate License Certificate Product Association

Promote License Certificate Product Association

CFG:LoadRemindersShared Components and ALL Transactional

Validate Reminders

Promote Reminders

CHG:LoadImpactedAreasTransactional Change and ALL Transactional

Validate Impacted Area

Promote Impacted Area

CHG:LoadWorkLogTransactional Change and ALL Transactional

Validate Infrastructure Change Work Info

Promote Infrastructure Change Work Info

CTR:LoadContract_RelationshipTransactional Contract and ALL Transactional

Validate Contract Relationship

Promote Contract Relationship

CTR:LoadWorkLogTransactional Contract and ALL Transactional

Validate Contract Work Info

Promote Contract Work Info

FIN:LoadCostsShared Components and ALL Transactional

Validate Financials

Promote Financials

FIN:LoadPaymentsShared Components and ALL Transactional

Validate Financials

Promote Financials

HPD:LoadHelpDeskTransactional Incident and ALL Transactional

Validate Incident

Promote Incident

HPD:LoadImpactedAreasTransactional Incident and ALL Transactional

Validate Impacted Areas

Promote Impacted Areas

HPD:LoadWorkLogTransactional Incident and ALL Transactional

Validate Incident Worklog

Promote Incident Worklog

PBM:LoadKnownErrorWorkLogTransactional Problem and ALL Transactional

Validate Known Error WorkLog

Promote Known Error WorkLog

PBM:LoadInvestigationWorkLogTransactional Problem and ALL Transactional

Validate Investigation WorkLog

Promote Investigation WorkLog

PBM:LoadSolutionWorkLogTransactional Problem and ALL Transactional

Validate Solution WorkLog

Promote Solution WorkLog

PBM:LoadImpactedAreasTransactional Problem and ALL Transactional

Validate Impacted Areas

Promote Impacted Areas

PBM:LoadSolutionDBAliasTransactional Problem and ALL Transactional

Validate Solution Alias

Promote Solution Alias

PBM:LoadSolutionDBMappingsTransactional Problem and ALL Transactional

Validate Solution Database Additional Mappings

Promote Solution Database Additional Mappings

RKM:LoadMultipleOper_CatTransactional Additional Category

Validate Operational Category

Promote Operational Category

RKM:LoadMultipleProdCategoryTransactional Additional Category

Validate Product Category

Promote Product Category

RKM:LoadMultipleBusinessSrvcTransactional Additional Category

Validate Business Services

Promote Business Services

RKM:LoadMulti_Reg_SiteGrp_SiteTransactional Additional Category

Validate Region, Site Grp, Site

Promote Region, Site Grp, Site

RKM:LoadMultipleOrg_DeptTransactional Additional Category

Validate Organization and Department

Promote Organization and Department

RKM:LoadMultipleCompanyTransactional Additional Category

Validate multi company

Promote multi company

RMS:LoadReleaseManifestAssocTransactional Release

Validate Release Manifest Association

Promote Release Manifest Association

RMS:LoadWorkLogTransactional Release

Validate Release WorkLog

Promote Release WorkLog

TMS:LoadWorkInfoTransactional Task and ALL Transactional

Validate Task WorkInfo

Promote Task WorkInfo

Was this page helpful? Yes No Submitting... Thank you

Comments