Overview of the data load process

This section provides you with an overview of how to use the Data Management tool to import data into your BMC Remedy ITSM applications. The section walks you through the following activities:

• reviewing pre-requisites
• planning activities
• populating data spreadsheets
• creating and running data load jobs 

Reviewing pre-requisites

Before you start planning your data load job, there are some things that you need to check and, if necessary, set up. This section describes the checks that you must perform for the general system configuration and for your own user settings.

• Configuring users and roles
• User level settings
• System configuration checks and tasks

Configuring users and roles

Create and Define the Data Management user IDs and the Data Management Administrator IDs. For information about how to do this, refer to the following topics:

• Data Management user roles (background information that describes the  Data Management user roles)
• Defining users and administrators for Data Management (a procedure for assigning Data Management roles to new and existing system user IDs)

User level settings

You can configure specific application preferences for each user so that they are enabled by default for all of their dataload jobs. 

For example, if you are performing bulk data loads, you might want to change the default value for the "Delete Job and Steps after" preference from the default value of 365 days, to a much lower value, so that the large volume of jobs and steps created by a bulk import are not retained in the system for an entire year. If other people in your organization also perform bulk data loads, then their application preferences need to be configured in the same way.

System configuration checks and tasks

To perform the following configuration checks and tasks, login as a system administrator.

• Ensure that escalations are enabled on your AR System server. Use the following procedure to check the AR System server settings and, if necessary, enable escalations:
  1. From the Application menu on the BMC Remedy ITSM Home page, select AR System Administration > AR System Administration console.  
  2. From the navigation pane of the AR System Administration console, select System > General > Server Information.
  3. From the Server Information dialog box, open the Configuration tab.
  4. Look for the Disable Escalation check box. If the check box is selected, clear the check mark from it. When the check box is clear, escalations are enabled.
  5. Click Apply, then click OK.
• If you will be performing data imports in an environment that uses server groups, BMC recommends you to perform the following actions: 
  • If you are using BMC Remedy IT Service Management 8.1.01, designate a default carte server.
  • If you are using BMC Remedy IT Service Management 8.1.02 or later versions, rank the servers in the server group by using the AR System Server Group Operation Ranking form.
• Check the UDM:Config form to verify that the correct connection details for the Atrium Integrator repository and Atrium Integrator name are configured for your Data Management jobs.

• Check the Remedy Application (RApp) Service passwords for  the "Remedy Application Service" user. This information is stored on the UDM:RAppPassword form. 
• Ensure that the BMC Remedy AR System repository for attachments and spreadsheets is configured with the correct Data Management Load file path. 

Planning activities

Before you start to populate the data management spreadsheets, run through the planning activities that are outlined in this section. Performing these activities up front, ensures that you thoroughly understand the nature of the data load jobs you will be running and their specific needs.

The planning activities discussed in this section include:

• Identifying forms
• Identifying data creation and update use cases
• Preparing for bulk data loads
• Identifying customizations
• Creating a test plan

Identifying forms

Data for BMC Remedy ITSM applications is stored on three types of forms, depending on the type of data they contain: 

• Foundation forms
• Process setup forms
• Transactional forms

Before you create a data load job, you need to identify which data form types  are involved in the data load. Familiarizing yourself with the types of data that you are loading helps you to better understand the nature of the data load and its need. This kind of familiarity with the data load job is important in later steps. 

Identifying data creation and update use cases

Ensure that you understand the forms and spreadsheets that your data load job will use to:

• add information to the database, if any.  
• update information in the database, if any.

Preparing for bulk data loads

Bulk data loads can present troubleshooting and performance challenges when the data quality cannot be guaranteed or if your system resources are not sufficient to run the data load processes. If you are running bulk data loads, keep the following points in mind when planning the data load jobs, to reduce the impact of any issues associated with data quality or performance. 

• If you are unsure of the quality of the data you are loading, BMC recommends that you load the data in smaller, manageable chunks. Loading bulk data in chunks makes it easier to troubleshoot the data load job if it encounters issues with data quality or formatting during the validate step. You cannot move on to the promote step step until all of the data that you are importing is valid. 
• Review the following topics to ensure your system has the required resources to run the data load jobs efficiently: 
  • Configuring performance enhancements for Data Management
  • Performance benchmarks for Data Management

Identifying customizations

The data load spreadsheets that BMC ships with the Data Management feature are designed to work with an out-of-the-box BMC Remedy ITSM system. If you have customized the database for the specific needs of your organization by adding fields to the data forms, you must account for these custom fields when planning the data load jobs. If you don't, the data load job will not pick up the data from the customized fields.   

Creating a test plan

Always test your dataload jobs before you run them on live data. Create a test plan that takes into account the following considerations:

• the forms into which you will be loading data 
• the volume of data you are loading
• your system performance and resources (memory, processing speed, network bandwidth, database capabilities, and so on)
• the number of threads configured for CAI
• UDM performance on the server.

When you finish drafting the test plan, review it with stakeholders and experts on the various system areas that your plan touches on.

Populating the data spreadsheets

When all of the prerequisites are in place and you have completed your test plan, you can start to work with the data load spreadsheets.

Identifying spreadsheets

• Identifying the spreadsheets for your use case http://docs.bmc.com/docs/display/public/itsm81/Spreadsheets+provided+and+target+form+mapping
• Using CTM_LoadPeopleModification tab on the People.xls: 
  • You can create a template with all support group associations and permissions that you would like to give a user or a login id. Specify the newly created template name and login id in the spreadsheet to load those support group associations or user permissions groups. Update the Group list in the BMC Remedy AR System User form with the support group associations and permissions.
  • A deadlock in the database happens if same person or login ID is repeated in a spreadsheet and the duplicate records are processed in different threads or processes in the database. To prevent the deadlock in the database, BMC recommends you to have one template with all support group associations, people permissions, and functional roles to apply to all users in one job. If multiple templates need to be added to a single person, divide the job per template rather than dividing a job per user. 

Identifying templates

Using the templates that BMC provides with the Data Management utility, you can quickly create data load jobs. You can use the data load templates as provided, or you can create custom templates for each Data Management user that based on the type of data loads the user will be performing.

Example

The following example describes how to create a custom template based on the People template, which is one of the BMC template types.

1. From the Application menu on the BMC Remedy ITSM Home page, select Data Management > Job console.
2. In the navigation pane of Job console, choose Other Functions > Search template.
3. In the Template Name field, type People, then click Search. The system shows the People template details.
4. Click Copy Template.
5. Provide a name for the new template that you are creating and select a template category from the Category menu. 

6. In the Steps section of the window, remove any steps that you do not need in the template. To do this, highlight the steps that you do not need and from the top of the Steps section, click Delete. 

   Note

   The job dependency and sequencing within the steps that remain are adjusted by the built-in UDM sequencing and dependency logic.

   

7. Click Save. 

Ensuring data quality

If the validate step of the data load job encounters missing required or conditionally required fields, unexpected null values, data formatting issues, and so on, it will not run successfully. Before running that data load job, you need to manually review the data in the spreadsheets to ensure that the data that they contain is free from these types of errors.

If you find errors, you must address them in the spreadsheet before you create the data load job. In some cases, you might need to address the data errors in the database and then generate a new set of spreadsheets.

Data quality application preferences

If you are unsure of your data quality, consider setting the Job Wait preference to "Manual Validate - Manual Promote" . This preference causes the the job to wait at the validate step so that you can review the data before the validation runs and gives you the opportunity proactively troubleshoot data issues. BMC recommends setting this user preference especially when loading bulk data in which data quality is uncertain.

Creating and running data load jobs

When you have ensured that the pre-requisites are met, completed the planning activities, and populated the spreadsheets, you are ready to work with the data load jobs. 

Each data load job has the following steps:

• The load step imports your data from your source files to the dataload staging forms. This step leverages Atrium Integrator jobs, which integrate source and target data stores with data transformation based on specific rules.

• The validate step ensures that the data that was loaded during the load step does not contain fatal errors. Each Validate step represents the validation of data for one staging form. Note that during this step, if errors are found you will need to address the data errors.
• The promote step either automatically or manually pushes all staging form records that are valid to their respective target forms within the BMC Remedy ITSM applications, based on the option selected for each Promote step within the job.

Data load job overview

You can use the dataload tool to import data into your BMC Remedy ITSM Suite of applications.

The main procedures of the dataload process are described in the following list:

1. Create a job using the Data Management Job Console. 
   For instructions, see Creating a job.

   Note

   For information about creating customized jobs, see Creating customized jobs.

2. Select a job template that includes a set of steps.
   For instructions on using a template supplied by BMC, see Creating a job using out-of-the-box templates. For information about customizing templates, see Creating customized job templates.
3. Run the job. Choose one of the following methods to run your job:
   • Schedule a job by using the Job Scheduler Console to run at a future date and time
     For instructions, see Scheduling jobs.
     Running a job includes loading, validating, and promoting data. For more information, see Completing steps.
   • Run the job manually and immediately instead of scheduling the job
     For instructions, see Running a job.

Troubleshooting and Best Practices

The following BMC knowledge articles provide valuable information about troubleshooting the Data Management feature and Data Management Best Practices.

• Troubleshooting Data Management
• Data Management Best Practices