Onboarding Foundation and Assignment data

The Data Management tool provides you with a wizard for importing (or "onboarding") Foundation data and Assignment data. Using the Onboarding wizard, you can review, and (if necessary) edit the imported data directly in the wizard. You also promote the data to your test or production database from the Onboarding wizard. You can use the Onboarding wizard when setting up new installations or any time that you need to add or update Foundation data.

When using the Onboarding wizard, keep the following points in mind:

• After you import the data using the Onboarding wizard, there is never a need to reimport the data, because you can edit the imported data directly from the UI of the Onboarding wizard.
• While Assignment data is not usually considered to be part of the Foundation data, Assignment data is treated like Foundation data in the Onboarding wizard and in this documentation.
• If you have alternative methods of setting up your Foundation data, or other ways to import data, you can dismiss the Onboarding wizard and proceed independently.
• This wizard is for both adding data and updating data.

Data updates supported

Previous versions of the Onboarding wizard supported only the importation of new Foundation and Assignment data. Starting with version 9.0 of BMC Remedy ITSM, however, you can use the Onboarding wizard to also update some existing Foundation and Assignment data. To update existing data, use a spreadsheet that contains updated data (instead of onboarding data) in the applicable Phase 1, Loading data into the staging forms step shown in the following table.

The following table lists the fields that you can update using the Onboarding Wizard and shows you in which step of Phase 1, Loading data into the staging forms you update the data.

Data flow diagrams

The following diagrams provide a high level view of how data flows from the existing application database, through the Onboarding wizard, into the application database of the new BMC Remedy ITSM installation. There are two methods for promoting data, Manual and Automatic, which are described later in this topic. You choose which data promotion method you want to use when you finish loading the data into the staging forms. Note how all of the data reviewing, editing, validating, and promoting operations are done directly from the Onboarding wizard. You don't need to close the wizard to perform any of the onboarding tasks.

Onboarding wizard, Manual promotion overview

Onboarding wizard, Automatic promotion overview

Foundation data is common to many forms and workflows; it supports the BMC Remedy ITSM applications and without it you cannot use the applications.  Foundation data includes the following data types:

• Company
• Organization
• Location
• Support group
• People
• Operational Catalog 
• Product Catalog 

You can read more about the Foundation module and Foundation data types here.

Through the use of "staging forms", the Onboarding wizard provides a single place from which you can enter, review, and edit the data that you are creating. In Data Management, staging forms are intermediate data forms that provide a working space for you to prepare the data before "promoting" it (that is, moving it to the corresponding production data forms).

If the Onboarding wizard detects any data exception conditions you can also edit the invalid data directly in the wizard. Data exceptions occur when a record does not pass the validation tests that the Onboarding Wizard runs. The validation tests ensure the data does not contain data exceptions.

Modes of operation

The Onboarding wizard has two modes of operation:

• Guided mode
• Unguided mode

When the Onboarding wizard opens, it prompts you to choose one of the operating modes explained in the following sections. If you close the Onboarding wizard before you promote the data, when you re-open the wizard it remembers the mode that you were using when you closed it. 

Guided mode

Recommended use: To provide the initial load of Foundation data in newly installed environments. You can, however, use Guided mode any time.

The purpose of Guided mode is to step you through the process of:

• Ensuring you identify at least one company as an operating company
• Entering data in a specific order
• Creating all of the required data

Operating company

The operating company is one of the key base components of the BMC Remedy ITSM applications and represents an internal organization or business unit for which your IT organization provides support and services.  Many important, core, foundational data elements cannot be created unless you have identified an operating company. Because the BMC Remedy ITSM applications are multi-tenant, you can identify more than one company as the operating company.

Unguided mode

Recommended use: Use Unguided mode to load additional Foundation data to production and test environments, after the initial load of Foundation data. You can, however, use Unguided mode at any time.

After your BMC Remedy ITSM applications have enough Foundation data for you to run them, you can return to the Onboarding wizard to add more data. You do this by selecting Data Management > Onboarding Wizard from the Application menu .

For example, after your initial load of the Foundation data, you might need to add several new support groups. To do this, you can open the Onboarding wizard and go straight to the Support Groups step to add just those records. You do not need to enter records for any of the other data types. We call this mode of operation the Unguided mode, because you do not need to follow the structured sequence of steps that is enforced in Guided mode.

Onboarding phases

The Onboarding wizard steps you through the process of onboarding data in distinct phases:

• Phase 1, loading data into the staging forms

• Phase 2, Review, Promote, and exception

Phase 1, Loading data into the staging forms

In Phase 1 you load data into the staging forms. In Data Management, staging forms are intermediate data forms that provide a working space for you to prepare the data before "promoting" it (that is, moving it to the corresponding production data forms).

The phase has multiple steps, corresponding to each of the Foundation data types supported by the Onboarding wizard, and a Finalize, Review and Activate step that you run after you have loaded all the data for your onboarding job. Depending on the mode you are using (Guided or Unguided) and the number of data types that you are entering, however, you might not run all of the steps. The following tabs provide detailed information about each step.

Data load methods

You load data into the staging forms using one of the methods described in the following table.

^{1 BMC strongly recommends that you read the Instruction tab in each spreadsheet. This tab contains important information about formatting, versioning, and how the spreadsheets are laid out.}

^{2 The standard data-load spreadsheets provided with Advanced Data Management do not work work with the Onboarding Wizard, which is designed specifically to work with the simplified spreadsheets.}

^{3 Some types of model data have dependencies on other data types, as noted in the preceding list. For example, If you import People template model data, you must also ensure that you either load Support Group data in the same data-load job, or that Support Group data is already on the target system. If you need to provide the Support Group data, you can either load the Support Group model data provided with the Onboarding wizard or you can load your own Support Group data. If the Support Group data is not present on the target system or is not a part of the current data load, the People template model data will fail the validation step.}

^{4 In most cases, if model data fails the validation process of the Promoting step, the data on which the model data is dependent either is not present on the target system or is not a part of the current data load.} 

^{5 If the model data does not exactly match the requirements of your organization, you can still use it. After you import the data into the staging forms, open the Data Visualizer and use it to edit the model data so that it conforms to your specific requirements.}

Using LDAP imports

Before you can use this method, however, a connection to the LDAP server must be configured by someone with DMT Admin permissions. Click here for information about configuring your system for the LDAP import method. You can configure connections to as many LDAP servers as you need to.

Using the LDAP import method, you can perform the following operations:

• Onboard non-support people data.
• Synchronize the directory with your people data, by running the LDAP import at regular intervals.

This method is available as a selection from  Step 6 People .

To use the LDAP import method

1. From the Onboarding wizard, click Step 6 People. 
2. From the  Create Data Manually  drop-down menu, select  Load data into the system using Lightweight Directory Access Protocol (LDAP) .
   The screen refreshes to display the fields you need for setting up the LDAP connection. 
3. From the  Setting Name  drop-down menu, select the LDAP connection that you want to use.

4. Click  Import .

   Important

   If any of the records that you import have an error, the system displays the Show Log link, which opens an error log listing the records with errors and the nature of the error. You can choose to correct these records in the active directory, and then run the import again, or to filter the records that have errors out of the LDAP import by using a custom search filter.

   By default, if 20% of the records in any given sampling of the LDAP import have errors, the import halts. You can configure the percentage of errors allowed before the import halts by updating the Maximum Number of Errors setting in the Step Error Handling Settings tab of the AROutput step, in the transformation used by the import. Click here for more information about changing the Maximum Number of Errors setting.

5. Perform the Phase 2  activities.

Phase 2, Reviewing, Promoting, Exception Handling

In Phase 2, you are able to review data directly from the Onboarding wizard and, if necessary, edit the data in the wizard. After you review the data, you run the Promoting step to push the data out to the production forms. The Promoting step also performs validation tests against the data to ensure data consistency. If the validation process detects any inconsistencies in the data, the validation process writes an exception message to the screen.

If you encounter exception messages, you perform the Exception Handling step. The process of Exception handling can be iterative. After you deal with any data exceptions, rerun the Promoting step. If the Onboarding wizard encounters further exceptions, you deal with those exceptions, then rerun the Promoting step, and so on. Repeat these steps until the Onboarding wizard promotes all of the data.

The steps associated with Phase 2 are handled through the Data Visualizer.

The following tabs provide more information about each step.

Reviewing and promoting data

You use the Data Visualizer feature of the Onboarding wizard to review and, if necessary, edit your data, and later to correct any data exceptions.  

When you finish Phase 1, click Finalize Review and Activate to move into Phase 2 and open Data Visualizer. From Data Visualizer, you can review the data that you entered to ensure it is correct before you promote it. If you discover a problem in a record, you can edit the record directly in Data Visualizer. You can also make bulk edits in the Data Visualizer by using the Modify All feature. For example, if you notice that one of the Assigned Group names has a spelling mistake and the mistake is repeated in all of the Assignment records, you can use the Modify All feature to update all of the affected records in a single operation.  

Data Visualizer displays the following data types in columns, grouped under the headings shown in the following table.

If there is a large volume of data, you can limit the number of records displayed in the columns by using the filters at the top of each column.

The filters cascade downward in the data hierarchy. If, for example, you filter by company, the rest of the records shown in Data Visualizer belong only to that company. If multiple filters appear at the top of a column, you can filter on multiple conditions.

Use these steps to review and, if necessary, update the data.

To review and promote data

1. When you complete Phase 1, click Finalize Review and Activate. 
2. Select how to review and activate the data: Manually or Automatically. This action opens the Data Visualizer.
3. In Data Visualizer, the Organizational Structures column is open by default. To expand other columns, click the arrow on the column panel. 
4. Review the data to ensure that it is correct.
5. If you find a problem with a record, double-click the record to open the data-editing window. 
6. Correct the problem, then click Save to close the window. 
7. Perform one of the following actions, after which the wizard Displays the Activation screen:
   1. If you selected Manually in Step 2, click Validate. The Onboarding wizard runs the validation tests, but does not promote any data until you tell it to. 
   2. If you selected Automatically in Step 2, click Promote. The Onboarding wizard runs the validation tests and automatically promotes valid data to the production database.
8. If the Activation screen shows that there are no data exceptions, perform one of the following actions:
   1. If you selected Manually in Step 2, click Promote to promote the data to the production forms and finish the job. 
   2. If you selected Automatically in Step 2, you are finished the job.
9. If the Activation screen shows that there are exceptions, you must perform Exception handling. 

Activation screen

Exception handling

Use these steps to correct any data exceptions that might occur during the Activate step.

To correct data exceptions

1. On the Activation screen, click Exceptions to open a list of records in the Data Visualizer that have data exceptions. This list also provides information about why the record caused the data exception. For detailed information about diagnosing a data exception, see Diagnosing data exceptions.

2. Open the data editing window by performing one of the following actions:

   • To edit a single record, double-click the record. 

   • To edit multiple records, select the records that you want to update, then double-click the selection (for People and Assignment records only).
   • To edit all records in single data structure (for example, all Company records), click Modify All (for People and Assignment records only).

3. Using the Data Editing window, correct the data exception, then click  Save  to close the window. The Onboarding wizard removes the corrected records from the Data Visualizer. 

   Important

   If you later want to see the records that you updated, click Corrected. The wizard redisplays the corrected records that previously were removed from view. See Controlling which records appear in the Data Visualizer for more information about how to control which records appear in the Data Visualizer.

4. Depending on how you chose to review and promote the data (as described in To review and promote data), click of the following buttons:  

   • Validate — Manual, validates the corrected data and then waits for you to promote it.

   • Promote — Automatic, validates the corrected data and automatically promotes data that passes validation.

5. If the validation test finds more data exceptions, repeat Exception handling iteratively, until either:
   • The validation checks no longer find data exceptions
   • You choose to ignore the data exceptions and start a new onboarding session
   • You choose to delete the records that are causing the data exceptions and address those issues at a later time
     
     See Running a new onboarding session, Starting a new onboarding session, and Deleting records for information about the different options for handling data exceptions. 

Diagnosing data exceptions

Data exceptions have associated codes and messages that can help you to diagnose the cause of an exception.

To view the code and message for a specific record, select the record in the Data Visualizer. The Onboarding wizard displays the associated code and message in the Exception Messages panel on the right side of the Data Visualizer. The code and the message appear under separate headings: Exception Codes and Exception Message.

The information under the Exception Message heading  concatenates  the codes and the messages. 

If multiple reasons for the exception exist, the Data Visualizer reports all of the related codes in a semicolon-separated list under the Exception Codes heading. The associated message text also appears in a semicolon-separated list under the Exception Message heading.

The exception messages always appear in the same order as the codes. 

Exception console

The Exception console is another tool that you can use to view and correct data exceptions. The Exception console organizes the exceptions in a tree format grouped by exception type. After you correct the exception, the Exception console stops displaying the record.

To use the Exception console

1. Select an exception type from the navigation tree. 
   Selecting an exception type lists all of the records with that type of data exception in the table.  
2. From the table, select one or more rows. By selecting multiple rows, you can perform bulk updates.
3. Use the editing panel below the table, perform one of the following actions:

   1. Update the field that contains the data exception with the correct content, then click Update Staging. If you selected more than one row in the preceding step, the change that you make is applied to all of the selected records.

      Important

      There is a pair of  icons to the right of each field:

      1. Using the left icon of the pair, you can expand the text editing field to make it easier to edit very large text entries.
      2. Using the right icon of the pair deletes the field contents and inserts the string NULL. Insert NULL w hen you specifically want to empty the contents of any given field. Placing NULL in the field ensures that the system interprets the field as empty.  

          

   2. Delete the selected records by clicking Delete Staging.

Controlling which records appear in the Data Visualizer

These buttons are available in Data Visualizer to help you control the records that it displays. The buttons that you see at any given time depend on what you are doing and where you are in the Review and Activate steps.

Deleting records

If you need to delete records from the Data Visualizer because they contain nonfatal data exceptions or because you chose to address the data exceptions related to those records at a later time, use the delete icons at the bottom of each column.

• Delete Selected deletes the record or records that you select in the column.
• Delete All deletes all of the records in the column.

Running a new onboarding session

When you finish promoting the data from an onboarding session and you are still logged on to the Onboarding Wizard, you can easily start another onboarding session with freshly imported data.

To start a new onboarding session, click New to return to the Phase 1 screen and to clear from view all data from the previous onboarding session.

• The New button does not delete the data from the staging forms used in Phase 1; it only removes the data from view. The data in the staging forms is kept in the staging forms according to value specified in the Keep Promoted Staging Records field in the Data Management application preferences.
• Data that has been activated using the Data Wizard is unaffected by the New button

See the Starting a new onboarding session use cases for more information.

Troubleshooting a job

If you think that your job might have stopped unexpectedly and you want to view details about its progress, you can check the job in the Data Management Job console. By looking at the job in the console, you can see information related to its progress. If the job has stopped unexpectedly, you can see, for example, on which form the job stopped and at which stage, either Validation or Promotion.

To look at a job in the Data Management Job console

1. From the Applications list on the IT Home page, select Data Management > Job Console.
2. On the Job Console, select the required job in the Jobs table and click View.

Troubleshooting an Onboarding Wizard job that uses the LDAP import method

If you are running an Onboarding Wizard job that uses the LDAP import method, the Data Management Job console entry for the job will not contain any information related to the LDAP aspect of the job.To perform troubleshooting on a job that uses the LDAP import method, click the Show log link, as shown in the following illustration.