Onboarding Foundation and Assignment data
The Data Management tool provides you with a wizard for importing (or "onboarding") Foundation data and Assignment data. Using the Onboarding wizard, you can review, and (if necessary) edit the imported data directly in the wizard. You also promote the data to your test or production database from the Onboarding wizard. You can use the Onboarding wizard when setting up new installations or any time that you need to add or update Foundation data.
When using the Onboarding wizard, keep the following points in mind:
- After you import the data using the Onboarding wizard, there is never a need to reimport the data, because you can edit the imported data directly from the UI of the Onboarding wizard.
- While Assignment data is not usually considered to be part of the Foundation data, Assignment data is treated like Foundation data in the Onboarding wizard and in this documentation.
- If you have alternative methods of setting up your Foundation data, or other ways to import data, you can dismiss the Onboarding wizard and proceed independently.
- This wizard is for both adding data and updating data.
Data updates supported
Previous versions of the Onboarding wizard supported only the importation of new Foundation and Assignment data. Starting with version 9.0 of BMC Remedy ITSM, however, you can use the Onboarding wizard to also update some existing Foundation and Assignment data. To update existing data, use a spreadsheet that contains updated data (instead of onboarding data) in the applicable Phase 1, Loading data into the staging forms step shown in the following table.
The following table lists the fields that you can update using the Onboarding Wizard and shows you in which step of Phase 1, Loading data into the staging forms you update the data.
|Fields that can be updated||Loading data into staging form step|
|Type||Step 1, Company|
|Country, State/Province, City, Zip/Postal Code, Street, Description, Region, Site Group||Step 3, Location|
|Is this a vendor group?, Is this a frontline group?, Group Email, Description||Step 4, Support Groups|
|License Type||Step 5, People Template|
|Last Name, First Name, Organization, Department, Site, Email Address, License Type||Step 6, People|
|Description||Step 9, Assignment|
Data flow diagrams
The following diagrams provide a high level view of how data flows from the existing application database, through the Onboarding wizard, into the application database of the new BMC Remedy ITSM installation. There are two methods for promoting data, Manual and Automatic, which are described later in this topic. You choose which data promotion method you want to use when you finish loading the data into the staging forms. Note how all of the data reviewing, editing, validating, and promoting operations are done directly from the Onboarding wizard. You don't need to close the wizard to perform any of the onboarding tasks.
Onboarding wizard, Manual promotion overview
Onboarding wizard, Automatic promotion overview
Foundation data is common to many forms and workflows; it supports the BMC Remedy ITSM applications and without it you cannot use the applications. Foundation data includes the following data types:
- Support group
- Operational Catalog
- Product Catalog
You can read more about the Foundation module and Foundation data types here.
Through the use of "staging forms", the Onboarding wizard provides a single place from which you can enter, review, and edit the data that you are creating. In Data Management, staging forms are intermediate data forms that provide a working space for you to prepare the data before "promoting" it (that is, moving it to the corresponding production data forms).
If the Onboarding wizard detects any data exception conditions you can also edit the invalid data directly in the wizard. Data exceptions occur when a record does not pass the validation tests that the Onboarding Wizard runs. The validation tests ensure the data does not contain data exceptions.
Modes of operation
The Onboarding wizard has two modes of operation:
When the Onboarding wizard opens, it prompts you to choose one of the operating modes explained in the following sections. If you close the Onboarding wizard before you promote the data, when you re-open the wizard it remembers the mode that you were using when you closed it.
Recommended use: To provide the initial load of Foundation data in newly installed environments. You can, however, use Guided mode any time.
The purpose of Guided mode is to step you through the process of:
- Ensuring you identify at least one company as an operating company
- Entering data in a specific order
- Creating all of the required data
The operating company is one of the key base components of the BMC Remedy ITSM applications and represents an internal organization or business unit for which your IT organization provides support and services. Many important, core, foundational data elements cannot be created unless you have identified an operating company. Because the BMC Remedy ITSM applications are multi-tenant, you can identify more than one company as the operating company.
Recommended use: Use Unguided mode to load additional Foundation data to production and test environments, after the initial load of Foundation data. You can, however, use Unguided mode at any time.
After your BMC Remedy ITSM applications have enough Foundation data for you to run them, you can return to the Onboarding wizard to add more data. You do this by selecting Data Management > Onboarding Wizard from the Application menu .
For example, after your initial load of the Foundation data, you might need to add several new support groups. To do this, you can open the Onboarding wizard and go straight to the Support Groups step to add just those records. You do not need to enter records for any of the other data types. We call this mode of operation the Unguided mode, because you do not need to follow the structured sequence of steps that is enforced in Guided mode.
The Onboarding wizard steps you through the process of onboarding data in distinct phases:
Phase 1, Loading data into the staging forms
In Phase 1 you load data into the staging forms. In Data Management,staging forms are intermediate data forms that provide a working space for you to prepare the data before "promoting" it (that is, moving it to the corresponding production data forms).
The phase has multiple steps, corresponding to each of the Foundation data types supported by the Onboarding wizard, and a Finalize, Review and Activate step that you run after you have loaded all the data for your onboarding job. Depending on the mode you are using (Guided or Unguided) and the number of data types that you are entering, however, you might not run all of the steps. The following tabs provide detailed information about each step.
In guided mode
When providing company information in Guided mode, the application asks you to indicate which of the companies is the "Onboarding" company. The Onboarding company is the company to which all of the other records that you create in steps 2 through 8 are associated.
In Guided mode.
In Guided mode.
In Guided mode.
Required, when creating Support people, strongly recommended for non-support people
Using People templates is a highly effective way to make sure that you have defined permissions, support groups, and functional roles for each of the user roles in your support organization. In Step 6, when you load People information, you specify which People template to use. Consider creating a People template for each user role in the organization.
If a person has a user role of Incident Management Specialist, you can use the Incident Management - Specialist template (provided as model data by BMC) to define the permissions and support groups that an incident management specialist must belong to. In Step 6, when defining a person who is an incident management specialist, you need only to point to the Incident Management Specialist template (by entering Incident Management - Specialist in the Template Name column) to ensure that the correct permissions, support groups, and functional roles are assigned. Using templates makes creating People records quicker and reduces the chances for errors.
In Guided mode
People templates are required when defining Support people.
Step 6 also provides you with a way to import People information for non-support people directly from your company's active directory using an LDAP import. Using this method to import People data is faster and more accurate than other methods. It also gives you the ability to set up regularly scheduled, automated imports from the active directory. Click here for information about using an LDAP import.
Operational Catalogs are not tied to a company. However, if you select the Global flag, either by clicking Global for manual entries or by adding a 0 to the Global column in the spreadsheet, the Onboarding Wizard will tie the catalog to all of the companies in your installation. If your Organization does not yet use Operational Catalogs, you can take this opportunity to introduce them by using the Model data available in this step.
Product Catalogs are not tied to a company. However, if you select the Global flag, either by clicking Global for manual entries or by adding a 0 to the Global column in the spreadsheet, the Onboarding Wizard will tie the catalog to all of the companies in your installation. If your Organization does not yet use Product Catalogs, you can take this opportunity to introduce them by using the Model data available in this step.
Yes, in both modes
When you click Finalize Review and Activate, the Onboarding wizard asks if you want to promote the data either manually or automatically:
- Manually – The wizard runs the validation tests against the data but does not promote valid records until you tell it to. Using this option, if the validation tests find exceptions in the data, you can address the exceptions and rerun the validation tests before any data is promoted to the production data set. Use this option if you want to ensure that all of your data is valid before any records are promoted.
- Automatically – The wizard runs the validation tests against the data and immediately promotes all of the valid records to the production data set. You must still address any records that did not pass the validation tests, to determine if the exception condition is fatal or benign. You can, however, address the data exception either immediately or at a later time. See the New button use cases for a description of how to troubleshoot data exceptions at a later time. Use this option if you have a high level of confidence that your data will pass the validation tests.
Data load methods
You load data into the staging forms using one of the methods described in the following table.
|Manual entry||If you have a relatively small number of records to load, use the manual method. When using the manual method, you type information directly into fields in the Onboarding wizard. The fields in the Onboarding wizard correspond with the fields on the data form. For example, when creating data for the Organization form, you are prompted to provide the Organization and Department for each record you add. See Onboarding data manually for a description of how to use this data-load method.|
|Spreadsheets||If you have a large number of records to load, use the spreadsheets that are provided with the Onboarding Wizard. These spreadsheets are simplified versions of the standard data-load spreadsheets provided with Advanced Data Management. The spreadsheets contain an Instructions tab and, in most cases, a single data tab with columns that correspond to the required, conditional, and most commonly used fields. The People spreadsheet contains multiple data tabs that allow you to make associations with support groups and functional roles. See Onboarding data using spreadsheets for a description of how to use this data-load method. Understanding the dataload spreadsheets provides a general overview of the spreadsheets. 1, 2|
Model data – Model data is a pre-defined set of typical data. You can use this model data to seed your new installation with data types that you might not currently have. Using model data to seed your system greatly reduces the amount of time needed for your organization to realize value from the new system. For example, if you load the model data for Product Catalog, the Product Type field is loaded with a set of common product types, such as Hardware System Component, Network Port, Service, and so on.
Model data is available for the following data forms:
|LDAP import||This method is available only for importing non-support People data. You can use the method to load non-support People data directly from your company's active directory.|
Click here for more information about the LDAP import method.
1 BMC strongly recommends that you read the Instruction tab in each spreadsheet. This tab contains important information about formatting, versioning, and how the spreadsheets are laid out.
2 The standard data-load spreadsheets provided with Advanced Data Management do not work work with the Onboarding Wizard, which is designed specifically to work with the simplified spreadsheets.
3 Some types of model data have dependencies on other data types, as noted in the preceding list. For example, If you import People template model data, you must also ensure that you either load Support Group data in the same data-load job, or that Support Group data is already on the target system. If you need to provide the Support Group data, you can either load the Support Group model data provided with the Onboarding wizard or you can load your own Support Group data. If the Support Group data is not present on the target system or is not a part of the current data load, the People template model data will fail the validation step.
4 In most cases, if model data fails the validation process of the Promoting step, the data on which the model data is dependent either is not present on the target system or is not a part of the current data load.
Data Visualizer and use it to edit the model data so that it conforms to your specific requirements.5 If the model data does not exactly match the requirements of your organization, you can still use it. After you import the data into the staging forms, open the
Using LDAP imports
Before you can use this method, however, a connection to the LDAP server must be configured by someone with DMT Admin permissions. Click here for information about configuring your system for the LDAP import method. You can configure connections to as many LDAP servers as you need to.
Using the LDAP import method, you can perform the following operations:
- Onboard non-support people data.
- Synchronize the directory with your people data, by running the LDAP import at regular intervals.
To use the LDAP import method, you need DMT User permissions. If you need to configure an LDAP connection, you need DMT Admin permissions.
This method is available as a selection from Step 6 People .
To use the LDAP import method
- From the Onboarding wizard, click Step 6 People.
- From the Create Data Manually drop-down menu, select Load data into the system using Lightweight Directory Access Protocol (LDAP) .
The screen refreshes to display the fields you need for setting up the LDAP connection.
- From the Setting Name drop-down menu, select the LDAP connection that you want to use.
Click Import .
If any of the records that you import have an error, the system displays the Show Log link, which opens an error log listing the records with errors and the nature of the error. You can choose to correct these records in the active directory, and then run the import again, or to filter the records that have errors out of the LDAP import by using a custom search filter.
By default, if 20% of the records in any given sampling of the LDAP import have errors, the import halts. You can configure the percentage of errors allowed before the import halts by updating the Maximum Number of Errors setting in the Step Error Handling Settings tab of the AROutput step, in the transformation used by the import. Click here for more information about changing the Maximum Number of Errors setting.
- Perform the Phase 2 activities.
Phase 2, Reviewing, Promoting, Exception Handling
In Phase 2, you are able to review data directly from the Onboarding wizard and, if necessary, edit the data in the wizard. After you review the data, you run the Promoting step to push the data out to the production forms. The Promoting step also performs validation tests against the data to ensure data consistency. If the validation process detects any inconsistencies in the data, the validation process writes an exception message to the screen.
If you encounter exception messages, you perform the Exception Handling step. The process of Exception handling can be iterative. After you deal with any data exceptions, rerun the Promoting step. If the Onboarding wizard encounters further exceptions, you deal with those exceptions, then rerun the Promoting step, and so on. Repeat these steps until the Onboarding wizard promotes all of the data.
The steps associated with Phase 2 are handled through the Data Visualizer.
The following tabs provide more information about each step.
In this step, you review the data and make updates to it, if required.
In this step, the Onboarding wizard pushes the data from the staging forms to the production forms.
This step occurs only when the Promoting step encounters data exceptions. In this step, you can edit or delete the data that caused the exception directly from the Onboarding wizard, as described in Correcting data exceptions, and then rerun the Promote step. The Onboarding wizard runs the promotion only against the records that returned exceptions.
Reviewing and promoting data
When you finish Phase 1, click Finalize Review and Activate to move into Phase 2 and open Data Visualizer. From Data Visualizer, you can review the data that you entered to ensure it is correct before you promote it. If you discover a problem in a record, you can edit the record directly in Data Visualizer. You can also make bulk edits in the Data Visualizer by using the Modify All feature. For example, if you notice that one of the Assigned Group names has a spelling mistake and the mistake is repeated in all of the Assignment records, you can use the Modify All feature to update all of the affected records in a single operation.
The Modify All feature allows 100 entries to be updated at a time.
Data Visualizer displays the following data types in columns, grouped under the headings shown in the following table.
If there is a large volume of data, you can limit the number of records displayed in the columns by using the filters at the top of each column.
The filters cascade downward in the data hierarchy. If, for example, you filter by company, the rest of the records shown in Data Visualizer belong only to that company. If multiple filters appear at the top of a column, you can filter on multiple conditions.
The People Template Permission Groups table is not affected by the Company filter. By default, this Permission Groups table shows all of the available People Template permission groups. You can, however, apply filters within the People Template Permission Groups table that let you control which permission groups appear in the table.
Use these steps to review and, if necessary, update the data.
To review and promote data
- When you complete Phase 1, click Finalize Review and Activate.
- Select how to review and activate the data: Manually or Automatically. This action opens the Data Visualizer.
- In Data Visualizer, the Organizational Structures column is open by default. To expand other columns, click the arrow on the column panel.
- Review the data to ensure that it is correct.
- If you find a problem with a record, double-click the record to open the data-editing window.
- Correct the problem, then click Save to close the window.
- Perform one of the following actions, after which the wizard Displays the Activation screen:
- If you selected Manually in Step 2, click Validate. The Onboarding wizard runs the validation tests, but does not promote any data until you tell it to.
- If you selected Automatically in Step 2, click Promote. The Onboarding wizard runs the validation tests and automatically promotes valid data to the production database.
- If the Activation screen shows that there are no data exceptions, perform one of the following actions:
- If you selected Manually in Step 2, click Promote to promote the data to the production forms and finish the job.
- If you selected Automatically in Step 2, you are finished the job.
- If the Activation screen shows that there are exceptions, you must perform Exception handling.
If you receive a message related to the number of available licenses during the promotion or validation steps, see License check.
Use these steps to correct any data exceptions that might occur during the Activate step.
To correct data exceptions
- On the Activation screen, click Exceptions to open a list of records in the Data Visualizer that have data exceptions. This list also provides information about why the record caused the data exception. For detailed information about diagnosing a data exception, see Diagnosing data exceptions.
Open the data editing window by performing one of the following actions:
To edit a single record, double-click the record.
- To edit multiple records, select the records that you want to update, then double-click the selection (for People and Assignment records only).
- To edit all records in single data structure (for example, all Company records), click Modify All (for People and Assignment records only).
Using the Data Editing window, correct the data exception, then click Save to close the window. The Onboarding wizard removes the corrected records from the Data Visualizer.
If you later want to see the records that you updated, click Corrected. The wizard redisplays the corrected records that previously were removed from view. See Controlling which records appear in the Data Visualizer for more information about how to control which records appear in the Data Visualizer.
Depending on how you chose to review and promote the data (as described in To review and promote data), click of the following buttons:
Validate — Manual, validates the corrected data and then waits for you to promote it.
Promote — Automatic, validates the corrected data and automatically promotes data that passes validation.
- If the validation test finds more data exceptions, repeat Exception handling iteratively, until either:
- The validation checks no longer find data exceptions
- You choose to ignore the data exceptions and start a new onboarding session
- You choose to delete the records that are causing the data exceptions and address those issues at a later time
See Running a new onboarding session, Starting a new onboarding session, and Deleting records for information about the different options for handling data exceptions.
Diagnosing data exceptions
Data exceptions have associated codes and messages that can help you to diagnose the cause of an exception.
To view the code and message for a specific record, select the record in the Data Visualizer. The Onboarding wizard displays the associated code and message in the Exception Messages panel on the right side of the Data Visualizer. The code and the message appear under separate headings: Exception Codes and Exception Message.
If a data exception occurs during the Promotion step, you will also see a heading for Exception Text, which contains information related to the data-promotion text.
The information under the Exception Message heading concatenates the codes and the messages.
If multiple reasons for the exception exist, the Data Visualizer reports all of the related codes in a semicolon-separated list under the Exception Codes heading. The associated message text also appears in a semicolon-separated list under the Exception Message heading.
The exception messages always appear in the same order as the codes.
The Exception console is another tool that you can use to view and correct data exceptions. The Exception console organizes the exceptions in a tree format grouped by exception type. After you correct the exception, the Exception console stops displaying the record.
To use the Exception console
- Select an exception type from the navigation tree.
Selecting an exception type lists all of the records with that type of data exception in the table.
- From the table, select one or more rows. By selecting multiple rows, you can perform bulk updates.
- Use the editing panel below the table, perform one of the following actions:
Update the field that contains the data exception with the correct content, then click Update Staging. If you selected more than one row in the preceding step, the change that you make is applied to all of the selected records.
ImportantThere is a pair of icons to the right of each field:
- Using the left icon of the pair, you can expand the text editing field to make it easier to edit very large text entries.
- Using the right icon of the pair deletes the field contents and inserts the string NULL. Insert NULL w hen you specifically want to empty the contents of any given field. Placing NULL in the field ensures that the system interprets the field as empty.
- Delete the selected records by clicking Delete Staging.
Controlling which records appear in the Data Visualizer
These buttons are available in Data Visualizer to help you control the records that it displays. The buttons that you see at any given time depend on what you are doing and where you are in the Review and Activate steps.
|Returns you to the Job Status screen. Available from the Review screen during the Reviewing step.|
|Removes all of the filters from the columns. Available at any time.|
|Refreshes the records shown in the Data Visualizer to show their current state. Always available.|
|Use this button to show only records that have not been validated or activated. Available before validating or promoting data.|
|Use this button to show records that contain data exceptions. Available after you validate (or validate and activate) the data.|
|Takes you to the Exception console . Available only when data exceptions exist.|
|Redisplays corrected records (when you correct a record that had a validation exception, the wizard otherwise removes the corrected record from view). Available after you validate (or validate and activate) the data.|
|Displays only validated data in the Data Visualizer. Available at any time after the Review step.|
|Displays the records that you activated (pushed out to the production or test database). When you click Promote , neither the edit nor the delete function is available in the data columns. Available after you validate and activate data.|
|Displays records related to the current onboarding job. When you click All Data, neither the edit nor the delete function is available in the data columns. Available at any time after the Review phase.|
If you need to delete records from the Data Visualizer because they contain nonfatal data exceptions or because you chose to address the data exceptions related to those records at a later time, use the delete icons at the bottom of each column.
- Delete Selected deletes the record or records that you select in the column.
- Delete All deletes all of the records in the column.
Running a new onboarding session
When you finish promoting the data from an onboarding session and you are still logged on to the Onboarding Wizard, you can easily start another onboarding session with freshly imported data.
To start a new onboarding session, click New to return to the Phase 1 screen and to clear from view all data from the previous onboarding session.
- The New button does not delete the data from the staging forms used in Phase 1; it only removes the data from view. The data in the staging forms is kept in the staging forms according to value specified in the Keep Promoted Staging Records field in the Data Management application preferences.
- Data that has been activated using the Data Wizard is unaffected by the New button
See the Starting a new onboarding session use cases for more information.
Troubleshooting a job
If you think that your job might have stopped unexpectedly and you want to view details about its progress, you can check the job in the Data Management Job console. By looking at the job in the console, you can see information related to its progress. If the job has stopped unexpectedly, you can see, for example, on which form the job stopped and at which stage, either Validation or Promotion.
To look at a job in the Data Management Job console
- From the Applications list on the IT Home page, select Data Management > Job Console.
- On the Job Console, select the required job in the Jobs table and click View.
Troubleshooting an Onboarding Wizard job that uses the LDAP import method
If you are running an Onboarding Wizard job that uses the LDAP import method, the Data Management Job console entry for the job will not contain any information related to the LDAP aspect of the job.To perform troubleshooting on a job that uses the LDAP import method, click the Show log link, as shown in the following illustration.