This version of the software is currently available only to customers in the Controlled Availability (CA) program.

Troubleshooting data import issues


This topic lists the issues that you might encounter while importing data using BMC Helix Data Manager. See the guidelines in this topic to obtain the appropriate logging and troubleshooting steps. 


Isuse symptoms

You might  encounter the following errors or failures while importing data using BMC Helix Data Manager: 

  • Migration jobs are not created successfully.
  • T table data import fails.
  • Data import fails as the value is too long for character-type data.
  • Data import fails due to field length mismatch.
  • Data import fails due to data type error.
  • Data import fails due to a unique key violation error.


Issue scope

The data import issues might occur for the following use cases:

  • On-premises to on-premises data migration.
  • On-premises to SaaS data migration.


Resolution

Verify the checks and troubleshoot the issue that you have faced.

Important

You must follow the steps described in the table in the specified order.  

Review the tasks in the table before looking for additional details under the  Common cause and possible solution  section.

Step

Task

Description

1

Review the source and target system configurations.

  1. Review the source and target system configurations by performing the following tasks:
    1. Log in to BMC Helix Data Manager.
    2. On the System Configuration page, select Source or Target System Configuration.
    3. Under Action, select Test System Connectivity.

      connectivity (1).PNG

For more information, see Registering-source-and-target-systems.

2. Review the defined file system connections by performing the following tasks:


    1. Confirm that the default connection path is correct.
      1. Log in to BMC Helix Data Manager.
      2. Select Configuration > Configure File System connections.
        Then, select Defined File Systems > Review Path.
    2. Confirm that the Test Write Data and Test Read Data & Files are successful.
      1. Under Configure, select the correct Source and Target Systems > Configure Connections.
      2. Validate Test Write Data and Test Read Data & Files.

               image-2024-5-3_20-57-20.png

              Test Export.png

              image-2024-5-3_20-53-26.png

               Test Import.png

For more information, see Validating-file-system-configuration.

3. Review the discovered Data Dictionary from the source and target system, and make sure that the Forms & Fields tab shows a valid list of forms (Base, Custom, Overlay).


    1. Log in to BMC Helix Data Manager.
    2. From Discovery & Analysis, select the Discovered Dictionary.

        Discovered DD.PNG

    3. Under Forms & Fields, select the correctly labeled and dated Data Dictionary and review the Custom, Base, and Overlay forms list.
      DD.png

 For more information, see Discovering-data-dictionaries.

4. To review if the version of the Migration Pack that is imported and published is valid:


    1. Log in to BMC Helix Data Manager.
    2. Select Data Migration > Migration Packs.
    3. Confirm that the Migration Pack Name and its Status are correct.
    4. Review the Included Forms tab to confirm that the correct form list is displayed.
      image-2024-3-22_15-55-37.png

For more information, see Adding custom forms to a migration pack or Creating-a-migration-pack-for-your-systems

5.  When creating a Migration Pack, the BMC Helix Data Manager automatically creates data mappings between the source forms and target forms. If BMC Helix Data Manager is unable to create a mapping automatically, the fields will be flagged with a Data Mapping issue.

The Data Mapping issues may include:


    • New fields exist in the target system.
    • Differences in enum value exist between the source and the target system.
    • Field length differences exist between the source and the target systems.
    • Data type differences exist between the source and the target systems. 

To review the form mapping and address the highlighted issues:

  1. Log in to BMC Helix Data Manager.
  2. Select Data Migration > Data Mapping.
  3. To filter the mapping results, click Status, and then select the appropriate filter from the drop-down list.

    image-2024-5-31_15-53-55.png

  4. To get more information about the error, click Field Mapping
    image-2024-5-31_15-58-31.png

For more information, see Data mapping.


2

Enable and collect the logs.

BMC Helix Data Manager Logs are enabled under the BMC Helix Data Manager directory on both source and target systems.
Find the logs in the following path:
\BMC Software Install Directory\Helix Data Manager\Engine\log\hdm.log
.

hdm log.png

Check the hdm.log to verify whether the Date and Time details of when the activity was carried out are recorded.    

3

Create a BMC Support case.

Make sure you capture detailed information by using the following list while creating a case with BMC Support:

  • What is the version and name of the Migration Pack?
    For example, 20.02 to 21.3, ITSM or Smart IT application
  • What is the version and type of source and target database server?
  • What is the hotfix version of BMC Helix Data Manager tool on the source and target system?
  • What are the steps to reproduce the issue or error?
  • Is the migration activity performed on the source or target system?
  • Is the migration taking place from an on-premises to an on-premises system or from an on-premises to BMC Helix system?
  • Is the hdm.log captured as explained in step 2?
  • Are there any screenshots of errors that were encountered?


After you determine a specific symptom or error message, use the following table to identify the cause and possible solution:

Symptom

Cause

Action

Reference

Data import on a specific form fails with an error message

Detail:  The import fails on a specific form with a permission denied error.

Error:

Error while executing SQL org.postgresql.util.PSQLException:ERROR:

permission denied for table

To view the error:

  1. Click the Error tab to see the primary error (Unable to import T table data).
  2. Click the Advanced tab for additional details on the error.

image-2024-5-31_17-59-34.png 



  • BMC Helix Data Manager database users do not have complete access to BMC Helix Data Manager schema tables.

  • Custom forms are moved to the target system after integrating the BMC Helix Data Manager user database. This action leads to the BMC Helix Data Manager database user not having schema-level access to the particular form or table.
  1. With the help of a database administrator, get complete access to the BMC Helix Data Manager database user for the specific table for which the error is occurring.
  2. If you encounter too many tables with access issues, drop and recreate the database integration step.

    Database Integration.PNG

    or

If you do not have access to run the database integration from BMC Helix Data Manager, provide the database administrator with the manual integration scripts found at this location. 

scripts to recreate hdm user.png

Knowledge article:  000422374

Data import on a specific form fails with an exception

Detail: Data import fails on a specific form with an exception ( For input string: "abcd" ).
This message is one of the specific error messages associated with the symptom.

Error:

Fatal error processing T file at line 72559 with requestId of 1234
java.lang.NumberFormatException: For input string: "abcd" 
at
java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) ~[?:?]
at java.lang.Integer.parseInt(Integer.java:652) ~[?:?]


  • BMC Helix Data Manager flags potential migration issues when there are errors in parsing the records and a delimiter is present in the record. The delimiter splits the record as an additional field during import, which causes the error.

After importing the data, if you encounter an error or observe that the target system has extra fields, use a custom delimiter and rerun the export migration job on the forms.

 image-2024-5-2_21-54-23.png

ImportantWe recommend using a custom delimiter with a lower memory value. For example, :hdm:.

To run the export migration job by using a custom delimiter, see Running an export migration job page and set the Delimiter field value to a custom value.

Knowledge article:  000422456

Data import fails due to field length mismatch

Detail:  The data import fails with an error for a specific field in a form as the field length in the target system does not match the field length in the source system.

Error:

  • value too long for type character varying(15)
  • violates check constraint "len_c260131108“


  • The fields causing the error are customized in the source system, but the customizations are not migrated to the target system.

  • Data model differences, like new forms and fields, can cause mapping failures. For example, these failures are seen during the BMC Helix Service Management application upgrades with multiple new elements and changes to enumerated values.
  • Review the data in the specified field on the source system to determine if it is needed. If the data is needed, increase the field length on the target system by creating an overlay; otherwise, ignore the error.

Use the following query on the staging table or the source table to find the maximum length of a field and alter the length of that column on the target system.

SELECT max(LENGTH(<column_name>)) FROM<form_name>

<column_name> is the column name whose maximum length you want to find.

<form_name> is the name of the form where the column is present.

Knowledge article:  000422457

Data import fails due to unique key violation error

Detail: The data import fails when you select a migration job for data import.

Error:  

  • Error while executing SQL org.postgresql.util.PSQLException: ERROR: duplicate key value violates
    unique constraint "i<>_t<>" 
    Detail:  Key (c<>)=(1)  already exists
  • Example: Error while executing SQL org.postgresql.util.PSQLException:
    ERROR: duplicate key value violates unique constraint
    "i4909_1_536870913_t4909"
    Detail: Key (c536870913, c536870914)=(1, ITSM) already exists

To view the error:

  1. Under Errors & Warnings, select a migration job for data import.
  2. Click the Error tab to see the primary error (Unable to import T table data).
    image-2024-5-31_18-5-25.png
  1. Click the Advanced tab to get more information about the error.
    image-2024-5-31_18-6-46.png

            

  • Duplicate data exists between the source system and the target system.
    For example, the custom Group ID in the source system matches with the new out-of-the-box Group ID in the target system. This match causes an error as both the IDs must be unique but they are the same.
    For more details about the custom Group ID range to be used, see Creating and managing access control groups.

  • Out-of-the-box data with conflicting Request IDs exists on the source system.
    This use case applies if the source system data is imported using the Data Import Tool.
    A unique key violation occurs because the user might have selected the following options on the Data Import Tool:
    • Handle Duplicate Request IDs By field with the value Generate New ID for All Records.
    • Match Duplicate Request By field with the value Request ID.

image-2024-5-30_22-43-25.png

  • The issue can occur when you change the database from case-sensitive to case-insensitive mode. This use case applies when moving from Oracle case-sensitive to Oracle case-insensitive mode or when moving from Oracle database to Microsoft SQL Server or PostgreSQL database, resulting in a unique key violation.
    For Oracle database considerations, see the AR System case-insensitive option for Oracle section in Database considerations for Oracle.


To avoid the unique key violation due to:

  1. Duplication of data between the source system and the target system:
    1. Check and correct the data in the source system.
    2. Run another export job from the source system after the data is fixed and then import the data to the target system.

  2. Out-of-the-box data with conflicting Request IDs exist on the source system:

    1. Check and correct the data in the source system or add a prefix to the data before importing it into the target system to allow the import process.

    1. Run another export job from the source system after the data is fixed and then import the data to the target system.

  1. Changing the database from case-sensitive to case-insensitive mode, refer to the Database considerations for Oracle document and plan your next steps accordingly.


Data import fails due to Data Mapping error

Detail: Data Mapping errors can directly affect the import of specific form data. Therefore, resolving any Data Mapping errors and warnings identified after configuring the Migration Pack is necessary.

The Data Mapping errors are as follows:

  • Target Field not mapped: The source does not contain the corresponding field.
  • Source Field not mapped: The Source Field does not exist in the target.
  • Target Field too small: The Target Field is smaller than the Source Field.
  • Target Field required: The Source Field is optional, but the Target Field is required.
  • Enum warning: There is a mismatch in the enum text.
  • Customizations (overlays) or custom objects are not migrated to the target system before the Data Discovery is completed.

  • The Data Dictionary is not updated on the target system.

  • Field mapping differences exist between the source and target systemsThis scenario can occur when the fields differ due to the data model version.
    For example, the forms on the target system contain new fields not found in the source system.

Review the different Data Mapping Status to understand the errors:

  1. Target Field not mapped:  This issue occurs when the corresponding field is unavailable in the source system.
    In this case, generate a new field mapping or ignore it if not required.
    image-2024-3-25_14-1-8.png

  2. Source Field not mapped: This issue occurs when no corresponding field is available in the target system.
    In this case, generate a new field on the target system or ignore it if not required.
    image-2024-3-25_14-3-25.png

  3. Target Field required:  This status confirms that the Target Field is defined as Required, but the Source Field is Optional.
    In this case, manage the null value or set it to the default value.

  4. Enum warning: Review the data on the specified field on the source system and validate whether the data is required or ignore it if not required.
    image-2024-4-30_20-30-44.png

Additional steps:

  • Re-run the Data Dictionary again if any changes are made in the source or target system for the HDM tool to read the changes.
  • If new data is discovered, you must use the latest version to create a copy of the Migration Pack to include the changes while analyzing the Data Mapping.


For additional details, see Data-mapping

Data Import fails because the integer is out of range

Detail: The following error occurs because the time stamp integer value is out of range.

Error:

Error while executing SQL org postgresql.util.SQLException:
ERROR: integer out of range

                

              


  • This use case applies to Oracle and non-Oracle migrations where the Oracle time stamps are too wide for Microsoft SQL Server or PostgreSQL to accept.

  • The cause of this issue is third-party data integrations or custom data.
  1. Locate and fix the data in the source system.
  2. Locate the form and run the following query for negative values:

    SELECT * from <form_name> where <field_name> <0 or<field_name>>2147483647

    <form_name> is the name of the affected form.
    <field_name> is the name of the time stamp field that needs to be checked.

  3. If the data is fixed in the source, perform the data export and import again.
    Important: For an Integer out-of-range scenario, you must locate the field or the column where the out-of-range error occurs.
  4. Compare the specific field properties in the source system and the target system to validate whether there are any differences.
  5. If the data cannot be fixed on the source, raise a support case to work with BMC R&D or Support to resolve the issue.



 

Tip: For faster searching, add an asterisk to the end of your partial query. Example: cert*