Troubleshooting dataload issues

While running data load operations, you might encounter certain issues such as jobs stuck in the In Progress state or jobs end with error detected job result. Ensure the correct entries in the spreadsheet data and Atrium Integrator Job list, multiple concurrent jobs are not running on a server, and the settings are configured correctly. Use this information to understand and resolve the issues.


Best Practices
When running data load operations, we recommend that you follow these tips:

  • If a job fails in the load step, you cannot run that job again. Cancel the job and run this job as a new job.
  • Make sure the environment is configured correctly for Unified Data Management (UDM). For example, the load balance configurations and data paths are configured correctly.
  • After you run the first UDM job, ensure that the DMT:SYS:SequencingEngine has all the sequence records.
  • Clean up the UDM variable and load forms such as the CTM:LoadPeople records, DMT:ThreadManager, and CAI:Events forms periodically, especially in case of large volumes.
  • Enable and run escalations on your primary server. For more information, see Knowledge Article number 000215041 (Support logon ID required).

Troubleshooting load jobs that are stuck in the In Progress state

Issue symptom

Jobs are stuck in the In Progress state.

Issue scope

This issue occurs while running a dataload job.

Resolution

  1. Make sure the following escalations are running on the same pool and are not shared with the other running escalations:
    1. DMT:DJS:SetStatus—This escalation triggers every one minute and sets the status of the z1D_Action field to CHECKTRANSSTATUS.
    2. Filter DMT:DJS:ChkStepCompleted—This escalation triggers after the z1D_Action field value is set.
      For more information about escalations pools, see Setting escalation pools.
  2. Ensure that the custom Pentaho job is configured correctly.
    For more information, see the following documentation and BMC Communities pages:

Troubleshooting configuration issues when loading task group templates

Issue symptom

The Task Template is imported and displayed in Available Task Groups, but it is not added in Current Associations section on Task Group Template form.

Issue scope

This issue occurs when you perform the following tasks:

  1. From the Applications list on the IT Home page, select Data Management > Job Console, import new task group template with new tasks that are created in Task_Process_Setup.xlsm spreadsheet. 
  2. After the data is loaded successfully, you add a new task template record in Task_Process_Setup.xlsm spreadsheet.
  3. Create a new job and import the updated Task_Process_Setup.xlsm spreadsheet.

Resolution

When you add a new task template in the Task Group that is already imported, make sure you add the Instance ID in the Parent - Template Identifier* column on the TMS_LoadAssociationTemplate tab for the new task record.

You can get the Instance ID by generating the report for this job. For more information about creating and generating a report, see Generating-a-report-based-on-the-search-results.

Troubleshooting load data errors

Issue symptom

The Load Data Errors field in the Step Details information (Step window) for the Load step displays the errors related to loading the data. The load errors are also displayed when Atrium Integrator has encountered an error. Errors range from a systematic Atrium Integrator issue (for example, connectivity issues) to an issue caused when the inbound data is not formatted as required (for example, invalid date formats in the dataload spreadsheets). Click View/Save Compressed Error File to see the errors or save them. 

(Click the image to expand it.)

 g_itsm_udm_load_errors.png

Issue scope

This issue occurs while running a dataload job.

Resolution

When a transformation (containing error handling) has trapped an error for specific data in the Spreadsheet tab or when a required field is missing or you have an incorrect selection of field values in your dataload spreadsheets. Each exception records the row, sheet, and the trapped error message and sends the output to a spreadsheet. Each Spreadsheet tab is included in its own Microsoft Excel file. You can open the spreadsheet to identify the row and the sheet that displays the error and also identify what section in the spreadsheet had incorrect data. 

Before performing any other troubleshooting procedures, check that the rows in the named spreadsheet have the correct cell formatting applied, as shown in the following list:

  • Row 1—General
  • All other rows—Text

Important

Errors related to incorrect spreadsheet cell formatting are displayed especially if you have copied and pasted information from one spreadsheet to another.

The level of detail provided in the Load Data Errors field is based on the log level specified in the Atrium Integrator jobs and from the Job Console. If you are a DMT administrator, to help you determine why load issues are occurring, alter the log level details to something more informational (for example, Detailed, Debug, Rowlevel, and so on) so that additional details can be passed back to the step load error.

If you are using the Debug level, we recommend that you perform the following actions:

  • To minimize the volume of data and determine what the warning message is, copy the required job and use only a subset of the data in the spreadsheet.
  • Always reset this log level back to its normal value of Minimal to reduce future performance concerns.
  • Ensure that users running UDM jobs or scheduled jobs are not loading data on the computer while you are performing the debugging process. It is easier to work with smaller sets of data when you are identifying issues. We recommend that you perform the debug process during off business hours.

Tips

  • If the load step is stuck in an In Progress state, verify that you have not selected the Disable Escalations server option on the AR System Administration Console under Configuration tab > System > General > Server Information.
  • While running a data load job for a big number of asset records, you must set the composite index of the following fields for a faster data load:
    • Company (Attribute added by ITSM extension)
    • Name
    • Serial Number

Troubleshooting AR System or plug-in errors

Issue symptom

The data load step fails with  Status as Error and a Job Result as Errors Detected .

Issue scope

When AR System or plug-in  related errors occur, the dataload job fails.

Resolution

The system administrator must resolve such errors and then create a new dataload job. For information about creating a new job, see Creating-a-data-load-job and for information about running a new job, see Running-and-monitoring-data-load-jobs .

(Click the image to expand it.)

Load error.png

Troubleshooting unique index violations error

Issue symptom

The ARERR 382 unique index violations error is displayed.

Issue scope

This issue occurs while running a dataload job.

Resolution

This error usually occurs when you import records with ID numbers that are numerically higher than the IDs that are currently assigned by your system. For information about the steps to be followed, see Importing-records-with-higher-ID-numbers-than-those-currently-issued-by-system.

Troubleshooting Carte server related errors

Issue symptom

The Load step in the Steps panel captures and displays the errors for the following Carte server issues:

  • The system is unable to establish a connection with the Carte server.
  • The Carte server restarts when a dataload job is run.

Resolution

When such errors occur, the Load step is completed with the Status as Completed and Job Result as Errors Detected. As a system administrator, you must review the spreadsheet to resolve such errors and then run the job again or create a new dataload job.

If there are errors on the Load step, you cannot run the job again. You must create a new job.

Resolutions for common errors

The following table lists the common errors, cause, and their workaround:

Issue symptom

Resolution

If you run a data management job by using the BMC template, and you attach the data management spreadsheet to the job. The data management job remains in the In Progress status or the following error is displayed:

You are unable to attach the spreadsheet to the job.
  1. Clean the browser cache and close the current browser session.
  2. Open the job console in a new browser session and run the job again.

If the plugin server is not available while the Data Management job is running at the Promote step. The Data Management job remains stuck at the Promote step. The step does not display any error message.

Extract the data that is not processed, create a new job with this data by using the correct spreadsheet, and run the job.

The load jobs point to the load balancer and a timeout connection error is displayed with the server name in the arjavaplugin.log file.

Verify that the load balance server is listed in the UDM:RAppPassword form and the Microsoft Windows HOST file contains the following parameters:

  • AR Server IP servergroupname
  • AR Server IP servergroupname.domain.net

In the Onboarding Wizard, when you click a step before the data in the table for the previous step is loaded, a caught exception error is displayed.

Make sure you wait for the data from the previous step to get loaded before you click the next step.

In case the error is displayed, click OK in the error dialog box.

The following error messages are displayed:

No file(s) specified!
Stop processing.
COM_LoadCompany.0 -
Error initializing step

[COM_LoadCompany] COM_LoadCompanyAlias.0 -
ERROR: No file(s)
specified! Stop processing.

COM_LoadCompanyAlias.0 -
Error initializing step
[COM_LoadCompanyAlias]
  1. In a working environment, run the following advanced search on the DMT:TransformationParam form to display the records related to this variable for the Atrium Integrator jobs:
    'Variable Name' = "DMTATTACHPATH"
  2. Export and then import the missing records from a working environment to a non-working environment.

The following error message is displayed:

CI-APP-AROutput.0 - Impact String :
couldn't convert String to Integer
CI-APP-AROutput.0 - Unparseable number: " 2000.0"

Verify this issue on the text fields that display a small green triangle. If you do not see the triangle, click the field to set the integer.

The load step is skipped.

Check the load step or as the DMT administrator, from the Job Console, check for a valid entry in the Atrium Integrator job list. If no data was loaded for this step or for the related validate and promote steps, ensure that you configure Atrium Integrator. For more information, see Transferring data from external data stores to BMC Helix CMDB.

The following error message is displayed:

Data management process flow is currently being initialized or rebuilt, this will finish shortly, please run the job in a few minutes.

For more information, see Data-Management-error-messages.

The load step stays in the In Progress state even though all the data is on the staging form in the Done state.

  1. As an administrator, check that the escalation server is turned on though the Server Information form.
  2. Query the UDM:ExecutionStatus form where the execution instance name is the Instance ID of the load step.
    If you do not find any record, the Atrium Integrator engine (custom AI Job/Transformation) did not get called.

The following error message is displayed:

java.lang.OutOfMemoryError: Java heap space

Ensure that multiple concurrent jobs are not running on the Carte server.

Increase the 1GB (default value) of heap space to add more concurrent number of jobs.

The following error messages are displayed when performing a search:

Error in plugin" Get List Entry
With Fields not supported on form
UDM:Execution (ARERR 8753) after
Load Balance name change.
Error in plugin : Invalid Execution
Instance. Execution Instance
AGHAA5V0GG40RAN7D2XYEGI6DXP3N5
does not exist or user Remedy
Application Service is not
allowed to access it.

(ARERR 8753) An application
command failed

(ARERR 4554) Application-Delete-Entry
"DMT:Action"

  • Verify that the server name is correct in the ar.cfg file.
  • Verify that the UDM load path is accessible.
  • Add the following load balance names to the Microsoft Windows HOST file:
    • AR Server IP servergroupname
    • AR Server IP servergroupname.domain.net

The following error messages are displayed:

Failed to initialize at least one step.
Execution can not begin!
No file(s) specified! Stop processing.
Error initializing step
[CTM_LoadPeopleModification
ERROR: Source folder/file [/opt/bmc/ARSystem/db/UDM/DJBnumber]
can not be found!

Verify that the UDM load path is accessible and that the path is a shared path with read and write access.

The Carte server stops responding and the following error message is displayed:

<ERROR> <ARDBCPluginRepository > < ARDBCPluginRepository.java:184 >
createEntry() FAILs in plugin:
ARSYS.ARDBC.PENTAHO
  1. Verify that the Carte server is running by entering the following URL:
    http://<carte host>:20000
  2. In the Windows Security window, enter the AR System server user name and password.
    From the current state of the jobs running on that Carte server that are displayed, if the server is down, the browser displays a message that the server cannot be connected.

The following exception message is displayed:

Exception in thread "main" java.lang.IllegalArgumentException: Malformed
\uxxxx encoding

Correct the path and restart AR System server.

The Carte process does not recognize the "\u" entry in the ar.conf file.

For example: BMC Software\ARSystem\ARServer\
Db\userupload.log.

The following error message is displayed:

92: "Timeout during database update --
the operation has been accepted
by the server and will usually
complete successfully",
"servername:32825 ONC/RPC call timed out";

Verify that the CAI plugin registry has the correct settings. The filter timeout value is not set to very low. Additionally, increase the AR System server timeout settings.

The following file does not open:

//servername/D$/Workspace/UDM/
DJB000000003317/cicmdbfile.txt

If the file is in a server group environment, verify if the servers are communicating with each other and if the file has a read and write access to the UDM path set during the configuration.

For more information, see the Knowledge Base article number 000097446.

The following error message is displayed:

12116: You do not have access
to modify the Company information
supplied. You must either have the Company added to your Access Restrictions, or have Unrestricted Access set to Yes in your People profile.

Comment out the AR_USER parameter in the kettle.properties file. Then, restart the Atrium Integrator.

The following error message is displayed:

Error Connecting to ARSystem 
Error while impersonating [my user name] Cannot establish a connection to the ARSystem server [admin server alias]:[TCP]

Add the alias name to the hosts file to loop back to localhost. See, KA000132182.

Additional information

Verify the following additional settings.

Settings

Description

Configuring the UDM by following the recommendations
and enabling the Data Management preferences for the dataload jobs.

Configuring the load file path for the AR System
attachments and spreadsheets repository.

CAI Plug-in Registry settings

  • Set the total threads value:
    • 3 - 6 for normal production server data loads
    • 8 or 12 for bulk data loads (if you are onboarding as an example)
  • Set the following private queue values:
    • 390620 - 390625
    • 390627 - 390629

(Click the image to expand it.)

CAI plugin registry.png

Confirm that the forms have the correct server name.

  • UDM:ExecutionInstance
  • UDM:PermissionInfo
  • UDM:Config
  • UDM:RAppPassword

Gathering logs

Review the following log files for load error details:

Log file name

Location

arcarte.log

C:\Program Files\BMC Software\ARSystem\Arserver\Db directory.

arjavaplugin.log

C:\Program Files\BMC Software\ARSystem\Arserver\Db directory.

Pentaho plugin

For information about the location, see Troubleshooting-Pentaho-plug-in-issues.


 

Tip: For faster searching, add an asterisk to the end of your partial query. Example: cert*