Troubleshooting fulfillment requests and CAI plug-in issues

In BMC Service Request Management, service requests generate fulfillment requests, such as incidents, change requests, and work orders by using the CAI plug-in interface forms. The CAI plug-in, which is related to the Remedy AR System server Java plug-in process, creates the fulfillment requests and syncs the status and the data between service requests and their associated fulfillment requests. When you create a fulfillment record, the entries use the SRM_OUT_CREATE_APP_REQUEST outbound command in the Event field on CAI:Events form.

When you are creating fulfillment requests, some issues related to CAI:Events records can occur.

Issue symptoms

The following are some of the issues related to the CAI:Events records:

The records are not processed until the plug-in server process is restarted.
The records process slowly.
The records are stuck in the Running status.
The records display the Error status and fulfillment tickets are not generated.

Issue scope

When there is an issue with the CAI plug-in.
When CAI:Events is not able to process the record or when the data is missing.
When the private queue is not configured for the CAI plug-in as the plug-in uses a FAST queue and shares the resources with other Remedy AR System server activities.
When there are a large number of records in the SRM:Variable table.
When multiple fulfillment record creation attempts are made by using the same InstanceId (Field 179) value.
When Remedy AR System server issues occur, for example, resource, database, network, or configuration issues.
When issues related to a particular SRD, approvals, CAI plug-in, or data occur.

Diagnosing and reporting an issue

After you identify the symptoms and scope of the issue, use the troubleshooting guide to help diagnose and resolve the issue or to create a BMC Support case.

Action	Steps
Troubleshoot	Check the following configuration settings: Check if the CAIPlugin.jar file exists in the <ARInstallationFolder>\pluginsvr\cai directory. This file connects to the Remedy AR System server by using the CAI Events process. Check the AR System Administration: Plugin Server Configuration form for the configuration information of the CAI plug-in, such as the path elements and make sure that the information is correct. Check the plug-in information settings in the following forms as the CAI plug-in receives the configuration information from these forms: The CAI Application Registry form—Defines the integrating applications, interface forms, logon information, and the local or remote plug-in location. The CAI Plugin Registry form—Enables you to define a private server queue that is used for the CAI AR System server API calls. The AR System Administration: AR System Configuration Generic UI form—Contains the configuration settings that refer to the private server queue defined in the CAI Plugin Registry. The AR System Administration: AR System Configuration Generic UI form—Includes the Server-Plugin-Alias setting (REMEDY.ARF.CAI REMEDY.ARF.CAI myServer:9999) that points to the correct plug-in server alias. For more information, see CAI plug-in configuration.
Gather the log files	For information about gathering the log files, see Troubleshooting-functional-areas-of-the-application.

Action

Steps

Troubleshoot

Check the following configuration settings:

Check if the CAIPlugin.jar file exists in the <ARInstallationFolder>\pluginsvr\cai directory.
This file connects to the Remedy AR System server by using the CAI Events process.
Check the AR System Administration: Plugin Server Configuration form for the configuration information of the CAI plug-in, such as the path elements and make sure that the information is correct.

Check the plug-in information settings in the following forms as the CAI plug-in receives the configuration information from these forms:
- The CAI Application Registry form—Defines the integrating applications, interface forms, logon information, and the local or remote plug-in location.
- The CAI Plugin Registry form—Enables you to define a private server queue that is used for the CAI AR System server API calls.
- The AR System Administration: AR System Configuration Generic UI form—Contains the configuration settings that refer to the private server queue defined in the CAI Plugin Registry.
- The AR System Administration: AR System Configuration Generic UI form—Includes the Server-Plugin-Alias setting (REMEDY.ARF.CAI REMEDY.ARF.CAI myServer:9999) that points to the correct plug-in server alias.

For more information, see CAI plug-in configuration.

Gather the log files

For information about gathering the log files, see Troubleshooting-functional-areas-of-the-application.

Resolutions for common issues

Symptom	Scope	Workaround
The fulfillment creation process fails, and the following error messages are displayed in the failed CAI:Events record. ARERR 306: Value does not fall within the limits specified for the field. ARERR 326: Required field cannot be blank. ARERR 44699: No group found using automated routing you need to manually select a group ARERR 1440853: No requirement manager group was found. Manually select a group from the menu. If no groups are defined, contact the administrator. ARERR 1291045: Operational categorization information is invalid for the specified Enterprise Use the menus to select this information.	When the mappings are incorrect or when there is missing data in the mappings.	Update the mapping in the related Service Request Definition (SRD). To find the corresponding Service Request ID for a stuck record in CAI:Events form: Open the CAI:Event form and select any entry that is stuck. Copy the Event GUID. Open the SRM:App Instance Bridge form and search with zTmpEventGUID = Event GUID. For more information, see the KA000149164 knowledge base article. Validate the variables defined in the process definition templates and their relationships to the exposed fields in the application object template. Reprocess the failed CAI:Events records by updating incorrect or missing values and starting the process again. Check the error message in the CAI:Events form and then capture the Filter, SQL, or API logs for review and update the data according to the error. The [confluence_iframe] macro is a standalone macro and it cannot be used inline. Click on this message for details. This macro generates standalone content. As a consequence you need to make sure to use a syntax that separates your macro from the content before and after it so that it's on a line by itself. For example in XWiki Syntax 2.0+ this means having 2 newline characters (a.k.a line breaks) separating your macro from the content before and after it. https://youtu.be/GPXb-Wrambg
Performance issues such as the fulfillment creation is slow. For example, records in the CAI:Events record stay in a running state for a long time, or the Remedy AR System server timeout error messages are displayed for the failed CAI:Events records.	When the private queue is not configured for the CAI plug-in as the plug-in uses a FAST queue and shares the resources with other Remedy AR System server activities.	Create a private queue for CAI. For more information, see Configuring the CAI Plugin Registry form.
The fulfillment records are created with a delay or not created at all.	When there are a large number of records in the SRM:Variable table.	Archive and clean up any records that are no longer required, based on the age and the request status. For more information, see Managing the Archiving process policies. For information about issues with the Process Definition Template (PDTs), see Troubleshooting-the-creation-of-AOTs-PDTs-and-SRDs.
There are issues with the CAI plug-in processing. For example, the following error messages are displayed in the arerror.log or the arjavaplugin.log files with similar timestamps as the CAI plug-in issue: SLMCS : Timeout during data retrieval due to busy server -- retry the operation (arservername) ARERR - 93 AssignEng:Timeout during database update -- the operation has been accepted by the server and will usually complete successfully (arservername) ARERR – 92 Dispatch : Operation cancelled due to error (ARAPPERR 4502)	When the Remedy AR System server issues occur. For example, resource, database, network, or configuration issues.	Check if there are any recent changes related to the Remedy AR System server, operating system level (patches), network, or database. For errors related to the server connection, verify the Source Server and Target Server field values in the CAI:Events form. In the server group environment, the LBServer.domain.individualserverdomain is appended in the Source Server and Target Server field values. Append this entry in the host file of all the servers and retry the CAI plug-in processing.
The service request is in the Submitted status, and the fulfillment record is not created.	When issues related to a particular SRD, approvals, a CAI plug-in or data occur.	Contact BMC Customer Support.
The service request is in the Initiated status and the fulfillment record or the CAI:Events record is not created.	When there is an issue with the CAI plug-in.	Capture the plug-in related log files and contact BMC Customer Support.
The service request is in the Initiated or in the In Progress status for a multi-step process and the CAI:Events record is not created when there are CAI:EventParams records.	When the CAI:Events is not able to process the record or when the data is missing.	Follow the steps documented in the Service Request fulfillment request not created, but records for it exist in CAI:EventParams article on BMC Communities.