Monitoring and managing pending distributed operations

When the DSO server retrieves a list of pending items to process from the distributed pending queue, the list sometimes includes duplicate items (items that have the same request ID for the same form). For example, suppose a filter containing a DSO action executes on Submit and Modify operations. If the following sequence of events occurs, two pending items for the same Help Desk request are added to the distributed pending queue:

1. A user submits a request through the "Help Desk" form.
2. A Push Fields action transfers some data in the request to another form.
3. The data transfer triggers a Modify operation on the Help Desk form.
4. The Submit operation on the Help Desk form that began in step 1 is completed.

This example adds two pending items to the distributed pending queue (one in step 3 and one in step 4) for the same request in the Help Desk form. In such situations, the DSO server performs the distributed operation specified in only the most recent duplicate item (in this example, the pending item created in step 4). For each of the other duplicate items, it adds the following entry to its log file and then deletes the item:

For information about DSO logging, see Configuring-Distributed-Server-Option-logging.

To view items in the pending distributed queue on your local AR System server

1. In the left pane of the AR System Administration Console, click System > Distributed Server Option > Pending DSO Operations.

2. In the Distributed Pending form, enter the appropriate search criteria, and click Search.
   For each item that appears in the results list, the form includes this information:

   Field	Description
   Pending ID	ID of the item.
   Pending Type	Distributed operation type (transfer, update, return, or delete).
   Form	Name of the source form.
   Source ID	ID of the source request.
   Other	String that specifies additional information about the distributed operation. The string includes one or more of these parameters: -e—ID of the target request in a distributed delete operation. -E—ID of the source request in a distributed delete operation. -o—Flag indicating a limit override in which the DSO server cannot initiate a transfer or return operation. -p—Name of the pool to which a distributed delete operation is assigned. -s—Name of the form involved in a distributed delete operation. -x—Name of the server involved in a distributed delete operation.
   Pool	Name of the pool to which the distributed transfer, update, or return operation is assigned.
   Status	Status of the item.

To remove items from the distributed pending queue

1. In the left pane of the AR System Administration Console, click System > Distributed Server Option > Pending DSO Operations.
2. In the Distributed Pending form, enter the appropriate search criteria, and click Search.
3. In the results list, select the item to remove.
4. Select Actions > Delete.
5. Select View > Refresh Search to update the list of items.

To log failed pending items

You can log failed pending items in the Distributed Pending Errors form. The form includes this information:

• The error message that is most relevant to the failure
• The operational stage in which the failure occurred

The form is available only on AR System server that have a license for DSO.

For more information about DSO logging, see Configuring-Distributed-Server-Option-logging.

1. Open the AR System Administration: Server Information form for the local server.
2. On the DSO tab, select Log Errors to DSO Pending Errors Form.
   
3. Click OK or Apply.
   Your changes take effect immediately. You do not have to restart the AR System server.

To retry failed pending items

1. Open the Distributed Pending Errors form in Search mode.
2. Enter the appropriate search criteria, and click Search.
3. In the results list, select one or more pending items to retry.
4. Perform one of these procedures:
   If you selected only one item in step 3:
   1. Select Retry.
   2. Click Save.
      If you selected multiple items in step 3:
   3. Select Actions > Modify All.
      The Details pane changes to Modify All mode, and a blank form is displayed.
   4. In the blank form, select Retry.
   5. Click Save.
   6. In the confirmation message, click Yes.
      The specified items are removed from the Distributed Pending Errors form and re-added to the Distributed Pending form. They are retried as follows:
      • If a polling interval is set for the local DSO server, the item is retried when the polling interval expires.
      • If the item contains distributed pool information and the pool has a polling interval, the item is retried when the pool polling interval expires. (See Creating-and-modifying-distributed-pools.)
      • All other items are retried when the backup polling interval expires or when a new pending item is created, whichever occurs first.

How errors affect pending items

The DSO server reads items from the distributed pending queue and performs the specified operation (transfer, update, return, or delete) for each item. By default, the DSO server checks for new pending items at the following times:

• When an internal signal mechanism triggered by workflow alerts the server.
• Automatically at two minutes after every hour, ensuring that all pending items are processed if the internal signal mechanism fails.

For information about when polling DSO servers and pools check for requests, see these sections:

• Creating-and-modifying-distributed-pools

If an item succeeds or if it fails with a nonrecoverable error, it is removed from the distributed pending queue. (You can configure DSO to move pending items that fail with nonrecoverable errors to the Distributed Pending Errors form. See the section To log failed pending items in this topic.)

By default, if a pending item fails with a recoverable error, such as an unavailable target server, DSO periodically retries the item as follows:

• Every 5 minutes for the first 30 minutes
• Every 30 minutes after the initial 30 minutes for up to 24 hours
• Every hour after the first 24 hours

To specify whether the Status value of failed items remains set to New or is changed to Retry, set the Mark Pending Retry field in the AR System Administration: Server Information form. (See the definition for "Mark Pending Retry" field in AR System Administration - Server Information form - DSO tab.)

To limit the number of times the item is retried, change the default Retries value in the distributed mapping. See the definition for "Retries" in Creating-distributed-mappings.

If you specify values in the Retries "Try for Maximum of Hours/Minutes " option and the specified time limit elapses, the item is removed from the distributed pending queue. This occurs even if one or more retries were not attempted. In this case, if the item includes a Transfer Status or Update Status field, it is set to Timeout.

By default, the system allows three minutes of connection time for processing each distributed operation. This might be an insufficient amount of time in some situations and might cause pending distributed operations to fail. See Configuring-the-RPC-timeout-setting.