AR System server queue wait times

A thread pool is a collection of worker threads that the AR System server uses to process concurrent client requests. The threads are allocated to queues from the thread pool on the per-need basis. A single thread from the pool can handle requests, such as client requests and scheduler requests (archiving, escalation, server statistics, and application statistics). 

BMC Remedy AR System API logging reports how long each API call waits before a thread starts to process the request.  BMC Remedy AR System Release notes and notices

The thread wait time is displayed after each call as follows: 

// :q : < wait time in seconds>

The following is an example log:

<API > <TID: 0000000338> <RPC ID: 0000012893> <Queue: List> <Client-RPC: 390620> <USER: Demo> <Overlay-Group: 1> /* Mon Jun 29 2015 23:40:00.8910 */ +ME      ARMergeEntry --  from Unidentified Client (protocol 21) at IP address 10.133.2.90 using RPC // :q:0.8s

Use BMC Remedy AR System API logging for the following situations:

• Users reporting slow response time for online transactions
• New implementations or upgrades during the performance benchmarking phase prior to placing the system in production
• Periodical check of the general health of the system

If the log shows sustained wait times of more than 0, increase the number of threads for the queue.

Increase the number of threads in the queue until your API log consistently shows thread wait times of 0. However, after CPU consumption on the AR System server reaches an average of approximately 70%, do not add more threads to the queue.

For list or private queues, a large result set from a query set might occur for every thread in the queue. For example, operations that consume a lot of memory (such as nested filters with push fields operations that update a large number of records) could occur on every thread in the queue. For this reason, memory is a consideration when increasing the thread count. Adding too many threads could cause memory paging to occur, which would severely impact system performance.

High queue thread wait times might not necessarily mean that not enough threads are set in the system. Instead, the wait times could be result of an application performance problem. For example, a transaction that users frequently run could have a poorly performing SQL. Every user executing the transaction ties up a thread while waiting for the database to execute the long-running SQL. If enough users are executing the transaction (and, in turn, the problem SQL), all of the threads could quickly become used. In this case, tune the SQL instead of adding more threads.