Configuring private queue threads

AR System Server divides the work into queues and threads. Queues are a pool of common threads. A thread is a single compute unit used to complete a task. The AR Server routes the work to a queue and thread. The thread runs the process and the result is sent back to the client which requested the work.

This topic contains information on:

Understanding Queues

Queues are divided based on their operations. Some common queues are Admin, Full Text Indexing, Escalations, Fast, and List. The Administrator cannot customize these queues. You can customize some queues for processes such as Normalization and Reconciliation. Any customizable queue is known as a Private Queue because it is used only by one specific process.

Important

Queues are sometimes referred to as a RPC Socket, RPC Queue, or just Socket, and should not be confused with Ports or TCP Sockets.

Most user traffic is processed in Fast and List queues. If a queue of 0 or NULL is used, then the user traffic will be routed to a queue based on the call being executed.

The following list shows the benefits of queues based on how you configure them:

  • A queue could be dedicated to a critical resource, like Email Engine on the server processing emails.
  • A queue can be used to limit interference to user facing queues.
  • Adding queues allows your system to process more work in parallel. Decreasing queues allows your system to consume less resources.

Understanding Threads

Threads are considered workers of queues. Each thread can execute one request at a time. AR Server allows administrators to set the number of threads available to each queue by providing a minimum and a maximum value.

Important

A thread cannot belong to more than one queue, and it can not switch queues once it is assigned.

The minimum value of threads are created on system startup. AR Server does not allow queues less than the minimum value set by the system.

Depending on the work, more threads are created and work is assigned to the threads up to the maximum value. Once the maximum value is reached, additional threads are not created, and the work is assigned to the first thread in another queue.

The process of assigning work to individual queues and threads is called Scheduling. Scheduling is done via the round-robin method to available threads.

Example

Jonnie is a Remedy admin at Calbro Services, where he has an AR System Server having a queue with configuration of two minimum threads and 4 maximum threads. This queue has two clients configured to connect to it.

Scenario 1:

If Client 1 sends 4 GLEWF (Get List Entry With Fields) request to this queue, at the same time Client 2 sends 4 of the same request, then all 4 threads will be spun up. The queue will execute Client 1's request first (one request per thread) and then immediately schedule Client 2's request. This process is called Request Queuing, as Client 2's request would be waiting in line for the first request to be completed.

Scenario 2:

Client 1 sends 1 GLEWF request in, and it takes 30 seconds to complete. But, after only one second Client 2 sends two new requests to be completed within 1 second. This action causes Thread 3 to spin up. But Thread 4 is not available yet. If Client 1 sends another request before the first one is complete, then Thread 4 is spun up. If in the last 5 seconds of the first request, 3 more requests came in, they are scheduled to Thread 2, 3, and 4 - as Thread 1 (even though it is next in the round robin scheme) is not available.

There is no scenario where the number of threads would reduce back to the minimum once the maximum is reached, unless you changed the configuration to be a lower number.

Important

Private Queues can be added and removed using Centralized Configuration without a server restart. Threads can be added or removed for all queues using Centralized Configuration without a server restart.

Configuring AR System RPC Queues and Threads is not a requirement for all systems and must not impact daily operations. This is a performance tuning exercise that should provide no functional impact on any system. Depending on the goals of the system, queues and threads allows administrators to either limit (throttle) certain operations or allow operations to be executed in parallel. More queues with higher thread counts cause more parallel operations to execute at the same time. Fewer queues with lower thread counts will cause fewer queries to execute at the same time.

You must make changes for both the AR Server and the Client for any changes with queues and threads to be effective. Making a change in either the AR Server or the Client (and not both) makes any performance issues worse.  Use the tables below to better understand both default Queues (ones that you cant change and should always have) and suggested Queues for processes that might be different based on your environment.


Example

The following examples show when the queues and threads are not effective:

Example 1: It would provide no benefit to have a Queue configured in AR Server, but no clients using it.

Example 2: It would provide no benefit to have a a single threaded plugin be assigned to its own Private Queue with 10 threads configured.

Example 3: It would cause a performance issue to have a 4 threaded client configured to a AR Server Queue where only 1 thread is assigned.

Example 4: It would cause a performance issue to have a client configured to a AR Server Queue, where no AR Server Queue is configured.

Configuring New Private Queues for AR Server

In order to configure a new Private Queue for the AR Server, the AR System Administrator can either add, remove, or modify the Private-RPC-Socket parameter in the Centralized Configuration form.

The syntax for this configuration is:

Private-RPC-Socket <SOCKET> <MIN THREADS> <MAX THREADS>

Example:

Private-RPC-Socket 390680 4 4

image2021-1-18_17-17-23.png

Considerations for Queues and Threads

You must always consider what a queue will be used for. Sometimes, you do not have control on some threads like Fast and List threads about the work they will process, as users and their interactions largely determines what work is sent to the threads for processing. Some things like Reconciliation queue are largely predictable since most reconciliation jobs are started at specific times.

Best practice
You must set client/plugin threads equal to the threads assigned to the Private queue.

AR System Default Queues

The following tables list tuning methods for several workload components.

Queue Number

Name

Description

390600

Admin

Single threaded Queue that is not defined in ar.conf file. This queue can not be modified.

390601

Alert

Queue used to process Alerts.

This queue defined in the ar.conf file. You can only increase or decrease the threads for this pool.

390602

FTS Indexing

Queue used to process FTS Indexing requests.

This queue is defined in the ar.conf file. You can only increase or decrease the threads for this pool.

390603

Escalation

Queue used to process escalations.

This queue is defined in the ar.conf file. You can only increase or decrease the threads for this pool.

Important

We recommend you to set the number of minimum and maximum threads to the same value. 

390620

Fast

The FAST queue is responsible for processing requests from clients that do not define a RPC Queue.

This queue is defined in the ar.conf file. You can only increase or decrease the threads for this pool. 

390621-390634

Private

Range of queues numbers that you can assign to any process.

390635

List

The LIST queue is responsible for processing request from clients that do not define a RPC Queue.

This queue is defined in the ar.conf file. You can only increase or decrease the threads for this pool. 

390636-390669

Private

Range of queues numbers that you can assign to any process.

390680-390694

Private

Range of queues numbers that you can assign to any process.


Suggested Private Queues

The following table is a compiled list of common settings that are recommended by BMC, although you can assign these numbers to any queue desired.

Queue Number

Name

Description

390619

Flashboards

The Flashboard Server is a single threaded process. You can create a queue to process these queries in the AR Server and assign that queue to the Flashboards server by modifying the ARSystemInstallDirectory/flashboards/server.conf file.

For more information, see Configuring BMC Remedy Flashboards.

390621

Email Engine

The Email Engine is a multi-threaded process. There is one thread per mailbox. You can create a queue to process these queries in the AR Server and assign that queue to the Email Engine by adding the following setting in the Centralized Configuration form:

com.bmc.arsys.emaildaemon.<serverName>.RPC=390681

For more information, see Configuring BMC Remedy Email Engine.

390622

App Query Plugin

The AppQuery Plugin is a multi threaded plugin that runs in the AR Shared Java Plugin Server.  This plugin has the ability to forward its queries to a private queue. This plugin can be configured by altering the following parameter in the Centralized Configuration form for the appropriate AR System Server:

REMEDY.ARDBC.APPQUERY.userDefined.Private-RPC-Socket

For more information, see the topic AppQuery plug-in configuration.

390623

DSO Config Plugin

The DSO Config Plugin is a single threaded plugin that can run multiple times in the AR Shared Java Plugin Server. This plugin has the ability to forward its queries to a private queue. This plugin can be configured by altering the following parameter in the Centralized Configuration form for the appropriate AR System Server:

DSO.FILTERCONFIGURATION.userDefined.rpcqueue

For more information, see the topic DSO Filter Configuration plug-in configuration.

390626

Loopback

The Loopback RPC Socket is an internal queue used by plugins and the AR Server when a plugin request routes back to the AR Server for additional queries. This plugin can be configured by altering the following parameter in the Centralized Configuration form for the appropriate AR System Server:

Plugin-Loopback-RPC-Socket

For more information, see the topic Private queues for loopback calls.

390636

Business Rules Engine

The Business Rules Engine is a single threaded plugin that is included in BMC Service Level Management. This plugin can be configured by altering the following parameter in the Centralized Configuration form for the appropriate AR System Server:

BR-RPC-Socket

For more information, see the topic Administering in the BMC Service Level Management documentation.

390637

Assignment Engine

The Assignment Engine process is an internal multi-threaded AR System process. This plugin can be configured by altering the following parameter in the Centralized Configuration form for the appropriate AR System Server:

AE-RPC-Socket

For more information, see the topic Configuring the Assignment Engine server settings.

390638

DSO

The DSO Server is a multi threaded plugin that facilitates the synchronizing of tickets between two different AR Server Environments. This plugin can be configured by altering the following parameter in the Centralized Configuration form for the appropriate AR System Server:

DSO-Local-RPC-Socket

For more information, see the topic Assigning an RPC program number to DSO.

390680

Approvals

The Approval Server is a multi threaded plugin that runs in the AR Shared Java Plugin Server.  This plugin has the ability to forward it's queries to a private queue. This plugin can be configured by adding the following parameter in the Centralized Configuration form for the appropriate AR System Server:

Approval-RPC-Socket

For more information, see the topic Configuring the BMC Remedy Approval Server.

390681

AR Normalization

The AR Normalization queue is not a dedicated queue for a specific client, but is rather a shared queue for several CMDB related plugins. This includes the Normalization Engine and the Deprecation plugin.

This plugin can be configured by altering the following parameter in the Centralized Configuration form for the appropriate AR System Server:

BMC.ARDBC.DEPRECATION.PLUGIN.userDefined.ar_rpc_queue

For more information, see the topics Deprecation configuration settings and Configuring the Normalization Engine.

390693

CAI Plugin

The Command Automation Interface (CAI) plugin is a multi-threaded plugin that runs in the AR Shared Java Plugin Server. This plugin has the ability to forward its queries to a private queue. You can configure this by modifying the Private Queue # on the CAI:PluginRegistry form.

For more information, see the topic Configuring command automation interface.

390698

Reconciliation

The Reconciliation Engine is a multi-threaded CMDB Plugin that can be extremely CPU and Database intensive. This plugin has the ability to forward its queries to a private queue.

This plugin can be configured by adding or updating the following parameter in the Centralized Configuration form for the appropriate AR System Server:

RE-RPC-Socket

For more information, see the topic Reconciliation Engine configuration settings.

390699

CMDB Normalization

The CMDB Normalization queue is not a dedicated queue for a specific client, but is a shared queue for several CMDB related plugins. This includes the Normalization Engine and the Deprecation plugin.

This plugin can be configured by altering the following parameter in the Centralized Configuration form for the appropriate AR System Server:

BMC.ARDBC.DEPRECATION.PLUGIN.userDefined.cmdb_rpc_queue

For more information, see the topics Deprecation configuration settings and Configuring the Normalization Engine.

Important

  • When defining an RPC Queue, you must configure the client to send its request on that queue and the AR Server to create the same RPC queue for processing the work.
  • If no queue is defined by the client, the AR Server routes the request to either Admin, List, or Fast Queues depending on the type of request submitted.
  • If a queue is defined by the client but the AR Server does not have that queue or it is a wrong number, then the request will be sent to either Admin, List, or Fast Queues. If the Queue is not in a appropriate range, then the client will get a error.


BMC CMDB thread sizing

BMC CMDB normalization and reconciliation processing are CPU intensive. By default, CMDB processes are executed using fast and list threads. Private queues allow you to restrict how much CPU is used by CMDB processes.

Normal day-to-day CMDB processing should not consume more than 20% of AR System server CPU capacity. The goal is to obtain optimal throughput for CMDB processing while not sacrificing response time for the users who are doing online transactions.

Configure a private queue by setting minimum and maximum thread settings to the number of CPU cores as the starting values. Adjust the thread settings based on the BMC CMDB workload, online workload, and thorough performance testing.     

Related Topics

Private queues for loopback calls

 

Tip: For faster searching, add an asterisk to the end of your partial query. Example: cert*