Configuring private queue threads
AR System Server divides the work into queues and threads. Queues are a pool of common threads. A thread is a single compute unit used to complete a task. The AR Server routes the work to a queue and thread. The thread runs the process and the result is sent back to the client which requested the work.
This topic contains information on:
Understanding Queues
Queues are divided based on their operations. Some common queues are Admin, Full Text Indexing, Escalations, Fast, and List. The Administrator cannot customize these queues. You can customize some queues for processes such as Normalization and Reconciliation. Any customizable queue is known as a Private Queue because it is used only by one specific process.
Important
Queues are sometimes referred to as a RPC Socket, RPC Queue, or just Socket, and should not be confused with Ports or TCP Sockets.
Most user traffic is processed in Fast and List queues. If a queue of 0 or NULL is used, then the user traffic will be routed to a queue based on the call being executed.
The following list shows the benefits of queues based on how you configure them:
- A queue could be dedicated to a critical resource, like Email Engine on the server processing emails.
- A queue can be used to limit interference to user facing queues.
- Adding queues allows your system to process more work in parallel. Decreasing queues allows your system to consume less resources.
Understanding Threads
Threads are considered workers of queues. Each thread can execute one request at a time. AR Server allows administrators to set the number of threads available to each queue by providing a minimum and a maximum value.
Important
A thread cannot belong to more than one queue, and it can not switch queues once it is assigned.
The minimum value of threads are created on system startup. AR Server does not allow queues less than the minimum value set by the system.
Depending on the work, more threads are created and work is assigned to the threads up to the maximum value. Once the maximum value is reached, additional threads are not created, and the work is assigned to the first thread in another queue.
The process of assigning work to individual queues and threads is called Scheduling. Scheduling is done via the round-robin method to available threads.
Example
Jonnie is a Remedy admin at Calbro Services, where he has an AR System Server having a queue with configuration of two minimum threads and 4 maximum threads. This queue has two clients configured to connect to it.
Scenario 1:
If Client 1 sends 4 GLEWF (Get List Entry With Fields) request to this queue, at the same time Client 2 sends 4 of the same request, then all 4 threads will be spun up. The queue will execute Client 1's request first (one request per thread) and then immediately schedule Client 2's request. This process is called Request Queuing, as Client 2's request would be waiting in line for the first request to be completed.
Scenario 2:
Client 1 sends 1 GLEWF request in, and it takes 30 seconds to complete. But, after only one second Client 2 sends two new requests to be completed within 1 second. This action causes Thread 3 to spin up. But Thread 4 is not available yet. If Client 1 sends another request before the first one is complete, then Thread 4 is spun up. If in the last 5 seconds of the first request, 3 more requests came in, they are scheduled to Thread 2, 3, and 4 - as Thread 1 (even though it is next in the round robin scheme) is not available.
There is no scenario where the number of threads would reduce back to the minimum once the maximum is reached, unless you changed the configuration to be a lower number.
Important
Private Queues can be added and removed using Centralized Configuration without a server restart. Threads can be added or removed for all queues using Centralized Configuration without a server restart.
Configuring AR System RPC Queues and Threads is not a requirement for all systems and must not impact daily operations. This is a performance tuning exercise that should provide no functional impact on any system. Depending on the goals of the system, queues and threads allows administrators to either limit (throttle) certain operations or allow operations to be executed in parallel. More queues with higher thread counts cause more parallel operations to execute at the same time. Fewer queues with lower thread counts will cause fewer queries to execute at the same time.
You must make changes for both the AR Server and the Client for any changes with queues and threads to be effective. Making a change in either the AR Server or the Client (and not both) makes any performance issues worse. Use the tables below to better understand both default Queues (ones that you cant change and should always have) and suggested Queues for processes that might be different based on your environment.
Example
The following examples show when the queues and threads are not effective:
Example 1: It would provide no benefit to have a Queue configured in AR Server, but no clients using it.
Example 2: It would provide no benefit to have a a single threaded plugin be assigned to its own Private Queue with 10 threads configured.
Example 3: It would cause a performance issue to have a 4 threaded client configured to a AR Server Queue where only 1 thread is assigned.
Example 4: It would cause a performance issue to have a client configured to a AR Server Queue, where no AR Server Queue is configured.
Configuring New Private Queues for AR Server
In order to configure a new Private Queue for the AR Server, the AR System Administrator can either add, remove, or modify the Private-RPC-Socket parameter in the Centralized Configuration form.
The syntax for this configuration is:
Private-RPC-Socket <SOCKET> <MIN THREADS> <MAX THREADS>Example:
Private-RPC-Socket 390680 4 4
Considerations for Queues and Threads
You must always consider what a queue will be used for. Sometimes, you do not have control on some threads like Fast and List threads about the work they will process, as users and their interactions largely determines what work is sent to the threads for processing. Some things like Reconciliation queue are largely predictable since most reconciliation jobs are started at specific times.
Best practice
AR System Default Queues
The following tables list tuning methods for several workload components.
| Queue Number | Name | Description |
|---|---|---|
| 390600 | Admin | Single threaded Queue that is not defined in ar.conf file. This queue can not be modified. |
| 390601 | Alert | Queue used to process Alerts. This queue defined in the ar.conf file. You can only increase or decrease the threads for this pool. |
| 390602 | FTS Indexing | Queue used to process FTS Indexing requests. This queue is defined in the ar.conf file. You can only increase or decrease the threads for this pool. |
| 390603 | Escalation | Queue used to process escalations. This queue is defined in the ar.conf file. You can only increase or decrease the threads for this pool. Important We recommend you to set the number of minimum and maximum threads to the same value. |
| 390620 | Fast | The FAST queue is responsible for processing requests from clients that do not define a RPC Queue. This queue is defined in the ar.conf file. You can only increase or decrease the threads for this pool. |
| 390621-390634 | Private | Range of queues numbers that you can assign to any process. |
| 390635 | List | The LIST queue is responsible for processing request from clients that do not define a RPC Queue. This queue is defined in the ar.conf file. You can only increase or decrease the threads for this pool. |
| 390636-390669 | Private | Range of queues numbers that you can assign to any process. |
| 390680-390694 | Private | Range of queues numbers that you can assign to any process. |
Suggested Private Queues
The following table is a compiled list of common settings that are recommended by BMC, although you can assign these numbers to any queue desired.
| Queue Number | Name | Description |
|---|---|---|
| 390619 | Flashboards | The Flashboard Server is a single threaded process. You can create a queue to process these queries in the AR Server and assign that queue to the Flashboards server by modifying the ARSystemInstallDirectory/flashboards/server.conf file. For more information, see
Configuring BMC Remedy Flashboards
|
| 390621 | Email Engine | The Email Engine is a multi-threaded process. There is one thread per mailbox. You can create a queue to process these queries in the AR Server and assign that queue to the Email Engine by adding the following setting in the Centralized Configuration form:
For more information, see
Configuring BMC Remedy Email Engine
|
| 390622 | App Query Plugin | The AppQuery Plugin is a multi threaded plugin that runs in the AR Shared Java Plugin Server. This plugin has the ability to forward its queries to a private queue. This plugin can be configured by altering the following parameter in the Centralized Configuration form for the appropriate AR System Server:
For more information, see the topic
AppQuery plug-in configuration
|
| 390623 | DSO Config Plugin | The DSO Config Plugin is a single threaded plugin that can run multiple times in the AR Shared Java Plugin Server. This plugin has the ability to forward its queries to a private queue. This plugin can be configured by altering the following parameter in the Centralized Configuration form for the appropriate AR System Server:
For more information, see the topic Error rendering macro 'link-window' Failed to transform the HTML macro template for display. Nested message: The XML content could not be parsed. There is a problem at line 4, column 113. Parser message: Duplicate attribute 'configuration'. at [row,col {unknown-source}]: [4,113] |
| 390626 | Loopback | The Loopback RPC Socket is an internal queue used by plugins and the AR Server when a plugin request routes back to the AR Server for additional queries. This plugin can be configured by altering the following parameter in the Centralized Configuration form for the appropriate AR System Server:
For more information, see the topic
Private queues for loopback calls
|
| 390636 | Business Rules Engine | The Business Rules Engine is a single threaded plugin that is included in BMC Service Level Management. This plugin can be configured by altering the following parameter in the Centralized Configuration form for the appropriate AR System Server:
|
| 390637 | Assignment Engine | The Assignment Engine process is an internal multi-threaded AR System process. This plugin can be configured by altering the following parameter in the Centralized Configuration form for the appropriate AR System Server:
|
| 390638 | DSO | The DSO Server is a multi threaded plugin that facilitates the synchronizing of tickets between two different AR Server Environments. This plugin can be configured by altering the following parameter in the Centralized Configuration form for the appropriate AR System Server:
|
| 390680 | Approvals | The Approval Server is a multi threaded plugin that runs in the AR Shared Java Plugin Server. This plugin has the ability to forward it's queries to a private queue. This plugin can be configured by adding the following parameter in the Centralized Configuration form for the appropriate AR System Server: |
| 390681 | AR Normalization | The AR Normalization queue is not a dedicated queue for a specific client, but is rather a shared queue for several CMDB related plugins. This includes the Normalization Engine and the Deprecation plugin. This plugin can be configured by altering the following parameter in the Centralized Configuration form for the appropriate AR System Server: |
| 390693 | CAI Plugin | The Command Automation Interface (CAI) plugin is a multi-threaded plugin that runs in the AR Shared Java Plugin Server. This plugin has the ability to forward its queries to a private queue. You can configure this by modifying the Private Queue # on the CAI:PluginRegistry form. For more information, see the topic
Configuring command automation interface
|
| 390698 | Reconciliation | The Reconciliation Engine is a multi-threaded CMDB Plugin that can be extremely CPU and Database intensive. This plugin has the ability to forward its queries to a private queue. This plugin can be configured by adding or updating the following parameter in the Centralized Configuration form for the appropriate AR System Server:
For more information, see the topic
Reconciliation Engine configuration settings
|
| 390699 | CMDB Normalization | The CMDB Normalization queue is not a dedicated queue for a specific client, but is a shared queue for several CMDB related plugins. This includes the Normalization Engine and the Deprecation plugin. This plugin can be configured by altering the following parameter in the Centralized Configuration form for the appropriate AR System Server:
|
Important
- When defining an RPC Queue, you must configure the client to send its request on that queue and the AR Server to create the same RPC queue for processing the work.
- If no queue is defined by the client, the AR Server routes the request to either Admin, List, or Fast Queues depending on the type of request submitted.
- If a queue is defined by the client but the AR Server does not have that queue or it is a wrong number, then the request will be sent to either Admin, List, or Fast Queues. If the Queue is not in a appropriate range, then the client will get a error.
BMC CMDB thread sizing
BMC CMDB normalization and reconciliation processing are CPU intensive. By default, CMDB processes are executed using fast and list threads. Private queues allow you to restrict how much CPU is used by CMDB processes.
Normal day-to-day CMDB processing should not consume more than 20% of AR System server CPU capacity. The goal is to obtain optimal throughput for CMDB processing while not sacrificing response time for the users who are doing online transactions.
Configure a private queue by setting minimum and maximum thread settings to the number of CPU cores as the starting values. Adjust the thread settings based on the BMC CMDB workload, online workload, and thorough performance testing.
Comments
Log in or register to comment.