Remedy AR System server architecture and scalability
Remedy AR System uses servers to manage data. The following table summarizes the main servers.
Remedy AR System server
Processes data it receives from Remedy AR System clients, and passes the data to the database server to be stored
Stores definitions and data for the Remedy AR System server
Remedy AR System server architecture components
The following figure depicts the infrastructure of the Remedy AR System server.
Remedy AR System server infrastructure
The following are the main components of the Remedy AR System server architecture:
- Application programming interface (API) — A set of functions and data structures that provide application programmers with access to the full functionality of a product. Developers can create clients written in C, Java, or REST API. The Remedy AR System APIs are documented at Developing an API program and in the ardocVerNum.jar file located in the ARSystemServerInstallDir\Arserver\api\doc folder (Windows) or the ARSystemServerInstallDir/Arserver/api/doc folder (UNIX).
- Access control and data validation — A security feature in which Remedy AR System administrators limit user access to forms, to specific fields within a form, to specific functions within the system, and to data stored within the system.
- Alerts — A client program that functions as a "desktop pager." This component of the Remedy AR System server supports desktop pages such as flashing icons, audible beeps, sound files, and message windows. For example, it can display a message alerting help desk personnel that a problem was assigned to them.
For more information about alerts, see Using alerts.
- Filters — Actions performed on the Remedy AR System server, which is the portion of the software that controls the flow of requests to an underlying database. As a request is processed by the server, the filter actions take place. Filters enable you to implement constraints, make decisions, and take action when operations are performed on data stored in Remedy AR System.
- Escalations — Actions performed on the server at specified times or time intervals. They are automated, time-based processes that search for requests that match certain criteria and take action based on the results of the search.
- Remedy AR SystemFilter (AR Filter) API Plug-In — A filter that offers a programming interface directly invoked by filter workflow. This provides a flexible and efficient mechanism for communicating with various applications or web services. Use of plug-ins reduces system overhead. Remedy AR filter plug-ins also apply to escalations.
See Remedy AR filter API plug-ins introduction.
- Remedy AR SystemExternal Authentication (AREA) — A process that accesses network directory services and other authentication services to verify user login name and password information. When you use an AREA plug-in, you do not need to maintain duplicate user authentication data in the Remedy AR System directories because the Remedy AR System server can access user identification information and user passwords at many locations.
See Remedy AR System external authentication.
- View form — A form that enables Remedy AR System to point to and access data in a database table created outside Remedy AR System. The table can be in the same database or in any other database accessible from the current Remedy AR System database.
See View forms.
- Vendor form — A form that enables Remedy AR System to access arbitrary external data sources through the use of a Remedy AR System Database Connectivity (ARDBC) plug-in. This type of form provides for easy integration with external data without replicating the data.
See Vendor forms introduction.
- Database servers — The Remedy AR System uses standard relational databases for storing and retrieving data. Architecturally, the database server is a set of processes that are completely separate from the Remedy AR System server processes. Physically, the database server processes can be running on the same computer as the Remedy AR System server or on a different computer. The database server can be run on any platform that the particular database supports.
Remedy AR System high-level design
The following figure depicts the high-level design of the Remedy AR System server.
Remedy AR System high-level design
Remedy AR System is built to handle high loads and a large user base. It supports clustered configurations with multiple Remedy AR System server instances serving a single user base, as well as a distributed architecture where you can set up independent or groups of clustered servers in separate locations.
To address scalability concerns, Remedy AR System applies multi-threaded service queues and clustering of multiple server instances. Remedy AR System also has data replication capabilities (DSO) that allow for follow-the-sun kind of global operations.
Remedy AR System server highlights
The following highlights signify the Remedy AR System server:
- Java based implementation so as to leverage industry proven tools and methodologies
- Spring Framework as a light-weight framework for dependency injection (DI), aspect oriented programming (AOP), transaction management, and so on
- ONC/RPC based API binding; Remottea based implementation (for ONC/RPC Server implementation to expose Remedy AR System API)
- JPA to provide persistence support for Remedy AR System entities into Remedy AR System database schema
- JDBC to support the various databases supported by the Remedy AR system
- Ehcache for transaction safe, cluster enabled fine grained cache support
- Quartz for enterprise scale scheduler
- JMS for messaging
Remedy AR System server implementation
Remedy AR System server is implemented with the following layers:
- RPC Server
- Façade Layer
- Domain Model
- Service Layer
- Persistence Layer (Entity Model)
and a few critical components that fulfill cross-cutting concerns such as Object cache, Access Control, and a lot of other utility/miscellaneous/glue code.
Remedy AR System server queues
A queue is a meeting point where remote procedure calls (RPCs) wait to be handled by the worker threads that process the RPCs. When a queue is created, it automatically starts the minimum number of threads specified for its thread type. The default for this setting is 1. See Remedy AR System Threads.
The following table lists the types of Remedy AR System queues. Each queue has an RPC program number associated with it.
Queues and related RPC program numbers
RPC program number
Full Text Indexing
390621-390634, 390636-390669, 390680-390694
Administration, alert, escalation, fast, and list queues use a fixed RPC program number. Private queues, however, can be configured to use any RPC program number within the ranges of RPC program numbers reserved for private queues.
The following sections describe the different types of queues:
The administration (admin) queue is the only Remedy AR System queue that can perform every operation within the system. It performs all administrative restructuring operations, guaranteeing the serialization and integrity of all restructuring operations. This queue can have only one thread.
All servers include an admin queue, which is the default server setting. Because an admin queue has a single thread available to handle requests, a server that has only an admin queue (and no fast or list queues) functions as a single-threaded server. The admin queue handles all administrative functions. If no other queues are configured, the admin queue can perform the functions of all other queues and all requests are placed in the admin queue.
The alert queue handles all alerts sent to registered clients. The alert queue handles only internal requests, not requests from outside the Remedy AR System server. The threads in this queue do not open database connections, so they do not use many resources.
The minimum thread count for the alert queue is 1.
To configure an alert queue, see 2019-09-24_23-13-31_Defining queues and configuring threads.
All servers automatically create an escalation queue unless Disable Escalations is configured. The escalation queue handles only internal requests, not requests from outside the Remedy AR System server. It handles escalations specified by the administrator and performs all escalation processing. The escalation queue is multithreaded.
A fast queue handles the operations that generally run to complete quickly without blocking access to the database. The fast queue handles all server operations, except for:
- Administrative operations that restructure the database. These operations use the administration queue.
ARGetEntryStatistics, and other API calls (which use the list queue).
For more information about API calls, see the Developing an API program.
One or more threads can serve a fast queue if the fast queue is configured. To configure a fast queue, see 2019-09-24_23-13-31_Defining queues and configuring threads.
A list queue handles Remedy AR System operations that might require significant time, block access to the database, or both. Examples of these operations include
One or more threads can serve a list queue if the list queue is configured. To configure a list queue, see 2019-09-24_23-13-31_Defining queues and configuring threads.
Administrators can also create private queues for specific applications or users that require dedicated access. For example, you might create a private queue for the Remedy Approval Server or other plug-in application, or for a user who is performing critical operations that you do not want blocked by other users. Private queues guarantee a certain bandwidth dedicated to clients using these queues.
Private queues support all operations except restructuring operations. Restructuring operations are supported only by the administration queue (see Administration queue). To configure a private queue, see 2019-09-24_23-13-31_Defining queues and configuring threads.
Each private queue can be supported by one or more threads. To connect a user to a private queue, see Configuring clients for Remedy AR System servers.
For more information about private queues, see the blog Utilizing Private Queues shared on BMC Communities.
Remedy AR System server threads
The term thread is short for thread of execution. Threads enable the server to process concurrent client requests. Each thread within the multithreaded server can carry out a client request before returning to the queue to process the next one. Start only as many threads as your database and system resources can reasonably support. The total number of threads cannot exceed the number of database connections available to the Remedy AR System server.
All threads within a process share network and system resources; therefore, consider carefully the available resources of your system and network when establishing the minimum and maximum thread settings for your server queues.
Remedy AR System uses the following types of threads:
A dispatcher thread routes requests to the appropriate queues. This thread receives connection requests from clients. The dispatcher thread then places the requests into the appropriate queue where each request can be handled by one of multiple worker threads.
Every call that the dispatcher thread receives is assigned an RPC ID that can be used to identify the call from the time the call is placed into a queue until a response is sent to the client.
In general, the dispatcher thread uses the following logic to dispatch calls:
- If no other queues are defined, the dispatcher thread routes all requests to the admin queue. If, however, fast and list queues are created in addition to the admin queue, the dispatcher routes client requests according to the type of operation being performed. If private queues are created, the dispatcher directs the call to the appropriate private queue based on the RPC program number of the request.
A request is routed to the appropriate queue based on its RPC program number. For example, a call that has RPC program number
390600is routed to the admin queue.
- If a call with RPC program number
390635(list) is received and no fast or list queue exists, the dispatcher thread routes the call to the admin queue. If only a list queue exists, the dispatcher thread places the call in that queue. If only a fast queue exists, the dispatcher thread directs the call to that queue. If both fast and list queues exist, the dispatcher routes the call to the appropriate queue based on the call number.
- If a call is received with RPC program number
390601(previously supported by the Notification server, which is now merged with the Remedy AR System server), the dispatcher routes the call to the fast queue.
- If a call is received with an RPC program number other than those specified for admin, fast, list, and Flashboards queues, the dispatcher identifies the call as destined for a private queue. If a private queue supporting the RPC program number exists, the dispatcher thread routes the call to that queue. If no private queue exists but a fast or list queue exists, the call is routed to the appropriate queue based on its RPC program number. If no fast or list queue exists, the call is routed to the admin queue.
- The escalation and alert queues do not receive calls from the dispatcher.
Worker threads respond to the RPCs dispatched to individual queues. Each queue creates one or more worker threads. The worker threads within a queue are identical and can handle any request. Only the worker thread started by the admin queue, however, can handle calls that modify definitions or server configuration settings.
On startup, each thread creates a connection to the database that it uses throughout its existence. If the thread is unable to establish a connection, it terminates itself, notifying the queue that no more threads are to be started. The database connection is dedicated to the thread, even when that particular thread is not busy.
Any available worker thread can remove the request from the queue, read the request, process it, and return results to the client before returning to the queue to respond to another request. When a request is placed in a queue and no existing threads are available to handle it, a new thread is started until the queue reaches the maximum number of threads allowed for its thread type.
The thread manager is responsible for ensuring that a thread is restarted if it dies.
Determining thread count and type
A major benefit of a multithreaded server is not having fast operations held up behind a slow list operation. Deciding how many fast and list threads you need depends on your particular setup and situation. For example, not specifying enough list threads might mean you have idle fast threads but an overloaded list queue.
Another consideration is that list threads require more memory than fast threads. For example, a complicated query might require a great deal of memory at a given moment. A few of these large queries can temporarily exhaust your system resources.
To determine how many threads of each type you need, start by examining the types of API calls by using the Remedy AR System Log Analyzer tool. For more information, see Viewing Remedy AR System Log Analyzer output. If your system processes many fast operations, you might decide to increase the number of fast threads. A different rule of thumb is to specify a larger maximum of list threads than fast threads, simply because fast operations are generally performed more quickly than list operations.
Do not specify an artificially high number of maximum threads because the threads would consume resources that other threads need. To set the number of minimum and maximum threads, see Setting ports and RPC numbers.
Remedy AR System server processes
The Remedy AR System server is a set of processes that run on an application's server host. The server is available for each of these operating systems:
- Linux (Red Hat and Novell SuSE)
- Microsoft Windows Server
- Oracle Microsystems Solaris
For the most accurate information about supported platforms and software, see in the Remedy ITSM Deployment online documentation.
The server is implemented as several service processes that are normally started automatically when the server host is powered up.
The Remedy AR System server has no direct user interface. Clients, such as the mid tier and other applications, communicate with Remedy AR System by means of a well-defined application programming interface (API). An API is one possible way to integrate with Remedy AR System. The APIs use remote procedure calls (RPCs) to communicate with the server. Both a C interface and a Java interface are available.
The Remedy AR System server processes all data entered through a client. As the workflow engine between client and database, the server writes data to the database when a request is created and retrieves data from the database when a client requests it. The server verifies that a user has permission to perform each transaction, thereby enforcing any access control defined in an application. The server also continuously evaluates the data in the database and each transaction to determine whether the server should perform workflow. The server might also perform workflow on a timed basis.
When a client submits a request to the server, the request enters through the API, goes through access control and data validation, filter processing, and then transactions are committed to the data repository as appropriate.
Following are the key processes in the Remedy AR System server:
- arserver process — the primary Remedy AR System server process. It handles requests from the clients and interacts with the database.
- Plug-in server process— A companion process to the Remedy AR System server. It loads configured plug-in modules to interface with external data while processing vendor forms, validates users with external authentication sources, and extends filter workflow. When the Remedy AR System server needs to access a plug-in, it interfaces with the plug-in server to perform the operation.
- Email engine process — the process that links arserver process with email systems. For example, users can submit service requests to an AR System server by sending an email message to an email account. In addition, workflow can cause email messages to be sent to particular email addresses as a notification option.
For more information about configuring Remedy AR System servers, see:
For more information about tuning AR System servers, see .