Remedy AR System server group deployment
Server groups are designed to work with Remedy Action Request System (AR System) in any type of supported environment that has more than one server, including large distributed setups that make use of hardware load balancers. The AR System server groups feature provides failover management for crucial operations, server scalability, application-level load balancing, and the sharing of floating licenses among the servers. A server group typically consists of two or more AR System servers that are managed separately. The servers share the same AR System database, but perform workflow and database updates independently from each other.
The servers within a server group need not have the same operating system, but you should ensure that any workflow that interacts with the operating system will run on all operating systems within the server group.
AR System servers in a server group
A server group acts as a single AR System server to support applications running on multiple AR System servers. The individual servers in the server group are configured to spread the load of shared services, and to provide backup for each other to ensure that those services are always available. BMC applications that are based on AR System, such as the BMC Atrium CMDB and its related applications, as well as the Remedy ITSM suite of applications can be installed and configured to operate within a server group.
To ensure high availability of AR System operations, a server group can be configured to provide failover protection by assigning rankings for specific AR System operations to the servers in the group.
It is required to always use the exact same version and patch level for all BMC software applications on each server included within a server group. And, to always upgrade each application on each server within the server group at the same time.
Benefits of using a server group
The following are the benefits of deploying Remedy AR System in a server group:
- Distribution of heavy user traffic across multiple AR System servers using a third-party load balancer.
- Automatic server failover. If a server becomes unavailable, the system seamlessly keeps running.
- Ease of administration; it has only one database to manage and back up.
- Sharing of all BMC software licenses among the AR System servers, except AR Server licenses.
- Capability of administering server settings across the entire group from a single console (Remedy Administration Console). When the workflow and applications are handled on the server that is designated as the administrative server, the changes are automatically propagated to other servers in the group.
- Specific servers in the group can also be configured to handle reporting, reconciliation, and other tasks that can impact performance, freeing up the remaining servers in the group to handle user traffic.
- Inexpensive servers can be added to a server group to increase the growth in users and workload without having to replace or upgrade a single server. The environment is easier and less expensive to scale.
Server groups also work in conjunction with hardware load balancers that direct user traffic to some or all servers in the group. For best performance and reliability, BMC recommends that you use a load balancer when implementing server groups. For specific details on using a load balancer, see the following topics on the Remedy AR System online documentation:
See the following examples of a simple and a complex AR system server group configurations.
Simple AR System server group deployment
In the simplest form, a server group can be setup with two AR System servers and a database server. Each of the AR System servers have all Remedy products installed.
In this example, the first server installed should be configured to be the primary administration server. Then, using the AR System Server Group Operation Ranking form, the applications should be distributed evenly across both servers (use the AR System Service Failover Ranking form for Email Engine and Normalization Engine). The distribution is done so that each server handles about half of the load, and each server has the other server configured for failover on each of its applications.
The exception to this is the Email Engine and the Flashboards server. Email Engine is ranked as per configured mailboxes. To achieve failover and load balancing of your Email workload, you can configure the Email Engine to have multiple mailboxes that are ranked differently across the servers. In a simple configuration, the Email Engine and the Flashboards server are installed on every system but can be configured for only one server. Configuring failover for those operations can be complex and may not be necessary in a simple configuration.
The full text search feature should be configured on each server. Each server has the ability to read from FTS, but only one of the servers that has a higher priority on the AR System Server Group Operation Ranking form can be set to write. It is also recommended that the FTS collection directory (location where the search index files are stored) and FTS configuration directory (location where the search configuration files are stored) be located on the same computer.
Complex AR System server group deployment
A more complex server group implementation contains three or more AR System servers. In this example we are using four AR System servers. The applications or components should be installed on more than one server in the group so that failover can be provided.
Again, the first server installed should be configured to be the primary administration server. Then, using the AR System Server Group Operation Ranking form, the applications are distributed evenly across multiple servers, so that each server handles a portion of the load, and each server has at least one other server configured for failover on each of its applications and components. You can also set up two servers to handle the user facing operations, while the other two servers handle the backend operations. Separating the user-facing and backend operations to different servers help provide optimal performance to the users.
In this case, even the Email Engine server and the Flashboards have a failover server assigned. Configuring failover for those operations is somewhat complex because it means that each server has to be specifically configured to handle those items, but the secondary server for each is suspended until the failover is activated.
The applications and components listed for the servers above are just the primary roles for each server. It is recommended that all applications and components be installed on each server. This is important because users could be accessing any components from any server in the group, and there are dependencies such as plug-ins and other binary files that could be called when a user opens certain forms, creates a new record, or modifies a record.
FTS is installed on each server of the server group. You must rank and configure FTS in the designated servers as per your deployment strategy. If more than one server are ranked for FTS, then all the servers send their search requests to the server that is ranked 1 for FTS. If the rank 1 server becomes unavailable, then the FTS operations fail over to the next ranked FTS server. However, you can configure the servers to send their search requests to specific indexer to balance the search load. We recommend that the FTS collection directory (location where the search index files are stored) is present on the same server as the AR Systems server.
Planning where software is installed in a server group
When installing server groups, make sure you know exactly what products you plan to install, and determine which servers will be running each product. BMC recommends that you install all the Remedy products on all servers, and then to use the AR System Server Group Operation Ranking form to distribute the load. The Email Engine and Normalization Engine use the AR System Service Failover Ranking form. However, there are may different ways to do this and each decision is based on the specific implementation.
For using the IP-Name parameter to perform administrative operations, create a list in a text file for each server and its IP address, as well as all accepted fully qualified names. This saves you time when configuring the other servers. Each server that is ranked for Administration operation must have the same set of entries containing all resolvable names for each server in teh server group. Include the IP address, short name, and fully qualified name for each server in the server group and the load balancer, if one is used. In the installation instructions, a two-system server group is used and the systems have the following information. (Ensure to obtain similar information for your implementation before you start the installation process.)
The following example uses a format that you can use the Centralized Configuration form for all the servers in your server group. It includes the entire set of IP-Name entries for this example.
You use IP-Name to store different IP addresses for the current server. This validates if a given action needs to be performed on the current server or on the remote server. If the current server name matches any of the IP-Name, "@" is stored in the database metadata and also in the exported DEF files so that objects become portable from one server to another. Else it continues to be the given server name (remote server name).
The following actions uses IP-Name:
- Direct SQL call from active link
- Run Process from active link
- Expand menu
- Call active link guide
- Server specified in table field
- Set Fields
- Push Fields
AR System Server 1: IP-Name: svr_grp_tst0 IP-Name: svr_grp_tst0.svgroup.com IP-Name: svr_grp_tst0.test.svgroup.com IP-Name: 126.96.36.199 AR System Server 2: IP-Name: svr_grp_tst3 IP-Name: svr_grp_tst3.svgroup.com IP-Name: svr_grp_tst3.test.svgroup.com IP-Name: 188.8.131.52
Additional information required
You must provide the following additional information:
- Server-Name: The short name that resolves to the load balancer.
- Domain-Name: The domain name of the load balancer. When concatenated to the Server-Name, it provides a Fully Qualified Server Name that uniquely resolves to the load balancer for the AR System servers.
Load Balancer Name: RemedyServerGroup.test.svgroup.com Server-Name: RemedyServerGroup Domain-Name: test.svgroup.com
How failover occurs in high-availability configuration
FTS requires that at least one of the servers in a server group act as a primary FTS indexing server. To provide failover, you can have two or more primary FTS indexing servers in the server group where each server acts as an independent FTS server and indexing FTS in their own Collection directory. Each FTS indexing server (known as primary FTS indexing server) manages its own separate local replica of the full text index data. When an indexing action takes place, each FTS indexing server indexes it independently.
For example, if you create a form and then index a field on that form, each FTS indexing server indexes that field. Because you can have multiple FTS servers, each with its own current copy of the FTS data, you can fail over to a second server.
An AR System server is designated as a primary FTS indexing server by having a ranking record for FTS in the AR System Server Group Operation Ranking form. Each FTS server defined in the ranking form acts as an indexing server and provides FTS search services to other servers in the server group. Each defined FTS server will always index, regardless of its ranking order. However, the server that is ranked 2 contains redundant FTS data and must be used for failover. This server is not intended to be a user-facing server.
Servers in the server group that are not ranked for FTS are search-only servers. The search-only servers are user facing servers and they connect to one of the FTS indexing servers to search the FTS data.
The servers that are not designated as indexing server should not be listed in the AR System Server Group Operation Ranking form.
The following figure shows the FTS plug-ins and FTS high-availability architecture in a server group.
BMC Remedy Full Text Search (FTS) plug-in and high-availability architecture
Primary FTS indexing servers always connect to their own local indexes. The other servers in the group can connect to any of the indexing servers. BMC recommends that all non-Primary FTS servers initially connect to the Primary FTS server which is ranked 1. The Server-Plugin-Alias parameter in the ar.cfg [or ar.conf] files of the servers specify the initial connection. This initial connection is known as the home connection.
While servicing an FTS request, if an FTS indexing server experiences a connection error, the request attempts to connect to another FTS indexing server (as specified in the AR System Server Group Operation Ranking form) until a server is found or there are no more to try.
For example, consider a scenario where you have two primary FTS indexing servers that are ranked 1 and 2 in the AR System Server Group Operation Ranking form. Your Server-Plugin-Alias parameter points to the server that is ranked 1. If this server experiences a connection error, the connection request goes to the server that is ranked 2. If for some reason even the server that is ranked 2 is down, the request then is returned as an error since no primary FTS indexing servers are available.
- In some cases, the indexes on the indexer servers can become out of sync. This can happen when you create a form and index a field on that form between scheduled indexing scans. If one of the indexer servers is down at the time of the scheduled indexing scan, the indexes on that server will be out of sync until the next scheduled indexing scan.
- A slight load on the database can occur if you are performing re-indexing and your database is running on a low memory.
- Avoid performing a full re-index at the same time on two FTS index servers running in your production environment, as failover will occur if the FTS indexing server is busy while performing a re-index.
- A slight latency in displaying the search results can occur when the failover happens as it is required to establish a connection to the next ranked server for FTS.
BMC Remedy AR System does not support Full Text Search if you have read-only database. For more information on using read-only database, see Using read-only database.