Default language.

Configuring a cluster

To ensure seamless failover, configure the mid tier on an Apache Tomcat cluster. (BMC does not support the clustering for web servers other than Tomcat.)

The BMC Remedy AR System installer allows you to only install or upgrade the BMC Remedy Mid Tier and does not provides an option for Tomcat clustering. To add the mid tier in a Tomcat cluster, install mid tiers on different virtual computers and configure the cluster manually with the following procedure.

  1. Stop Tomcat.

  2. Open the arsys.xml file located in the TomcatInstallationFolder\conf\Catalina\localhost folder in a text editor, and replace the <Manager> tag with the following XML:

    <Manager className="org.apache.catalina.ha.session.DeltaManager"
      expireSessionsOnShutdown="false"
      notifyListenersOnReplication="true" />
  3. Open the server.xml file located in the TomcatInstallationFolder\conf folder in a text editor and make the following changes:
    • Locate the <Engine> tag and add the jvmRoute attribute.
      Use Node1 for the first node, Node2 for the second, and so on.
      <Engine name="Catalina" defaultHost="localhost" jvmRoute="Node1">

    • Copy the <Cluster> tag information from the attached file and paste it after the <Engine> tag in your server.xml file. You can download the sample server.xml file. 

      Note

      If you are using Tomcat version 8.0.38 and above, you must remove the following XML tag from the server.xml file.

      <ClusterListener className="org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener"/>

      For versions of Tomcat prior to 8.0.38, you can use the server.xml file as is without making any changes.

      <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster" channelSendOptions="8">
      <Manager className="org.apache.catalina.ha.session.DeltaManager"
                         expireSessionsOnShutdown="false"
                         notifyListenersOnReplication="true"/>
      <Channel className="org.apache.catalina.tribes.group.GroupChannel">
               <Membership className="org.apache.catalina.tribes.
      membership.McastService"
                              address="<ipAddress>"
                              port="45570"
                              frequency="500"
                              dropTime="3000"/>
             <Receiver className="org.apache.catalina.tribes.transport.nio.NioReceiver"
                            address="auto"
                            port="4000"
                            autoBind="100"
                            selectorTimeout="5000"
                            maxThreads="6"/>
             <Sender className="org.apache.catalina.tribes.transport.
      ReplicationTransmitter"      >
                   <Transport className="org.apache.catalina.tribes.transport.
      nio.PooledParallelSender"      />
             </Sender>
      <Interceptor className="org.apache.catalina.tribes.group.
      interceptors.TcpFailureDetector"      />
           <Interceptor className="org.apache.catalina.tribes.group.
      interceptors.MessageDispatchInterceptor"      />
      </Channel>
      <Valve className="org.apache.catalina.ha.tcp.ReplicationValve"
      filter=".*\.gif|.*\.js|.*\.jpeg|.*\.jpg|.*\.png|.*\.htm|.*\.html|.*\.css|.*\.txt|.*\.jsp|.*\.swf|.*BackChannel/*|.*./resources/.*|.*./sharedresources/.*|.*./plugins/.*|.*./pluginsignal/.*|.*./imagepool/.*"
      statistics="false" />
      <Valve className="org.apache.catalina.ha.session.JvmRouteBinderValve"/>
      <Deployer className="org.apache.catalina.ha.deploy.FarmWarDeployer"
           tempDir="/tmp/war-temp/"
           deployDir="/tmp/war-deploy/"
           watchDir="/tmp/war-listen/"
           watchEnabled="false"/>
      <ClusterListener className="org.apache.catalina.ha.session.
      JvmRouteSessionIDBinderListener"      />
      <ClusterListener className="org.apache.catalina.ha.session.
      ClusterSessionListener"      />
      </Cluster>

      Note

      The default value for the <ipAddress> variable is 228.0.0.4. This IP address and port number defines a unique cluster. However, for each member of a cluster, ensure that they have the same multicast IP address and port number.

  4. Enter the multicast IP address of the host on which you want the session replication to take place in the Address value instead of auto. For example, in the <Receiver> tag, modify address="auto" to address="172.x.x.x". For more information, see FAQ-about-cluster-configuration.

  5. (Optional) Add two new file handlers 5cluster.org.apache.juli.FileHandler and 6cluster.org.apache.juli.FileHandler in the handlers section to the logging.properties file.

    handlers = 1catalina.org.apache.juli.FileHandler, \
              2localhost.org.apache.juli.FileHandler, \
              3manager.org.apache.juli.FileHandler, \
      4host-manager.org.apache.juli.AsyncFileHandler\
              5cluster.org.apache.juli.FileHandler, \
      6cluster.org.apache.juli.FileHandler, \
       java.util.logging.ConsoleHandler

  6. (Optional) To increase the cluster logging, change the log level to FINE for the following file handlers in the logging.properties file located in the TomcatInstallationFolder/conf folder:
    5cluster.org.apache.juli.FileHandler and 6cluster.org.apache.juli.FileHandler.

    5cluster.org.apache.juli.FileHandler.level = INFO
    5cluster.org.apache.juli.FileHandler.directory = ${catalina.base}/logs
    5cluster.org.apache.juli.FileHandler.prefix = cluster.
     
    6cluster.org.apache.juli.FileHandler.level = INFO
    6cluster.org.apache.juli.FileHandler.directory = ${catalina.base}/logs
    6cluster.org.apache.juli.FileHandler.prefix = ha.
     
    org.apache.catalina.tribes.MESSAGES.level = INFO
    org.apache.catalina.tribes.MESSAGES.handlers = 5cluster.org.apache.juli.FileHandler

    org.apache.catalina.tribes.level = INFO
    org.apache.catalina.tribes.handlers = 5cluster.org.apache.juli.FileHandler


    org.apache.catalina.ha.level = INFO
    org.apache.catalina.ha.handlers = 6cluster.org.apache.juli.FileHandler
  7. Restart Tomcat.
  8. Verify whether the node is added to the cluster:
    1. Open the ha.date.log file located at /opt/apache/tomcat7.0/logs for Linux and <C:\Program Files\Apache Software Foundation\Tomcat7.0\logs> for Windows.
    2. Search for memberAdded entry. If you have added four mid tiers to the cluster, you should see three memberAdded entries and their IP addresses in the file for the other three members. Ensure that the IP address does not represent a localhost.

Note

To add an extra mid tier to the cluster, perform the same configurations on the extra (n+1) mid tier.

To access the mid tier configuration URL, use the HTTP URL in the following format:

http://mtMachineName:8080/arsys/shared/config/config.jsp

Best practices for configuring a cluster

To improve application scalability, we recommend that you increase the File Descriptor limit in mid tiers running in a cluster. If you have a cluster of 2 mid tiers with 12 tenants, which is serving 3600 users, set the File Descriptor limit to 35000.

Note

Average load per tenant is considered as 300 users.

To set the File Descriptor limit, perform the following steps:

  • Edit the /etc/security/limits.conf file and add the following:
root  soft  nofile   35000
  • Edit the /etc/bashrc file and set the unlimit value as follows:
if [ "$EUID" = "0" ]; then

   ulimit -n 35000

fi
  • Edit the /etc/sysctl.conf file and increase the File Descriptor limit for the system as follows:

fs.file-max = 210000

Note

Ensure that you set the system-wide limit of File Descriptors to approximately 6 times of the per process limit set in the limits.conf file.

Refer to the following table for guidance on deciding the File Descriptor limit:

S No

Component

Variable

Description

1

Tomcat connections

maxThreads setting in Tomcat HTTP Connector

maxThreads defines the maximum connections used by Tomcat when
BIO Connector is used.

2

Tomcat NIO Receivers

Default setting is 6

NIO Receivers for Session Replications.

3

Tomcat and Mid Tier Class Loaders

Estimate is 1200

Class loader for JAR files loaded by Tomcat. This includes JARs in Mid Tier distribution, configured Third-party JARs, and DVF plug-ins.

4

Mid Tier properties files

1 + 1 * Number of Tenants

1 for global configuration and 1 per tenant

5

Mid Tier Cache files

27 x 2 = 54

27 categories x 2 files per category. Data files are always open.
Index files are read into Memory at startup and closed and later written to
during Mid Tier shutdown.

6

Mid Tier Cache lock file

1

Single file for Cache lock status

7

Java API RPC connections to AR Server

arsystem.pooling_max_connections_per_server from Mid Tier Configuration * Number of Tenants

Configured by arsystem.pooling_max_connections_per_server setting
in Mid Tier Configuration

8

Java API JMS Connections for CCS

1 + 1 * Number of Tenants

1 for ECCS and 1 per Tenant

9

Attachments and Reports

Estimate is 20% * Maximum Concurrent Users on the Mid-Tier Node

Attachment files and report files created by MT on Disk

10

SSO Agent - Files

1

A single Configuration file is used by the SSO Agent which is queried from the SSO Server

11

SSO Agent – Connections to SSO Server

maxThreads setting in Tomcat HTTP Connector

SSO Agent opens a URLConnection to SSO Server when it logs in a
User or validates SSO Token with SSO Server

12

Safety Factor

25% * Sum of (#1 to #11)



Total


Sum of #1 to #12 rounded up to nearest 5000 on 6th higher side

Related topics

 

Tip: For faster searching, add an asterisk to the end of your partial query. Example: cert*