This documentation supports the 9.0 version of Remedy Action Request System.

To view the latest version, select the version from the Product version menu.

Configuring a cluster

Notes

  • BMC recommends you to configure a BMC Remedy Mid Tier cluster using Tomcat.
  • BMC does not supports the clustering for webservers other than Tomcat.
  • BMC Remedy AR System installer allows you to only install or upgrade the BMC Remedy Mid Tier and does not provides an option for Tomcat clustering.
  • You should setup the cluster manually, following the steps provided in this topic.

See the  Compatibility matrix  in the BMC Remedy ITSM Deployment online documentation for supported version of Tomcat. 

Ensure seamless failover by configuring the mid tier on an Apache Tomcat clusterTo add the mid tier in a Tomcat cluster, install mid tiers on different virtual computers and configure the cluster with the following procedure.

  1. Stop Tomcat.

  2. Open the arsys.xml file located in the TomcatInstallationFolder\conf\Catalina\localhost folder in a text editor, and replace the <Manager> tag with the following XML:

     <Manager className="org.apache.catalina.ha.session.DeltaManager"
       expireSessionsOnShutdown="false"
       notifyListenersOnReplication="true" />
    
  3. Open the server.xml file located in the TomcatInstallationFolder\conf folder in a text editor and make the following changes:
    • Locate the <Engine> tag and add the jvmRoute attribute.
      Use Node1 for the first node, Node2 for the second, and so on.
      <Engine name="Catalina" defaultHost="localhost" jvmRoute="Node1">

    • Copy the <Cluster> tag information from the attached file and paste it after the <Engine> tag in your server.xml file. You can download the sample server.xml file. 

      Note

      If you are using Tomcat version 8.0.38 and above, you must remove the following XML tag from the server.xml file.

      <ClusterListener className="org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener"/>

      For versions of Tomcat prior to 8.0.38, you can use the server.xml file as is without making any changes.

      <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster" channelSendOptions="8">
      <Manager className="org.apache.catalina.ha.session.DeltaManager"
                         expireSessionsOnShutdown="false"
                         notifyListenersOnReplication="true"/>
      <Channel className="org.apache.catalina.tribes.group.GroupChannel">
                <Membership className="org.apache.catalina.tribes.
      membership.McastService"
                              address="<ipAddress>" 
                              port="45570"
                              frequency="500"
                              dropTime="3000"/>
              <Receiver className="org.apache.catalina.tribes.transport.
      nio.NioReceiver"
                            address="auto"
                            port="4000"
                            autoBind="100"
                            selectorTimeout="5000"
                            maxThreads="6"/>
              <Sender className="org.apache.catalina.tribes.transport.
      ReplicationTransmitter">
                    <Transport className="org.apache.catalina.tribes.transport.
      nio.PooledParallelSender"/>
              </Sender>
      <Interceptor className="org.apache.catalina.tribes.group.
      interceptors.TcpFailureDetector"/>
            <Interceptor className="org.apache.catalina.tribes.group.
      interceptors.MessageDispatch15Interceptor"/>
      </Channel>
       <Valve className="org.apache.catalina.ha.tcp.ReplicationValve"
      filter=".*\.gif|.*\.js|.*\.jpeg|.*\.jpg|.*\.png|.*\.htm|.*\.html|.*\.css|.*\.txt|.*\.jsp|.*\.swf|.*BackChannel/*|.*./resources/.*|.*./sharedresources/.*|.*./plugins/.*|.*./pluginsignal/.*|.*./imagepool/.*"
      statistics="false" />
      <Valve className="org.apache.catalina.ha.session.JvmRouteBinderValve"/>
       <Deployer className="org.apache.catalina.ha.deploy.FarmWarDeployer"
           tempDir="/tmp/war-temp/"
           deployDir="/tmp/war-deploy/"
           watchDir="/tmp/war-listen/"
           watchEnabled="false"/>
      <ClusterListener className="org.apache.catalina.ha.session.
      JvmRouteSessionIDBinderListener"/>
      <ClusterListener className="org.apache.catalina.ha.session.
      ClusterSessionListener"/>
      </Cluster>
      

      Note

      The default value for the <ipAddress> variable is 228.0.0.4. This IP address and port number defines a unique cluster. However, for each member of a cluster, ensure that they have the same multicast IP address and port number.

  4. Enter the multicast IP address of the host on which you want the session replication to take place in the Address value instead of auto. For example, in the <Receiver> tag, modify address="auto" to address="172.x.x.x". For more information, see FAQ about cluster configuration.
  5. (Optional) Add two new file handlers 5cluster.org.apache.juli.FileHandler and 6cluster.org.apache.juli.FileHandler in the handlers section to the logging.properties file.

    handlers = 1catalina.org.apache.juli.FileHandler, \
               2localhost.org.apache.juli.FileHandler, \
               3manager.org.apache.juli.FileHandler, \
               5cluster.org.apache.juli.FileHandler, \
    		   6cluster.org.apache.juli.FileHandler, \ 
    		   java.util.logging.ConsoleHandler
  6. (Optional) To increase the cluster logging, change the log level to FINE for the following file handlers in the logging.properties file located in the TomcatInstallationFolder/conf folder:
    5cluster.org.apache.juli.FileHandler and 6cluster.org.apache.juli.FileHandler.

    5cluster.org.apache.juli.FileHandler.level = INFO
    5cluster.org.apache.juli.FileHandler.directory = ${catalina.base}/logs
    5cluster.org.apache.juli.FileHandler.prefix = cluster.
     
    6cluster.org.apache.juli.FileHandler.level = INFO
    6cluster.org.apache.juli.FileHandler.directory = ${catalina.base}/logs
    6cluster.org.apache.juli.FileHandler.prefix = ha.
     
    org.apache.catalina.tribes.MESSAGES.level = INFO
    org.apache.catalina.tribes.MESSAGES.handlers = 5cluster.org.apache.juli.FileHandler
    
    org.apache.catalina.tribes.level = INFO
    org.apache.catalina.tribes.handlers = 5cluster.org.apache.juli.FileHandler
    
    
    org.apache.catalina.ha.level = INFO
    org.apache.catalina.ha.handlers = 6cluster.org.apache.juli.FileHandler
  7. Restart Tomcat.
  8. Verify whether the node is added to the cluster:
    1. Open the ha.date.log file located at /opt/apache/tomcat7.0/logs for Linux and <C:\Program Files\Apache Software Foundation\Tomcat7.0\logs> for Windows.
    2. Search for memberAdded entry. If you have added four mid tiers to the cluster, you should see three memberAdded entries and their IP addresses in the file for the other three members. Ensure that the IP address does not represent a localhost.

Note

To add an extra mid tier to the cluster, perform the same configurations on the extra (n+1) mid tier.

To access the mid tier configuration URL, BMC recommends that you use the HTTP URL in the following format:

http://mtMachineName:8080/arsys/shared/config/config.jsp

Recommendations

BMC recommends that to improve application scalability, increase the File Descriptor limit in mid tiers running in a cluster. If you have a cluster of two mid tiers with 12 tenants, which is serving 3600 users, set the File Descriptor limit to 35,000.

Note

Average load per tenant is considered as 300 users.

To set the File Descriptor limit, perform the following steps:

  • Edit the /etc/security/limits.conf file and add the following:

root  soft  nofile   35000

  • Edit the /etc/bashrc file and set the unlimit value as follows:
if [ "$EUID" = "0" ]; then

    ulimit -n 35000

fi
  • Edit the /etc/sysctl.conf file and increase the File Descriptor limit for the system as follows:

fs.file-max = 210000

Note

Ensure that you set the system-wide limit of File Descriptors to approximately 6 times of the per process limit set in the limits.conf file.

Refer to the following table for guidance on deciding the File Descriptor limit:

S NoComponentVariableDescription
1Tomcat connections

maxThreads setting in Tomcat HTTP Connector

maxThreads defines the maximum connections used by Tomcat when
BIO Connector is used.

2

Tomcat NIO Receivers

Default setting is 6

NIO Receivers for Session Replications.

3

Tomcat and Mid Tier Class Loaders

Estimate is 1200Class loader for JAR files loaded by Tomcat. This includes JARs in Mid Tier distribution, configured Third-party JARs, and DVF plug-ins.
4

Mid Tier properties files

1 + 1 * Number of Tenants

1 for global configuration and 1 per tenant

5

Mid Tier Cache files

27 x 2 = 5427 categories x 2 files per category. Data files are always open.
Index files are read into Memory at startup and closed and later written to
during Mid Tier shutdown.
6

Mid Tier Cache lock file

1

Single file for Cache lock status

7Java API RPC connections to AR Serverarsystem.pooling_max_connections_per_server from Mid Tier Configuration * Number of TenantsConfigured by arsystem.pooling_max_connections_per_server setting
in Mid Tier Configuration
8Java API JMS Connections for CCS1 + 1 * Number of Tenants1 for ECCS and 1 per Tenant
9

Attachments and Reports

Estimate is 20% * Maximum Concurrent Users on the Mid-Tier Node

Attachment files and report files created by MT on Disk

10

SSO Agent - Files

1

A single Configuration file is used by the SSO Agent which is queried from the SSO Server

11SSO Agent – Connections to SSO Server

maxThreads setting in Tomcat HTTP Connector

SSO Agent opens a URLConnection to SSO Server when it logs in a
User or validates SSO Token with SSO Server
12Safety Factor25% * Sum of (#1 to #11) 
 Total Sum of #1 to #12 rounded up to nearest 5000 on 6th higher side

Related topics

Was this page helpful? Yes No Submitting... Thank you

Comments

  1. Andreas Mitterdorfer

    The logging section in 5. is missing following:

    You have to add

    5cluster.org.apache.juli.FileHandler and 6cluster.org.apache.juli.FileHandler

     to the handlers section in logging.properties.

    Added lines in bold:

    handlers = 1catalina.org.apache.juli.FileHandler, 2localhost.org.apache.juli.FileHandler, 3manager.org.apache.juli.FileHandler, 4host-manager.org.apache.juli.FileHandler, 5cluster.org.apache.juli.FileHandler, 6cluster.org.apache.juli.FileHandler, java.util.logging.ConsoleHandler

    Sep 11, 2015 01:32
    1. Prachi Kalyani

      Hello Andreas,

      I have updated the topic.

      Thanks,

      Prachi

      Sep 11, 2015 08:02
  2. Dirk Reuter

    Hi,

    Is it possible to configure a BMC Remedy Mid Tier cluster with a load balancer between the cluster and i.e 10 AR Server?

    On "https://docs.bmc.com/docs/display/public/ars201401/Tomcat+cluster+architecture" only a configuration without Loadbalancer is shown and the sentence "Use the symmetric configuration in which both mid tiers use the same AR System server" is a little bit confusing.

    Kind regards,

    Dirk

    Sep 27, 2016 04:56