Error: Invalid spaceKey on retrieving a related space config.

Configuring a cluster

Notes

  • BMC recommends you to configure a BMC Remedy Mid Tier cluster using Tomcat.
  • BMC does not support the clustering for webservers other than Tomcat.
  • BMC Remedy AR System installer allows you to only install or upgrade the BMC Remedy Mid Tier and does not provides an option for Tomcat clustering.
  • You should setup the cluster manually, following the steps provided in this topic.

See the  Compatibility matrix in the BMC Remedy ITSM Deployment online documentation for supported version of Tomcat. 

Ensure seamless failover by configuring the mid tier on an Apache Tomcat clusterTo add the mid tier in a Tomcat cluster, install mid tiers on different virtual computers and configure the cluster with the following procedure.

  1. Stop Tomcat.

  2. Open the arsys.xml file located in the TomcatInstallationFolder\conf\Catalina\localhost folder in a text editor, and replace the <Manager> tag with the following XML:

    <Manager className="org.apache.catalina.ha.session.DeltaManager"
      expireSessionsOnShutdown="false"
      notifyListenersOnReplication="true" />
  3. Open the server.xml file located in the TomcatInstallationFolder\conf folder in a text editor and make the following changes:

    • Locate the <Engine> tag and add the jvmRoute attribute.
      Use Node1 for the first node, Node2 for the second, and so on.
      <Engine name="Catalina" defaultHost="localhost" jvmRoute="Node1">

    • Copy the <Cluster> tag information from the attached file and paste it after the <Engine> tag in your server.xml file. You can download the sample server.xml file. 

      Note

      If you are using Tomcat version 8.0.38 and above, you must remove the following XML tag from the server.xml file.

      <ClusterListener className="org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener"/>

      For versions of Tomcat prior to 8.0.38, you can use the server.xml file as is without making any changes.

      <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster" channelSendOptions="8">
      <Manager className="org.apache.catalina.ha.session.DeltaManager"
                         expireSessionsOnShutdown="false"
                         notifyListenersOnReplication="true"/>
      <Channel className="org.apache.catalina.tribes.group.GroupChannel">
                <Membership className="org.apache.catalina.tribes.
      membership.McastService"
                              address="<ipAddress>" 
                              port="45570"
                              frequency="500"
                              dropTime="3000"/>
              <Receiver className="org.apache.catalina.tribes.transport.
      nio.NioReceiver"
                            address="auto"
                            port="4000"
                            autoBind="100"
                            selectorTimeout="5000"
                            maxThreads="6"/>
              <Sender className="org.apache.catalina.tribes.transport.
      ReplicationTransmitter">
                    <Transport className="org.apache.catalina.tribes.transport.
      nio.PooledParallelSender"/>
              </Sender>
      <Interceptor className="org.apache.catalina.tribes.group.
      interceptors.TcpFailureDetector"/>
            <Interceptor className="org.apache.catalina.tribes.group.
      interceptors.MessageDispatchInterceptor"/>
      </Channel>
       <Valve className="org.apache.catalina.ha.tcp.ReplicationValve"
      filter=".*\.gif|.*\.js|.*\.jpeg|.*\.jpg|.*\.png|.*\.htm|.*\.html|.*\.css|.*\.txt|.*\.jsp|.*\.swf|.*BackChannel/*|.*./resources/.*|.*./sharedresources/.*|.*./plugins/.*|.*./pluginsignal/.*|.*./imagepool/.*"
      statistics="false" />
      <Valve className="org.apache.catalina.ha.session.JvmRouteBinderValve"/>
       <Deployer className="org.apache.catalina.ha.deploy.FarmWarDeployer"
           tempDir="/tmp/war-temp/"
           deployDir="/tmp/war-deploy/"
           watchDir="/tmp/war-listen/"
           watchEnabled="false"/>
      <ClusterListener className="org.apache.catalina.ha.session.
      JvmRouteSessionIDBinderListener"/>
      <ClusterListener className="org.apache.catalina.ha.session.
      ClusterSessionListener"/>
      </Cluster>
      

      Note

      The default value for the <ipAddress> variable is 228.0.0.4. This IP address and port number defines a unique cluster. However, for each member of a cluster, ensure that they have the same multicast IP address and port number.

  4. Enter the multicast IP address of the host on which you want the session replication to take place in the Address value instead of auto. For example, in the <Receiver> tag, modify address="auto" to address="172.x.x.x". For more information, see FAQ about cluster configuration.
  5. (Optional) Add two new file handlers 5cluster.org.apache.juli.FileHandler and 6cluster.org.apache.juli.FileHandler in the handlers section to the logging.properties file.

    handlers = 1catalina.org.apache.juli.FileHandler, \
               2localhost.org.apache.juli.FileHandler, \
               3manager.org.apache.juli.FileHandler, \
               5cluster.org.apache.juli.FileHandler, \
    		   6cluster.org.apache.juli.FileHandler, \ 
    		   java.util.logging.ConsoleHandler
  6. (Optional) To increase the cluster logging, change the log level to FINE for the following file handlers in the logging.properties file located in the TomcatInstallationFolder/conf folder:
    5cluster.org.apache.juli.FileHandler and 6cluster.org.apache.juli.FileHandler.

    5cluster.org.apache.juli.FileHandler.level = INFO
    5cluster.org.apache.juli.FileHandler.directory = ${catalina.base}/logs
    5cluster.org.apache.juli.FileHandler.prefix = cluster.
     
    6cluster.org.apache.juli.FileHandler.level = INFO
    6cluster.org.apache.juli.FileHandler.directory = ${catalina.base}/logs
    6cluster.org.apache.juli.FileHandler.prefix = ha.
     
    org.apache.catalina.tribes.MESSAGES.level = INFO
    org.apache.catalina.tribes.MESSAGES.handlers = 5cluster.org.apache.juli.FileHandler
    
    org.apache.catalina.tribes.level = INFO
    org.apache.catalina.tribes.handlers = 5cluster.org.apache.juli.FileHandler
    
    
    org.apache.catalina.ha.level = INFO
    org.apache.catalina.ha.handlers = 6cluster.org.apache.juli.FileHandler
  7. Restart Tomcat.
  8. Verify whether the node is added to the cluster:
    1. Open the ha.date.log file located at /opt/apache/tomcat7.0/logs for Linux and <C:\Program Files\Apache Software Foundation\Tomcat7.0\logs> for Windows.
    2. Search for memberAdded entry. If you have added four mid tiers to the cluster, you should see three memberAdded entries and their IP addresses in the file for the other three members. Ensure that the IP address does not represent a localhost.

Note

To add an extra mid tier to the cluster, perform the same configurations on the extra (n+1) mid tier.

To access the mid tier configuration URL, BMC recommends that you use the HTTP URL in the following format:

http://mtMachineName:8080/arsys/shared/config/config.jsp

Recommendations

BMC recommends that to improve application scalability, increase the File Descriptor limit in mid tiers running in a cluster. If you have a cluster of two mid tiers with 12 tenants, which is serving 3600 users, set the File Descriptor limit to 35,000.

Note

Average load per tenant is considered as 300 users.

To set the File Descriptor limit, perform the following steps:

  • Edit the /etc/security/limits.conf file and add the following:

root  soft  nofile   35000

  • Edit the /etc/bashrc file and set the unlimit value as follows:
if [ "$EUID" = "0" ]; then

    ulimit -n 35000

fi
  • Edit the /etc/sysctl.conf file and increase the File Descriptor limit for the system as follows:

fs.file-max = 210000

Note

Ensure that you set the system-wide limit of File Descriptors to approximately 6 times of the per process limit set in the limits.conf file.

Refer to the following table for guidance on deciding the File Descriptor limit:

S NoComponentVariableDescription
1Tomcat connections

maxThreads setting in Tomcat HTTP Connector

maxThreads defines the maximum connections used by Tomcat when
BIO Connector is used.

2

Tomcat NIO Receivers

Default setting is 6

NIO Receivers for Session Replications.

3

Tomcat and Mid Tier Class Loaders

Estimate is 1200Class loader for JAR files loaded by Tomcat. This includes JARs in Mid Tier distribution, configured Third-party JARs, and DVF plug-ins.
4

Mid Tier properties files

1 + 1 * Number of Tenants

1 for global configuration and 1 per tenant

5

Mid Tier Cache files

27 x 2 = 5427 categories x 2 files per category. Data files are always open.
Index files are read into Memory at startup and closed and later written to
during Mid Tier shutdown.
6

Mid Tier Cache lock file

1

Single file for Cache lock status

7Java API RPC connections to AR Serverarsystem.pooling_max_connections_per_server from Mid Tier Configuration * Number of TenantsConfigured by arsystem.pooling_max_connections_per_server setting
in Mid Tier Configuration
8Java API JMS Connections for CCS1 + 1 * Number of Tenants1 for ECCS and 1 per Tenant
9

Attachments and Reports

Estimate is 20% * Maximum Concurrent Users on the Mid-Tier Node

Attachment files and report files created by MT on Disk

10

SSO Agent - Files

1

A single Configuration file is used by the SSO Agent which is queried from the SSO Server

11SSO Agent – Connections to SSO Server

maxThreads setting in Tomcat HTTP Connector

SSO Agent opens a URLConnection to SSO Server when it logs in a
User or validates SSO Token with SSO Server
12Safety Factor25% * Sum of (#1 to #11) 
 Total Sum of #1 to #12 rounded up to nearest 5000 on 6th higher side

Related topics

Was this page helpful? Yes No Submitting... Thank you

Comments

  1. Otto Von schwerin

    If you download and use the attached sampel server.xml file in step 2 the tomcat cluster wont work. But if you copy and past from the code on the page it works. If you compare them there are some differences. Ex the filter string.

    Oct 05, 2016 08:13
    1. Anagha Deshpande

      Hello Otto,

      Apologies for the late response.

      The attached server.xml file is a sample file. You need to manually perform the modifications.

      Regards,

      Anagha

       

      Jun 27, 2017 01:16
  2. Stefan Hall

    Point 2 from ARS9.0 documenttaion is missing. It seems important for a working cluster:

    Open the arsys.xml file located in the TomcatInstallationFolder\conf\Catalina\localhost folder in a text editor, and replace the <Manager> tag with the following XML:

    <Manager className="org.apache.catalina.ha.session.DeltaManager"
      expireSessionsOnShutdown="false"
      notifyListenersOnReplication="true" />
    Feb 06, 2017 12:13
    1. Prachi Kalyani

      Hello Stefan,

      We are verifying this. We will updated the topic soon.

      Thanks,

      Prachi

      Feb 13, 2017 06:16
      1. Anagha Deshpande

        Hello Stefan,

        You need not make any change in the arsys.xml file. This is a part of cluster configuration and is referred in the server.xml file.

        Regards,

        Anagha

         

        Jun 27, 2017 01:11
  3. Wu Honda

    And how about <distributable/> tag? is it must configured in /midtier/WEB-INF/web.xml?

     

    Apr 24, 2017 03:19
    1. Nidhi Das

      Hello Wu,

      I will discuss this with the concerned SME and will write back to you.

      Regards,
      Nidhi

      Apr 25, 2017 12:20
      1. Anagha Deshpande

        Hello Wu,

        Out-of-the box the web.xml file has the <distributable/> tag. If the <distributable/> tag is not present in the web.xml file, you need to create it. 

        Regards,

        Anagha

        Jun 27, 2017 01:05
  4. Subhash Sonwane

    It would be great if you create a small demo to explain how to configure tomcat in cluster.

    Aug 29, 2017 01:40
    1. Anagha Deshpande

      Hello Subhash,

      Thank you for your suggestion. We will work on it.

      Regards,

      Anagha 

      Aug 29, 2017 09:51
  5. Stefan Hall

    seems not working under TC8.

    Following the TC documentation it should be , without "15"

    Nov 08, 2017 04:39
    1. Kamalakannan Srinivasan

      Hi Stefan,

      Thank you for your comment. The change has been made to the document.

      Regards,
      Kamal

      Dec 14, 2017 02:51
  6. Alejandro Vidaurri de la cruz

    Agreed Stefan

    It should say 

    <Interceptor className="org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor"/>


    instead of 

    <Interceptor className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor"/>

    Nov 08, 2017 12:55
    1. Kamalakannan Srinivasan

      Hi Alejandro,

      Thank you for your comment. The change has been made to the document.

      Regards,
      Kamal

      Dec 14, 2017 02:50
  7. John Kelley

    Are there any instructions for clustering in a non enabled multicast environment?

    May 03, 2018 01:30
    1. Anagha Deshpande

      Hello John,

      I will check this with the SME and will respond back.

      Regards,

      Anagha


      May 03, 2018 10:23
    1. Anagha Deshpande

      Hello John,

      I will check with the SME and will respond back.

      Regards,

      Anagha

      May 06, 2018 08:32
      1. Anagha Deshpande

        Hello John,

        The example of Tomcat clustering without multicasting is available here.

        Regards,

        Anagha

        Jun 22, 2018 03:35