Troubleshooting out-of-memory exceptions in the BMC Remedy Mid Tier
You can perform the following checks while troubleshooting out-of-memory exceptions in Mid Tier:
Checks to be performed in your Tomcat servlet container
- Make sure that multiple contexts for arsys (Mid Tier) are not present. Having multiple contexts for the Mid Tier application can lead to cache corruption, abnormally large cache folder size and out-of-memory errors. This issue is typically observed when Tomcat is used as a servlet container.
- Tomcat instance (or servlet engine instance) used for hosting Mid Tier should not be used for hosting any application other than Mid Tier. If any other application is hosted in the same Tomcat instance (or servlet engine instance), this can have an adverse effect on heap requirements and scalability of Mid Tier.
Following are the recommended JVM parameter settings when the servlet engine hosts only the Mid Tier web application:
- Initial memory pool — 1536 MB (
-Xmsparameter for Java)
- Maximum memory pool — 2048 MB (
-Xmxparameter for Java)
PermGenallocation — 256 MB (
-XX:PermSizeparameter for Java)
The maximum memory pool size of 2048 MB is the minimum recommended value. Depending on the deployment scenario, this may need to be revised. For example, in a deployment scenario with heavy searches of up to 5000 objects being performed by users, maximum memory pool may need to be revised to 2048 MB or more.
When mid tier is deployed in a virtual environment, setting
-Xmx to the same number is recommended as acquiring additional memory after Mid Tier startup may not be possible in virtual environment.
Application level checks
- In Midtier/WEB-INF/lib, make sure that multiple copies of same jar file are not present. For example, when you apply any hot-fix, the Midtier.jar is copied to WEB-INF/lib and the older Midtier.jar is backed up to Midtier_backup.jar. Due to this Tomcat loads the redundant libraries/jar and this could result in unexpected behavior.
To resolve this, it is recommended to rename Midtier.jar to a non-Jar file such as Midtier.jar.bak or remove it from the lib folder completely. Verify that the Mid Tier does not get deployed twice due to adding it to the webapps folder of the servlet container.
- Check the Mid Tier Configuration Tool - Connection Settings page.
The value of Maximum Connections per Server parameter should be set to 80 which is its default value.
If the value of the Maximum Connections per Server parameter is set at a higher number, it might have a undesirable effect on heap consumption. See Connection Pool Setting parameters.
AR System server level checks
In the AR System Configuration Generic UI form, check the values for the following parameters:
Specifies whether unqualified searches can be performed on the AR System server. Valid values are T (allows unqualified searches) or F (disallows unqualified searches). The default value is T. The recommended value is F (disallow). Unqualified searches are
ARGetListEntryWithFieldscalls, in which the qualifier parameter is
NULLor has an operation value of 0 (
AR_COND_OP_NONE). Such searches might cause performance problems because they return all requests for a form. This is especially problematic for large forms. It is recommended to turn-off unqualified searches otherwise the volume of search results could be large resulting in out-of-memory errors on the mid tier. For information about
Allow-Unqual-Queries, see Configuration settings A-B.
The maximum number of requests returned by a search. The default value is no server-defined maximum (entry is not defined). The recommended value is 2000 (recommendation may need to be changed on a case by case basis). Because users can also specify the maximum number of requests returned through Search Preferences in the AR System User Preference form, the actual maximum is the lower of these values. It is recommended to set a value for this parameter as unqualified searches can yield a high number of results resulting in unpredictable system behavior. Increasing it would affect the heap size requirement of mid tier and cause an out-of-memory exception.
For information about
Max-Entries-Per-Query, see Configuration settings E-M.
Analyze HeapDumps generated at the time of the out-of-memory error
If out-of-memory errors continue, then study the HeapDump (created when the JVM begins to run out of memory). To do so, add the following Java parameters to the Java options and then use the Mid Tier. A HeapDump is created when an out-of-memory error occurs. For more information about collecting Java HeapDump for AR System server, see Knowledge Base article ID KA000101754.
Pathshould be replaced with Correct Folder path. Please ensure sufficient disk space for the heap dump.
The first parameter requires a dash after the colon as shown. The other parameter does not require the dash.