Performance tuning checklists
The following checklists summarize the actions that are needed to improve BMC Remedy AR System performance:
Creating a problem statement
- Characterize throughput or response time.
- Before problem was observed
- After problem was observed
- List any system changes that occurred just before the problem was observed.
- Patch changes
- Configuration changes
- Workload changes
- User count changes
- List observations
- Is the entire system slow or is the slow performance specific to particular user, particular location, or particular transaction only?
- For global deployments, is the performance issue related to users accessing the system over WAN only? Is it related to a specific site?
- Is the performance issue intermittent? If so, what is the frequency? When are issues occurring?
Monitoring CPU consumption
- Collect CPU usage data for each computer in the configuration.
- Make sure less than 70% of the total CPU capacity is used on each system in the configuration.
- Record which processes are consuming most of the CPU resources.
- Record whether I/O wait time is a substantial component of CPU usage. This indicates that the I/O subsystem is not keeping up with demand.
Monitoring memory consumption
- Make sure that no system in the configuration is running out of physical memory and doing swapping.
- Be aware of 64-bit and 32-bit limitations on the OS and applications.
- Track memory growth over time to identify any memory leaks.
Tuning the mid tier
This section provides information and guidelines for tuning the mid tier.
Fine tuning the network for HTTP protocol
- Turn on HTTP keep-alive for all network segments in your network infrastructure.
- If your HTTP keep-alive service supports the following optional HTTP keep-alive parameters, use the following values:
- Keep-alive count: Infinite (minimum 5000)
- Connection timeout: 90000 ms (minimum 60000 ms)
- If HTTP session affinity is necessary, set up load balancing for the web tier by using cookie insert rather than source IP binding.
- If deployment is over HTTPS, off-load SSL handling to a hardware load balancer when possible.
- If deployment is over HTTPS, secure only the necessary network segments because additional encryption and decryption takes time and resources.
Fine tuning the web stack
- Set the host JVM PermSize as follows:
If you have more custom Data Visualization Modules, you might need to increase this item. Monitor the JVM for the correct size.
Select your Garbage Collection (GC) according to the following table but monitor, compare, and confirm that the selection is best because GC behaves differently for different hardware and heap size.
Low-pause Throughput 1 CPU 2+ CPU 1 CPU 2+ CPU Young generation Copying Collector (default)
Parallel Copying Collector
Copying Collector (default)
Parallel Scavenge Collector
Old generation Mark-Compact Collector (default)
Mark-Compact Collector (default) Mark-Compact Collector (default) Permanent generation GC not configurable but can be turned off via
JVM arg -Xnoclassgc
- Activate the 64-bit JVM hybrid mode:
If you run into JVM stability issues, see JVM heap size setting.
- Set the host JVM heap size:
- Configure your Tomcat connector as follows (for standard port 80):
<Connector URIEncoding="UTF-8" acceptCount="100" connectionTimeout="60000"maxHttpHeaderSize="8192" maxKeepAliveRequests="5000" maxThreads="600" port="80" protocol="HTTP/1.1" redirectPort="8443"/>
Fine tuning the mid tier web application
- To fine tune the mid tier web application, set the values as described in the following table.
Mid tier parameter or service
Enable Cache Persistence (mid tier cache serialization)
For a production environment, this parameter is always set to on.
Prefetch or preload service
Use prefetch only when a specific set of AR System forms are known. BMC recommends using preload.
Recommended preload procedure
Set both parameters to the same value to reflect how often you want the browser to check with the mid tier for updates.
Definition Change Check Interval
In a production environment, turn this parameter off and use manual synchronization when your changes are applied to the AR System server.
Connection pool settings
Configure by using the Mid Tier Configuration Tool to avoid having to restart the mid tier. The changes take effect you click Save. For more information, see Configuring the mid tier connection pool.
Browser hardware requirements and settings
- Make sure that client hardware meets the following minimum recommendation: duo core CPU with 4 GB RAM.
- If Internet Explorer 6 or later is used, make sure the browser observes the cache directives.
Tuning the AR System server
- Allocate AR System server resources.
- Create a dedicated integration server and move resource-intensive batch or background processes to this server.
- Set fast threads to 3 times the number of CPU cores, and set list threads set to 5 times the number of CPU cores. Fine tune later if needed.
- Use OS tools such as Performance Monitor (Microsoft Windows) to monitor the server.
- Because of the heavy memory usage of a CopyCache operation, manage administrative changes through user behavior and correct configuration:
- Perform administrative changes during nonpeak hours.
- Set the Max-Num-Caches option in the ar.cfg (ar.conf) file to manually set a limit to the number of cache copies that can exist at one time.
- Set the Delay-Recache-Time option in the ar.cfg (ar.conf) file to the number of seconds to wait before the server makes the latest cache available to all threads. The recommended value is 300 (5 minutes).
- Set the Max-Entries-Per-Query option to no more than 2000 entries.
- Design efficient queries.
- For Oracle Solaris, use
libumemfor memory management.
- Increase the number of threads in the queue until your API log consistently shows thread wait times of 0.
- To limit the number of AR System server threads, assign a Private-RPC-Socket (private queue) and set the appropriate minimum and maximum threads.
- For any component that runs as a C plug-in, consider the number of fast, list, escalation, private threads that are likely to communicate simultaneously with the particular type of plug-in (AREA, ARDBC, Filter API) and set the appropriate option in the ar.cfg (ar.conf) file (Plugin-AREA-Threads, Plugin-ARDBC-Threads, and Plugin-Filter-API-Threads).
- Set the <numCoreThreads> value in the pluginsvr_config.xml file to the value that provides maximum performance without consuming too many resources.
- For BMC Atrium CMDB, set up a private queue with minimum and maximum thread settings of 2 and 2 as the starting values. Adjust the thread settings based on the BMC Atrium CMDB workload and thorough performance testing.
- For the Full Text Search (FTS) private queue, adjust the default thread setting of 1 upward if there is a backlog of entries to be indexed.
- To adjust the number of sender threads, edit the following options in the EmailDaemon.properties file as needed:
Set the following AR System configuration parameters:
3 times the number of CPUs
5 times the number of CPUs
0 (development mode off)
0 (all logging off for a production environment)
Set to a value appropriate for your production environment
600 (10 minutes)
For more information on Tuning the AR System server, see the blog AR Server shared on BMC Communities.
Tuning an Oracle database
When tuning an AR System Oracle database server, do the following:
- Make sure that alert.log file is clean.
- Use an appropriately sized System Global Area (SGA).
- Use Automatic Workload Repository (AWR) reports to collect diagnostics data. Review the top-timed events for potential bottlenecks.
- Monitor the database for the top SQL statement. You can use an AWR report for this purpose. Make sure that the SQL execution plans are optimal.
- If high database space growth is a concern, convert CLOB storage to In Row.
- If you need to make the Oracle database case-insensitive, set the
NLS_SORTparameters in Oracle 10g.
- For expensive SQL statements:
- Make sure that the database statistics are current and representative of the data volume and data distribution.
- Review the execution plan and look for common red flags such as Merge Join Cartesian operation, full table scans, index full scan, bitmap from/to conversion, and so on.
- Implement additional indexes if appropriate. BMC provides indexes out of the box. However, there might be cases where additional indexes are required based on the way the application is configured and used. Typically these requirements are documented in product manuals.
Tuning an SQL database
- Set PARAMETERIZATION option to FORCED for the AR System database. For more information see, Improving performance with parameterization
- Set the READ_COMMITTED_SNAPSHOT database option to ON for the AR System database. For more information, see Reducing deadlocks by using snapshot isolation
- Set the SQL Server MAXDOP option to the appropriate value.