Performance tuning checklists

The following checklists summarize the actions that are needed to improve BMC Remedy AR System performance:

Creating a problem statement

Characterize throughput or response time.
- Before problem was observed
- After problem was observed

List any system changes that occurred just before the problem was observed.
- Patch changes
- Configuration changes
- Workload changes
- User count changes

List observations
- Is the entire system slow or is the slow performance specific to particular user, particular location, or particular transaction only?
- For global deployments, is the performance issue related to users accessing the system over WAN only? Is it related to a specific site?
- Is the performance issue intermittent? If so, what is the frequency? When are issues occurring?

Monitoring CPU consumption

Collect CPU usage data for each computer in the configuration.
Make sure less than 70% of the total CPU capacity is used on each system in the configuration.
Record which processes are consuming most of the CPU resources.
Record whether I/O wait time is a substantial component of CPU usage. This indicates that the I/O subsystem is not keeping up with demand.

Monitoring memory consumption

Make sure that no system in the configuration is running out of physical memory and doing swapping.
Track memory growth over time to identify any memory leaks.

Tuning the mid tier

This section provides information and guidelines for tuning the mid tier.

Fine tuning the network for HTTP protocol

Turn on HTTP keep-alive for all network segments in your network infrastructure.
If your HTTP keep-alive service supports the following optional HTTP keep-alive parameters, use the following values:
- Keep-alive count: Infinite (minimum 5000)
- Connection timeout: 90000 ms (minimum 60000 ms)
If HTTP session affinity is necessary, set up load balancing for the web tier by using cookie insert rather than source IP binding.
If deployment is over HTTPS, off-load SSL handling to a hardware load balancer when possible.
If deployment is over HTTPS, secure only the necessary network segments because additional encryption and decryption takes time and resources.

Fine tuning the web stack

For Java 8 and above, set the host JVM size as:
-XX:MaxMetaspaceSize=512m
For earlier Java versions, set the host JVM PermSize as follows:
-XX:PermSize=256m
If you have more custom Data Visualization Modules, you might need to increase this item. Monitor the JVM for the correct size.

Select your Garbage Collection (GC) according to the following table but monitor, compare, and confirm that the selection is best because GC behaves differently for different hardware and heap size.

	Low-pause		Throughput
	1 CPU	2+ CPU	1 CPU	2+ CPU
Young generation	Copying Collector (default)	Parallel Copying Collector `-XX:+UseParNewGC` Important: Remove this parameter on Java 11.	Copying Collector (default)	Parallel Scavenge Collector `-XX:UseParallelGC`
Old generation	Mark-Compact Collector (default)	Concurrent Collector `-XX:+UseConcMarkSweepGC`	Mark-Compact Collector (default)	Mark-Compact Collector (default)
Permanent generation	GC not configurable but can be turned off via `JVM arg -Xnoclassgc`

Activate the 64-bit JVM hybrid mode:
-XX:+UseCompressedOops
If you run into JVM stability issues, see JVM heap size setting.
Set the host JVM heap size:
-Xms4096m -Xmx4096m
Configure your Tomcat connector as follows (for standard port 80):
<Connector URIEncoding="UTF-8" acceptCount="100" connectionTimeout="60000"maxHttpHeaderSize="8192" maxKeepAliveRequests="5000" maxThreads="600" port="80" protocol="HTTP/1.1" redirectPort="8443"/>

Fine tuning the mid tier web application

To fine tune the mid tier web application, set the values as described in the following table.

Mid Tier parameter or service	Recommended value
Enable Cache Persistence (mid tier cache serialization)	For a production environment, this parameter is always set to on.
Prefetch or preload service	Use prefetch only when a specific set of AR System forms are known. Best practice We recommend using preload.
Recommended preload procedure	Turn on Enable Cache Persistence. Turn on preload. Allow preload to finish preloading all user-facing AR System forms. Turn off preload (allowing statistical service to take over).
`arsystem.formhtmljs_expiry_interval` `arsystem.resource_expiry_interval` (Resource Check Interval)	Set both parameters to the same value to reflect how often you want the browser to check with the mid tier for updates. Best practice We recommend that you set these parameters to 86400 (1 day). However, if your deployment environment is not changed, you can set the value higher, such as 604800 (1 week). For the new values to take effect, restart the mid tier.
Definition Change Check Interval	In a production environment, you can deselect the Perform Check parameter, and use manual synchronization when your changes are applied to the AR System server.
`arsystem.log_category` `arsystem.log_level`	`arsystem.log_category=INTERNAL` `arsystem.log_level=Severe` These settings can also be applied by using the Mid Tier Configuration Tool.
Connection pool settings	Configure by using the Mid Tier Configuration Tool to avoid having to restart the mid tier. The changes take effect you click Save. For more information, see Configuring the mid tier connection pool. Maximum Connections per Server — Set to one-third the anticipated number of maximum concurrent users. Idle Connections per Server — Use the default value of 5 unless your need is different. Connection Timeout — Set to slightly less than the TCP idle time for your load balancer (depending on the network latency between the mid tier and the AR System server). If no load balancer is used, use the default value. Connection Lifespan — Enable and set the value to 15 minutes. If you do not have a load balancer, you can use this setting to prevent stale connections.

Browser hardware requirements and settings

Make sure that client hardware meets the following minimum recommendation: duo core CPU with 4 GB RAM.
If Internet Explorer 6 or later is used, make sure the browser observes the cache directives.

Tuning the AR System server

Allocate AR System server resources.
- Create a dedicated integration server and move resource-intensive batch or background processes to this server.
- Set fast threads to 3 times the number of CPU cores, and set list threads set to 5 times the number of CPU cores. Fine tune later if needed.
Use OS tools such as Performance Monitor (Microsoft Windows) to monitor the server.
Set the Max-Entries-Per-Query option to no more than 2000 entries.
Design efficient queries.
Increase the number of threads in the queue until your API log consistently shows thread wait times of 0.
To limit the number of AR System server threads, assign a Private-RPC-Socket (private queue) and set the appropriate minimum and maximum threads.
For any component that runs as a C plug-in, consider the number of fast, list, escalation, private threads that are likely to communicate simultaneously with the particular type of plug-in (AREA, ARDBC, Filter API) and set the appropriate option in the ar.cfg (ar.conf) file (Plugin-AREA-Threads, Plugin-ARDBC-Threads, and Plugin-Filter-API-Threads).
Set the <numCoreThreads> value in the pluginsvr_config.xml file to the value that provides maximum performance without consuming too many resources.
For BMC Atrium CMDB, set up a private queue with minimum and maximum thread settings of 2 and 2 as the starting values. Adjust the thread settings based on the BMC Atrium CMDB workload and thorough performance testing.
For the Full Text Search (FTS) private queue, adjust the default thread setting of 1 upward if there is a backlog of entries to be indexed.
To adjust the number of sender threads, edit the following options in the EmailDaemon.properties file as needed:

com.bmc.arsys.emaildaemon.NumberOfSenderThreads=<number>
com.bmc.arsys.emaildaemon.MailboxPollingUnitIsMinutes=false

Set the following AR System configuration parameters:

Parameter	Recommended value
`Private-RPC-Socket (Fast:390620)`	3 times the number of CPUs
`Private-RPC-Socket (List:390635)`	5 times the number of CPUs
`Next-ID-Block-Size`	100
`Server-Side-Table-Chunk-Size`	1000
`Allow-Unqual-Queries`	F (disallow)
`Cache-Mode`	0 (development mode off)
`Debug-mode`	0 (all logging off for a production environment)
`Minimum-API-Version`	Set to a value appropriate for your production environment
`CMDB-Cache-Refresh-Interval`	600 (10 minutes)

For more information on Tuning the AR System server, see the blog AR Server shared on BMC Communities.

Tuning an Oracle database

When tuning an AR System Oracle database server, do the following:

Make sure that alert.log file is clean.
Use an appropriately sized System Global Area (SGA).
Set cursor_sharing to EXACT (default).
- Use Automatic Workload Repository (AWR) reports to collect diagnostics data. Review the top-timed events for potential bottlenecks.
- Monitor the database for the top SQL statement. You can use an AWR report for this purpose. Make sure that the SQL execution plans are optimal.
Set b_tree_bitmap_plans to False
If high database space growth is a concern, convert CLOB storage to In Row. This will also improve performance in few areas.
If you need to make the Oracle database case-insensitive, set the NLS_COMP and NLS_SORT parameters in Oracle 10g.
For expensive SQL statements:
- Make sure that the database statistics are current and representative of the data volume and data distribution.
- Review the execution plan and look for common red flags such as Merge Join Cartesian operation, full table scans, index full scan, bitmap from/to conversion, and so on.
- Implement additional indexes if appropriate. BMC provides indexes out of the box. However, there might be cases where additional indexes are required based on the way the application is configured and used. Typically these requirements are documented in product manuals.

Tuning an SQL database

Set PARAMETERIZATION option to SIMPLE for the AR System database.
Set the READ_COMMITTED_SNAPSHOT database option to ON for the AR System database.
Set the SQL Server MAXDOP option to the appropriate value.