This documentation applies to the 8.1 version of Remedy Action Request System, which is in "End of Version Support."

To view the latest version, select the version from the Product version menu.

Performance tuning checklists

The following checklists summarize the actions that are needed to improve BMC Remedy AR System performance:

Creating a problem statement

  • Characterize throughput or response time.
    • Before problem was observed
    • After problem was observed
  • List any system changes that occurred just before the problem was observed.
    • Patch changes
    • Configuration changes
    • Workload changes
    • User count changes
  • List observations
    • Is the entire system slow or is the slow performance specific to particular user, particular location, or particular transaction only?
    • For global deployments, is the performance issue related to users accessing the system over WAN only? Is it related to a specific site?
    • Is the performance issue intermittent? If so, what is the frequency? When are issues occurring?

Monitoring CPU consumption

  • Collect CPU usage data for each computer in the configuration.
  • Make sure less than 70% of the total CPU capacity is used on each system in the configuration.
  • Record which processes are consuming most of the CPU resources.
  • Record whether I/O wait time is a substantial component of CPU usage. This indicates that the I/O subsystem is not keeping up with demand.

Monitoring memory consumption

  • Make sure that no system in the configuration is running out of physical memory and doing swapping.
  • Be aware of 64-bit and 32-bit limitations on the OS and applications.
  • Track memory growth over time to identify any memory leaks.

Tuning the mid tier

This section provides information and guidelines for tuning the mid tier.

Fine tuning the network for HTTP protocol

  • Turn on HTTP keep-alive for all network segments in your network infrastructure.
  • If your HTTP keep-alive service supports the following optional HTTP keep-alive parameters, use the following values:
    • Keep-alive count: Infinite (minimum 5000)
    • Connection timeout: 90000 ms (minimum 60000 ms)
  • If HTTP session affinity is necessary, set up load balancing for the web tier by using cookie insert rather than source IP binding.
  • If deployment is over HTTPS, off-load SSL handling to a hardware load balancer when possible.
  • If deployment is over HTTPS, secure only the necessary network segments because additional encryption and decryption takes time and resources.

Fine tuning the web stack

  • Set the host JVM PermSize as follows:
    -XX:PermSize=256m
    If you have more custom Data Visualization Modules, you might need to increase this item. Monitor the JVM for the correct size.
  • Select your Garbage Collection (GC) according to the following table but monitor, compare, and confirm that the selection is best because GC behaves differently for different hardware and heap size.

     Low-pauseThroughput
     1 CPU2+ CPU1 CPU2+ CPU
    Young generationCopying Collector (default)

    Parallel Copying Collector

    -XX:+UseParNewGC

    Copying Collector (default)

    Parallel Scavenge Collector

    -XX:UseParallelGC

    Old generationMark-Compact Collector (default)

    Concurrent Collector

    -XX:+UseConcMarkSweepGC

    Mark-Compact Collector (default)Mark-Compact Collector (default)
    Permanent generationGC not configurable but can be turned off via JVM arg -Xnoclassgc
  • Activate the 64-bit JVM hybrid mode:
    -XX:+UseCompressedOops
    If you run into JVM stability issues, see JVM heap size setting.
  • Set the host JVM heap size:
    -Ms2048m -Mx2048m
  • Configure your Tomcat connector as follows (for standard port 80):
    <Connector URIEncoding="UTF-8" acceptCount="100" connectionTimeout="60000"maxHttpHeaderSize="8192" maxKeepAliveRequests="5000" maxThreads="600" port="80" protocol="HTTP/1.1" redirectPort="8443"/>

Fine tuning the mid tier web application

  • To fine tune the mid tier web application, set the values as described in the following table.

Mid tier parameter or service

Recommended value

Enable Cache Persistence (mid tier cache serialization)

For a production environment, this parameter is always set to on.

Prefetch or preload service

Use prefetch only when a specific set of AR System forms are known. BMC recommends using preload.

Recommended preload procedure

  1. Turn on Enable Cache Persistence.
  2. Turn on preload.
  3. Allow preload to finish preloading all user-facing AR System forms.
  4. Turn off preload (allowing statistical service to take over).

arsystem.formhtmljs_expiry_interval
arsystem.resource_expiry_interval

Set both parameters to the same value to reflect how often you want the browser to check with the mid tier for updates.

BMC recommends setting these to 84600 (1 day). However, if your deployment environment is not changed, you can set the value higher, such as 604800 (1 week). For the new values to take effect, restart the mid tier.

Definition Change Check Interval

In a production environment, turn this parameter off and use manual synchronization when your changes are applied to the AR System server.

arsystem.log_category
arsystem.log_level

arsystem.log_category=INTERNAL
arsystem.log_level=Severe

These settings can also be applied by using the Mid Tier Configuration Tool.

Connection pool settings

Configure by using the Mid Tier Configuration Tool to avoid having to restart the mid tier. The changes take effect you click Save. For more information, see Configuring the mid tier connection pool.

  • Maximum Connections per Server — Set to one-third the anticipated number of maximum concurrent users.
  • Idle Connections per Server — Use the default value of 5 unless your need is different.
  • Connection Timeout — Set to slightly less than the TCP idle time for your load balancer (depending on the network latency between the mid tier and the AR System server). If no load balancer is used, use the default value.
  • Connection Lifespan — Enable and set the value to 15 minutes. If you do not have a load balancer, you can use this setting to prevent stale connections.

Browser hardware requirements and settings

  • Make sure that client hardware meets the following minimum recommendation: duo core CPU with 4 GB RAM.
  • If Internet Explorer 6 or later is used, make sure the browser observes the cache directives.

Tuning the AR System server

  • Allocate AR System server resources.
    • Create a dedicated integration server and move resource-intensive batch or background processes to this server.
    • Set fast threads to 3 times the number of CPU cores, and set list threads set to 5 times the number of CPU cores. Fine tune later if needed.
  • Use OS tools such as Performance Monitor (Microsoft Windows) to monitor the server.
  • Because of the heavy memory usage of a CopyCache operation, manage administrative changes through user behavior and correct configuration:
    • Perform administrative changes during nonpeak hours.
    • Set the Max-Num-Caches option in the ar.cfg (ar.conf) file to manually set a limit to the number of cache copies that can exist at one time.
    • Set the Delay-Recache-Time option in the ar.cfg (ar.conf) file to the number of seconds to wait before the server makes the latest cache available to all threads. The recommended value is 300 (5 minutes).
  • Set the Max-Entries-Per-Query option to no more than 2000 entries.
  • Design efficient queries.
  • For Oracle Solaris, use libumem for memory management.
  • Increase the number of threads in the queue until your API log consistently shows thread wait times of 0.
  • To limit the number of AR System server threads, assign a Private-RPC-Socket (private queue) and set the appropriate minimum and maximum threads.
  • For any component that runs as a C plug-in, consider the number of fast, list, escalation, private threads that are likely to communicate simultaneously with the particular type of plug-in (AREA, ARDBC, Filter API) and set the appropriate option in the ar.cfg (ar.conf) file (Plugin-AREA-Threads, Plugin-ARDBC-Threads, and Plugin-Filter-API-Threads).
  • Set the <numCoreThreads> value in the pluginsvr_config.xml file to the value that provides maximum performance without consuming too many resources.
  • For BMC Atrium CMDB, set up a private queue with minimum and maximum thread settings of 2 and 2 as the starting values. Adjust the thread settings based on the BMC Atrium CMDB workload and thorough performance testing.
  • For the Full Text Search (FTS) private queue, adjust the default thread setting of 1 upward if there is a backlog of entries to be indexed.
  • To adjust the number of sender threads, edit the following options in the EmailDaemon.properties file as needed:

      com.bmc.arsys.emaildaemon.NumberOfSenderThreads=<number>
      com.bmc.arsys.emaildaemon.MailboxPollingUnitIsMinutes=false

  • Set the following AR System configuration parameters:

    Parameter

    Recommended value

    Private-RPC-Socket (Fast:390620)

    3 times the number of CPUs

    Private-RPC-Socket (List:390635)

    5 times the number of CPUs

    Next-ID-Block-Size

    100

    Server-Side-Table-Chunk-Size

    1000

    Allow-Unqual-Queries

    F (disallow)

    Cache-Mode

    0 (development mode off)

    Debug-mode

    0 (all logging off for a production environment)

    Minimum-API-Version

    Set to a value appropriate for your production environment

    CMDB-Cache-Refresh-Interval

    600 (10 minutes)

For more information on Tuning the AR System server, see the blog AR Server shared on BMC Communities.

Tuning an Oracle database

When tuning an AR System Oracle database server, do the following:

  • Make sure that alert.log file is clean.
  • Use an appropriately sized System Global Area (SGA).
  • Set cursor_sharingto FORCE.
    • Use Automatic Workload Repository (AWR) reports to collect diagnostics data. Review the top-timed events for potential bottlenecks.
    • Monitor the database for the top SQL statement. You can use an AWR report for this purpose. Make sure that the SQL execution plans are optimal.
  • If high database space growth is a concern, convert CLOB storage to In Row.
  • If you need to make the Oracle database case-insensitive, set the NLS_COMP and NLS_SORT parameters in Oracle 10g.
  • For expensive SQL statements:
    • Make sure that the database statistics are current and representative of the data volume and data distribution.
    • Review the execution plan and look for common red flags such as Merge Join Cartesian operation, full table scans, index full scan, bitmap from/to conversion, and so on.
    • Implement additional indexes if appropriate. BMC provides indexes out of the box. However, there might be cases where additional indexes are required based on the way the application is configured and used. Typically these requirements are documented in product manuals.

Tuning an SQL database

This version of the documentation is no longer supported. However, the documentation is available for your convenience. You will not be able to leave comments.

Comments

  1. Mauricio Perez

    Hi, i am having some troubles with Garbage Collection (GC) table.

    How to discover if my environment is Young, Old or Permanent generation? And about Low-pause/Throughput?

    Feb 06, 2014 10:17
  2. Aparna Saxena

    Thank you for your comment, Mauricio.

    You might want to refer to - JVM runtime analysis. Do let me know if this helps.

    Thanks,

    Aparna

    Feb 07, 2014 01:26
  3. Mauricio Perez

    Hello Arpana,

    The provided link have helped me, thanks!

    Kind regards,

    Mauricio.

    Feb 17, 2014 07:43
  4. Tom Zmijewski

    I noticed in the recommendations above the settings for maxKeepAliveRequests and connectionTimeout are more in line with the default settings within the tomcat configuration than I've seen recommended in previous versions.

    In the 7.6 Performance Tuning for BSM document, for example, the recommended settings are:

    maxKeepAliveRequests = infinite

    connectionTimeout="90000"

    Has there been an update or change to the structure of the web server or midtier in 8.1 to bump these settings down or is this meant to be a more generic/base recommendation to be increased if necessary?

    Thanks,

    Tom

    Aug 06, 2014 03:17
    1. James Wagner

      Hi Tom,

      Thanks for your comment. I will contact a subject matter expert on whether the parameters in question are considered "generic" and for appropriate values if they should be changed. I will post a follow-up as soon as progress is made.

      Thanks - Jim Wagner

      Aug 08, 2014 04:45