Performance, Scalability, and Sizing

Configuration guidance

This section offers guidance on appropriate settings for the configuration parameters for a BMC Server Automation Application Server. For details about using the blasadmin tool to control the configuration parameters, see Using the Application Server Administration console (blasadmin) to configure Application Servers.

Apart from some general discussion, this section organizes BMC configuration recommendations according to type for single-purpose Application Servers. An Application Server can, of course, be configured to provide the combined services of the single-purpose Application Server (an Application Server of type ALL). In this case, you must modify the parameter recommendations, usually by adding recommended values for the same parameter for different Application Server types.

For more information about recommended settings for blasadmin parameters, see Sizing Application Servers.

About Java memory

Effective operation of a large Java system like the BMC Server Automation Application Server depends critically on the availability of sufficient heap memory. This section provides an overview of some considerations for correctly sizing Java memory for BMC Server Automation.

Process space, Java heap, and native heap

A Java process comprises two distinct memory areas: the Java heap and the native heap. Both heaps, together with the Java executable code itself, must fit within the footprint of a single process. Increasing the maximum size of the Java heap necessarily decreases the maximum possible size of the native heap that can fit within a certain process size.

The java heap contains Java objects and accounts for most of the memory required by a running Application Server. The Java heap is managed by the Java garbage collector (GC), and so is sometimes called GC heap.

The native heap (also sometimes called the C heap) contains thread stacks, file handles, and other objects not managed by the Java garbage collector.

If the maximum Java heap size is set too low, it is possible to run out of Java heap memory. If the maximum Java heap size is set too high, it is possible to run out of native heap memory. To complicate matters further, peak memory use for either the Java heap or the native heap depends on the precise workload being considered, as well as timing effects between concurrently operating threads. Therefore, the recommendations that follow are merely that: recommendations, not guarantees or absolute limits.

32-bit processes

A process running under any 32-bit operating system is limited to 4 GB of virtual address space, from which the operating system must reserve a significant portion for itself. For example, 32-bit Windows divides the entire address space in half, allowing an application process only 2 GB total private process space. For large Java applications like Application Servers, this process space limit imposes a ceiling on the number of threads that can be accommodated within a single Application Server.

64-bit processes

A process running under a 64-bit operating system has access to a much larger virtual address space. Compared to a 32-bit Java process performing equivalent work, a 64-bit Java process also requires a larger Java heap, typically 50% or more larger than the 32-bit Java process. For an example, see http://users.elis.ugent.be/~leeckhou/papers/SPE06.pdf.

Recommended Java heap settings

This section describes recommended Java heap sizes for Application Servers running under different operating systems. These are recommendations only and must be adjusted in light of observed conditions, especially out-of-memory errors.

For 32-bit processes, BMC recommends operating system-specific Java heap size values according to the table below. Due to memory constraints, job servers using 32-bit processes should be configured to use no more than 50 work item threads.

For 64-bit processes, BMC recommends that the Java heap size be increased as indicated in the table, if there is sufficient physical memory to support this setting.

The following table provides details about the relevant blasadmin setting:

About thread pools

An Application Server maintains several thread pools, each dedicated to a specific purpose. Selecting appropriate sizes for each of the various thread pools is one of the most important configuration choices for an Application Server.

Each thread consumes resources, especially memory, even when idle. Java threads may also consume operating system resources such as thread handles. While executing, a thread consumes even more memory. Regardless of additional performance considerations, available process size limits the number of threads available in an Application Server.

Threads within the same process share certain data structures, especially caches, which are not shared between threads in different processes. Increasing the number of threads within a single process, especially threads within a particular thread pool, has two consequences:

• Serendipity: Because there are more threads contributing to the process-wide caches, any given item request from any thread is more likely to be fulfilled from the cache, because another thread is more likely to have already placed the element in the cache. This phenomenon has a mildly positive effect on overall performance as the number of threads increases.
• Contention: Because some operations on some data structures require exclusive access, as the number of threads increases, there is a greater likelihood of one thread having to wait for another thread's exclusive access to conclude. This effect degrades per-thread performance as the number of threads increases; that is, increasing the number of threads is subject to diminishing returns, sometimes sharply so.

As the number of threads in a process grows, the negative contention effects grow more rapidly than do the positive serendipity effects. Doubling the number of threads in a pool improves performance, but does not double it. Each additional thread provides a smaller and smaller net benefit, while still consuming as much memory and other resources as any other thread.

About database connections

Connections between an Application Server and the database are managed in three connection pools, with each pool devoted to a different purpose. Each connection pool allows the configuration of a minimum and maximum number of connections, although BMC recommends leaving the minimum value at zero for all connection pools.

Configuring a database pool's maximum size to be too high wastes resources, and for large installations, may risk exceeding the total capacity of the database server. You must also ensure that the database server has sufficient capacity to service all the connections from all the connection pools for all the Application Servers in the environment. BMC recommends working with the DBA and database vendor to ensure that you have this capacity, particularly for very large installations.

Conversely, configuring a database pool with a maximum size that is too low can degrade performance, as a thread requesting a database connection from an empty connection pool blocks until a connection becomes available.

Recommendations for job servers

This section provides general recommendations for configuring Application Servers established as job servers.

Recommendations for the work item thread pool

The work item thread pool is the thread pool whose configuration has the greatest effect on overall job performance. However, selecting the best size for this thread pool involves trade-offs, and the best size will be different for different environments.

Most jobs generate one or more work items per target host, so a job targeted at thousands of servers can be expected to result in thousands of work items being queued for processing. Further, the work items themselves tend not to be CPU intensive. In light of these considerations, it is usually desirable to allocate a generous number of work item threads for a job server.

For 32-bit Application Servers, BMC recommends a setting of 50 work item threads. Configuring a larger number of work item threads risks an OutOfMemoryError under the process size limitations of 32-bit processes.

For 64-bit Application Servers, larger available process spaces make it possible to use a larger number of work item threads. The number of work item threads to configure is primarily determined by the effects of contention between work item threads.

The following table provides details about the relevant blasadmin setting:

Recommendations for the lightweight work item thread pool

Lightweight work item threads are of benefit primarily for Deploy Jobs. For installations in which Deploy Jobs represent a significant fraction of the workload, BMC suggests a value of 200 threads for lightweight work items. The default value of 0 threads for lightweight work items uses ordinary work item threads for the execution of all work items, lightweight or not.

The following table provides details about the relevant blasadmin setting:

Recommendations for the BlExec service and thread pool

An Application Server's BlExec service maintains a pool of threads for the execution of asynchronous tasks involving communication with remote targets. It is not normally necessary to change the BlExec service's configuration settings; the default values produce good results in most cases.

The following table provides details about the relevant blasadmin settings:

Other parameters

For completeness, this section describes some additional configuration parameters related to thread pool sizes for job servers. These configuration parameters do not normally require adjustment from their default values.

The following table provides details about the relevant blasadmin settings:

Recommendations for database connections

For best job server performance, BMC recommends allowing the job execution connection pool to grow up to twice the number of work item threads (MaxWorkItemThreads in AppServer module).

The following table provides details about the relevant blasadmin settings:

Process Spawner considerations

As memory size increases, so too does the cost of spawning a new process directly from the Application Server. For NSH script jobs, a job server can be configured to use a Process Spawner to spawn subprocesses, rather than spawning them directly. The Process Spawner is simply a process with a small memory footprint that can spawn new processes without the penalty of the Application Server's large memory footprint. As the configured size of the Application Server grows, the benefit of using the Process Spawner increases.

The following table provides details about the relevant blasadmin setting:

Recommendations for Configuration servers

Estimating client connections

Parameter value settings for configuration (UI) servers should be based on the number of client connections you anticipate being made to the configuration server. This value is the total of the number of client connections from UI clients (RCP) and from BLCLI clients.

The number of client connections opened by a UI client varies over time and depends on the operations that the user is engaged in at any given moment. As an initial estimate, BMC recommends a planning figure of 2.5 client connections for each concurrent GUI user.

The BLCLI client uses exactly one client connection for its execution, and is usually much more short-lived than an interactive user's GUI session.

Thus, the peak demand estimate for client connections for the configuration server is:
2.5 * (number of simultaneous GUI users) + (number of simultaneous BLCLI commands)
If multiple configuration servers with a load balancer will be established, then the total load for client connections can be divided across the number of configuration servers that will be established.

Recommendations for the client connection service and thread pool

The client connection service is responsible for managing connections from client processes in Application Servers acting as configuration (UI) servers. The client connection service maintains a pool of threads for servicing client requests.

The following table describes the parameters that most strongly affect the performance of the client connection service. In most cases, it is not necessary to change these parameters from their default values.

The following table provides details about the relevant blasadmin settings:

Recommendations for database connections

Similarly, for best configuration (UI) server performance, BMC recommends allowing the pool of database connections for client service threads to grow up to twice the number of client connection service threads (MaxWorkerThreads in AppServer module).

The following table provides details about the relevant blasadmin setting:

Recommendations for load balancer environments

When a load balancer is used to share load across multiple configuration servers, the following adjustments to the application servers' configuration parameters should be made.

The following table provides details about the relevant blasadmin settings:

Recommendations for NSH Proxy servers

BMC recommends the use of NSH Proxy servers as a best practice for security.

For best performance, configure the NSH Proxy server for the anticipated number of concurrent NSH connections it will be expected to handle.

Keep in mind that some jobs, such as file deploy jobs, may occasionally require as many as three simultaneous NSH connections per executing work item. User-written NSH script jobs may require even more simultaneous NSH connections, depending on what actions the scripts perform.

The following table provides details about the relevant blasadmin settings:

Recommendations for load balancer environments

The following application server configuration parameters are recommended for application servers using a load balancer with NSH Proxy servers.

The following table provides details about the relevant blasadmin settings:

Managing User Mappings

This section provides recommendations by BMC for the security configuration files used for RSCD agents.

For more general information about these configuration files and their role in the administration of BMC Server Automation, see Setting up configuration files.

Managing the exports File

During testing or development, it may be convenient to use the exports file to map all external users to a common local user (such as root or Administrator). However, BMC strongly recommends that, for production use, the exports file on each target not include mappings of '*'.

As mentioned above, BMC recommends using the exports file to restrict access to deployed agents. The exports file should allow connections only from job servers and NSH Proxy servers.

Managing the users File

BMC recommends establishing "ACL Push" jobs for all hosts, and running these jobs on a regular basis. An "ACL Push" job can update the contents of the users file on each managed host according to permissions established for each server.

BMC also recommends leaving the PUSH_ACL_NO_USERS_FLAG server property set to true for all servers. This will cause a 'nouser' entry to be added to each server's users file by the "ACL Push" job.

Managing the users.local File

BMC recommends adding an entry for BLAdmins:* to the users.local file for every server. Because this role cannot be deleted, it provides a kind of safety net in the event that everyone else's permissions for that server become revoked, for example, by accident.

Storage Sizing Guidance

This section provides some ballpark figures for estimating data storage requirements for a BMC Server Automation installation.

Database Sizing for BladeLogic Server Automation

Many factors influence the amount of storage consumed by the database that supports a BladeLogic Server Automation installation. The number of servers under management, the number of frequency of jobs run for each target, and the required retention period for historic job run data are factors of particular importance.

For new installations for which no previous baseline data is available, BMC suggests a planning figure of 15MB/managed server as a guideline for configuring the BMC Server Automation database server.

Oracle Database Recommendations

An Oracle database user is allowed a certain number of processes and sessions. The maximum of these numbers can be calculated in the following manner:

Connection = Process
Sessions = (1.1 * Processes) + 5

The Connections/Processes is a sum of the value of the MaxJobExecutionConnections, MaxGeneralConnections, and MaxClientConnections parameters from all application server deployments that connect to the database plus 20 connections for the ETL.

Patch Repository Sizing

BladeLogic patch content, stored in patch repositories, varies in size from application to application and vendor to vendor. The figures presented in the following table give some idea of the patch storage requirements for various Windows applications, as observed in November, 2011.