Recommendations for Application Servers of type Job

This topic provides general recommendations for sizing Application Servers defined as type Job. The term job server refers to Application Servers of type Job.

Number of Job Servers

Determining the number of Job Servers (deployments) can be done in a few different ways. As a starting point for a new environment, you should have one Job Server (with 100 Work Item Threads) per 2500 RSCD agents. After the initial usage of the environment, you should consider several factors for scaling the number of Job Servers. Depending on the job usage profile of the environment, you might find that you can have a higher or lower ratio of Job Servers per RSCD agents.

If there are certain jobs that must run in a specific amount of time, more job servers can be added to scale to that run time (for example, a Compliance Job must run across 5,000 servers in three hours).
Because Job run time can be dependent on network and database performance, those items should also be considered.
If there are certain jobs that are resource intensive on the Application Server, having more job servers will distribute the load across and decrease the run time of the job.
Using the Jconsole, determine whether the Work Item Threads are being maxed out during Job runs.
Consider also the typical number of targets in a job, the desired run time, and the frequency of job runs.

Work item thread pool settings

The work item thread pool is the thread pool whose configuration has the greatest effect on overall job performance. However, selecting the best size for this thread pool involves trade-offs, and the best size is different for different environments.

Most jobs generate one or more work items per target host, so a job targeted at thousands of servers can result in thousands of work items being queued for processing. Furthermore, the work items themselves tend not to be CPU intensive. In light of these considerations, you should usually allocate a generous number of work item threads for a job server.

For 32-bit Application Servers, BMC recommends a setting of 50 work item threads. Configuring a larger number of work item threads risks an OutOfMemory error under the process size limitations of 32-bit processes.

For 64-bit Application Servers, larger available process spaces make it possible to use a larger number of work item threads. The number of work item threads to configure is primarily determined by the effects of contention between work item threads.

The following table shows the blasadmin information required to make these sizing adjustments:

Component

Parameter

Description and recommended value

AppServer

MaxWorkItemThreads

Number of threads that can be used to execute job parts, for parallelism

For BMC Server Automation versions 8.0 and earlier, BMC recommends 50 work item threads for both 32-bit and 64-bit Application Servers.
For BMC Server Automation versions 8.1 and later, significant improvements were implemented to reduce contention between work item threads. In these releases, BMC recommends 100 work item threads for 64-bit Application Servers.

Lightweight work item thread pool settings

Lightweight work item threads are of benefit primarily for Deploy Jobs.

Component	Parameter	Description and recommended value
`AppServer`	`MaxLightweightWorkItemThreads`	Number of threads that can be used to execute lightweight job parts, for parallelism If Deploy Jobs represent a significant percentage of the workload, BMC recommends a value up to 200; otherwise use 0, the default. The default value of 0 threads for lightweight work items uses ordinary work item threads for the execution of all work items, lightweight or not.

BlExec service and thread pool settings

An Application Server's BlExec service maintains a pool of threads for the execution of asynchronous tasks involving communication with remote targets. You do not normally need to change the BlExec service's configuration settings; the default values produce good results in most cases.

Component	Parameter	Description and recommended value
`AppServer`	`EnableAsyncExecution`	Enables or disables the asynchronous execution framework for jobs that allow it (for example, Deploy Jobs). You can set the value to true or false. The default setting is true. BMC recommends leaving asynchronous execution enabled.
`BlExec`	`MaxSocketConnections`	Maximum simultaneous sockets opened by the BlExec service BMC recommends using the default value of 500 in most cases. A higher value allows more simultaneous connections; a lower value reduces the demand for file descriptors.
`BlExec`	`NumWorkerThreads`	Number of worker threads used by the BlExec service BMC recommends using the default value of 20.

Other parameters related to thread pools and connections

For completeness, this section describes some additional configuration parameters related to thread pool sizes for job servers. These configuration parameters do not normally require adjustment from their default values.

Component	Parameter	Description and recommended value
`AppServer`	`MaxApprovalThreads`	Number of approval threads This parameter governs a small pool of threads used to communicate with BMC Atrium Orchestrator.
`AppServer`	`MaxJobs`	Maximum number of jobs that the Application Server can execute simultaneously, regardless of the availability of resources to execute the jobs. Note that this limit applies to one Application Server only, and not to the whole deployed environment. The default is 20. If you run many single target jobs (such at patch-generated Deploy Jobs), increasing this value can remove a bottleneck condition where MaxJobs is hit but there are available WorkItem Threads.
`AppServer`	`MaxJobThreads`	Maximum number of threads that can be used to execute a job, for parallelism This parameter specifies the maximum size for the job execution pool. BMC recommends a value of 5. The job execution pool is distinct from the work item thread pool. Although most of the work involved in executing a job is delegated to work items, some of the work (such as creating the work items) is the responsibility of the job and is carried out by the job execution pool.
`AppServer`	`MaxClientContexts`	Number of maximum client connections to the Application Server BMC recommends estimating peak client connection demand, as described in Estimating client connections in Recommendations for Application Servers of type Configuration, and setting `MaxClientContexts` to this value. Note that in the case of Application Servers of type Job, client connections are established for BLCLI commands in NSH scripts that run on the Job server. In the absence of sufficient information about peak client connection demand, BMC recommends using the default value of 200.
`AppServer`	`MaxWorkerThreads`	Number of client connection worker threads BMC recommends using the default value of 10, or approximately 5 percent of the value of `MaxClientContexts`.
`AppServer`	`MaxHeapSize`	Maximum heap size for this Application Server. Recommended values: For 32-bit systems: Microsoft Windows: 1024 MB Linux: 1536 MB Oracle Solaris: 2048 MB For 64-bit systems: Windows: 6144 MB Linux: 6144 MB Solaris: Not applicable
`AppServer`	`MaxJMXConnections`	These connections are used for communication between Application Servers. In environments with many Application Server instances, this setting may need to be increased from the default of 20.
`AppServer`	`MinPort,MaxPort`	These settings specify a range of available ports used for communication between Application Servers. The default range provides for 50 ports (BasePort+50-BasePort+9899) which is sufficient for most environments.
`AppServer`	`MaxTimeForCancelToFinish`	Use the default value, which is 10.
`JobFactory`	`GlobalDefaultJobParallelism`	Global default value for job parallelism made available to the user This parameter has no direct effect on the operation of the Application Server. Rather, it is the default value that appears in the UI for a job's maximum parallelism option, when that option is selected. The recommended value is 30. In a large environment, you might want to set a lower number to constrain how many WorkItemThreads a single job can consume by default. Note: This setting is not applied to File Deploy Jobs and Network Shell Script Jobs.

Database connection settings

The number of available database connections can affect Job Server performance. The database connection settings should be set in conjunction with the setting for MaxWorkItemThreads in the AppServer module.

Component	Parameter	Description and recommended value
`Database`	`MaxJobExecutionConnections`	Maximum connections in the pool for the job execution thread group. In BMC Server Automation versions 8.1 and later, this connection pool is used by NSH script jobs to log output and all other job types. In versions 8.0 and earlier, the NSH script jobs do not use this connection pool. BMC recommends a value that is twice the number of work item threads (`MaxWorkItemThreads`).
`Database`	`MaxGeneralConnections`	Maximum connections in the pool for the general thread group. In BMC Server Automation versions 8.1 and later, the default setting for this parameter should be sufficient. In BMC Server Automation versions 8.0 and earlier, this connection pool is used by NSH script jobs to log output. In those versions, BMC recommends a value of `2 × MaxWorkItemThreads`, especially for environments that depend heavily on NSH script jobs.
`Database`	`MinGeneralConnections`	Controls the minimum number of database connections created on startup for the General Connection pool. Because connections are created on demand, this has no impact on performance.
`Database`	`MaxClientConnections`	Maximum connections in the pool for the general thread group. In BMC Server Automation versions 8.1 and later, the default setting for this parameter should be sufficient. In BMC Server Automation versions 8.0 and earlier, this connection pool is used by NSH script jobs to log output. In those versions, BMC recommends a value of `2 × MaxWorkItemThreads`, especially for environments that depend heavily on NSH script jobs.
`Database`	`MinClientConnections`	Controls the minimum number of database connections created on startup for the Client Connection pool. Because connections are created on demand, this has no impact on performance. The default is 0.
`Database`	`MaxIdleTime`	Maximum idle time, in seconds, for database connections. The Application Server closes database connections that are idle longer than the specified timeout. If there is a network device between the Application Server and database with a lower idle timeout, then this setting should be adjusted. The default is 600 seconds (10 minutes).
`Database`	`MinTimeToLog`	Controls logging of long running database queries. Leave at 0 to disable this functionality.
`Database`	`FetchSize`	Number of rows fetched simultaneously from the database. The default is 100 rows.
`Database`	`IdleConnectionTestPeriod`	Number of seconds that connections are idle before being tested. The default is 600 seconds (10 minutes).

Process Spawner considerations

As memory size increases, so does the cost of spawning a new process directly from the Application Server. For NSH script jobs, a job server can be configured to use a Process Spawner to spawn subprocesses, rather than spawning them directly. The Process Spawner is simply a process with a small memory footprint that can spawn new processes without the penalty of the Application Server's large memory footprint. As the configured size of the Application Server grows, the benefit of using the Process Spawner increases.

Component	Parameter	Description and recommended value
`ProcessSpawner`	`SpawnExternally`	Controls whether processes are spawned outside the Application Server For BMC Server Automation versions 8.1 and later, BMC recommends using the Process Spawner for all job servers.