Recommendations for Application Servers of type Job

This topic provides general recommendations for sizing Application Servers defined as type Job. The term job server refers to Application Servers of type Job.

Number of Job Servers

Determining the number of Job Servers (deployments) can be done in a few different ways. As a starting point for a new environment, you should have one Job Server (with 100 Work Item Threads) per 2500 RSCD agents. After the initial usage of the environment, you should consider several factors for scaling the number of Job Servers. Depending on the job usage profile of the environment, you might find that you can have a higher or lower ratio of Job Servers per RSCD agents.

  • If there are certain jobs that must run in a specific amount of time, more job servers can be added to scale to that run time (for example, a Compliance Job must run across 5,000 servers in three hours).
  • Because Job run time can be dependent on network and database performance, those items should also be considered.
  • If there are certain jobs that are resource intensive on the Application Server, having more job servers will distribute the load across and decrease the run time of the job.
  • Using the Jconsole, determine whether the Work Item Threads are being maxed out during Job runs.
  • Consider also the typical number of targets in a job, the desired run time, and the frequency of job runs.

Work item thread pool settings

The work item thread pool is the thread pool whose configuration has the greatest effect on overall job performance. However, selecting the best size for this thread pool involves trade-offs, and the best size is different for different environments.

Most jobs generate one or more work items per target host, so a job targeted at thousands of servers can result in thousands of work items being queued for processing. Furthermore, the work items themselves tend not to be CPU intensive. In light of these considerations, you should usually allocate a generous number of work item threads for a job server.

For 32-bit Application Servers, BMC recommends a setting of 50 work item threads. Configuring a larger number of work item threads risks an OutOfMemory error under the process size limitations of 32-bit processes.

For 64-bit Application Servers, larger available process spaces make it possible to use a larger number of work item threads. The number of work item threads to configure is primarily determined by the effects of contention between work item threads.

The following table shows the blasadmin information required to make these sizing adjustments:

Component

Parameter

Description and recommended value

AppServer

MaxWorkItemThreads

Number of threads that can be used to execute job parts, for parallelism

  • For BMC Server Automation versions 8.0 and earlier, BMC recommends 50 work item threads for both 32-bit and 64-bit Application Servers.
  • For BMC Server Automation versions 8.1 and later, significant improvements were implemented to reduce contention between work item threads. In these releases, BMC recommends 100 work item threads for 64-bit Application Servers.

Lightweight work item thread pool settings

Lightweight work item threads are of benefit primarily for Deploy Jobs.

Component

Parameter

Description and recommended value

AppServer

MaxLightweightWorkItemThreads

Number of threads that can be used to execute lightweight job parts, for parallelism

If Deploy Jobs represent a significant percentage of the workload, BMC recommends a value up to 200; otherwise use 0, the default. The default value of 0 threads for lightweight work items uses ordinary work item threads for the execution of all work items, lightweight or not.

BlExec service and thread pool settings

An Application Server's BlExec service maintains a pool of threads for the execution of asynchronous tasks involving communication with remote targets. You do not normally need to change the BlExec service's configuration settings; the default values produce good results in most cases.

Component

Parameter

Description and recommended value

AppServer

EnableAsyncExecution

Enables or disables the asynchronous execution framework for jobs that allow it (for example, Deploy Jobs). You can set the value to true or false. The default setting is true.

BMC recommends leaving asynchronous execution enabled.

BlExec

MaxSocketConnections

Maximum simultaneous sockets opened by the BlExec service

BMC recommends using the default value of 500 in most cases. A higher value allows more simultaneous connections; a lower value reduces the demand for file descriptors.

BlExec

NumWorkerThreads

Number of worker threads used by the BlExec service

BMC recommends using the default value of 20.

Other parameters related to thread pools and connections

For completeness, this section describes some additional configuration parameters related to thread pool sizes for job servers. These configuration parameters do not normally require adjustment from their default values.

Component

Parameter

Description and recommended value

AppServer

MaxApprovalThreads

Number of approval threads

This parameter governs a small pool of threads used to communicate with BMC Atrium Orchestrator.

AppServer

MaxJobs

Maximum number of jobs that the Application Server can execute simultaneously, regardless of the availability of resources to execute the jobs. Note that this limit applies to one Application Server only, and not to the whole deployed environment.

The default is 20. If you run many single target jobs (such at patch-generated Deploy Jobs), increasing this value can remove a bottleneck condition where MaxJobs is hit but there are available WorkItem Threads.

AppServer

MaxJobThreads

Maximum number of threads that can be used to execute a job, for parallelism

This parameter specifies the maximum size for the job execution pool. BMC recommends a value of 5. The job execution pool is distinct from the work item thread pool. Although most of the work involved in executing a job is delegated to work items, some of the work (such as creating the work items) is the responsibility of the job and is carried out by the job execution pool.

AppServer

MaxClientContexts

Number of maximum client connections to the Application Server

BMC recommends estimating peak client connection demand, as described in Estimating client connections in Recommendations for Application Servers of type Configuration, and setting MaxClientContexts to this value. Note that in the case of Application Servers of type Job, client connections are established for BLCLI commands in NSH scripts that run on the Job server.

In the absence of sufficient information about peak client connection demand, BMC recommends using the default value of 200.

AppServer

MaxWorkerThreads

Number of client connection worker threads

BMC recommends using the default value of 10, or approximately 5 percent of the value of MaxClientContexts.

AppServerMaxHeapSize

Maximum heap size for this Application Server.

Recommended values:

  • For 32-bit systems:
    • Microsoft Windows: 1024 MB
    • Linux: 1536 MB
    • Oracle Solaris: 2048 MB
  • For 64-bit systems:
    • Windows: 6144 MB
    • Linux: 6144 MB
    • Solaris: Not applicable

AppServer

MaxJMXConnections

These connections are used for communication between Application Servers. In environments with many Application Server instances, this setting may need to be increased from the default of 20.

AppServer

MinPort,MaxPort

These settings specify a range of available ports used for communication between Application Servers. The default range provides for 50 ports (BasePort+50-BasePort+9899) which is sufficient for most environments.

AppServerMaxTimeForCancelToFinishUse the default value, which is 10.

JobFactory

GlobalDefaultJobParallelism

Global default value for job parallelism made available to the user

This parameter has no direct effect on the operation of the Application Server. Rather, it is the default value that appears in the UI for a job's maximum parallelism option, when that option is selected.

The recommended value is 30. In a large environment, you might want to set a lower number to constrain how many WorkItemThreads a single job can consume by default.

Note: This setting is not applied to File Deploy Jobs and Network Shell Script Jobs.

Database connection settings

The number of available database connections can affect Job Server performance. The database connection settings should be set in conjunction with the setting for  MaxWorkItemThreads in the AppServer module.

Component

Parameter

Description and recommended value

Database

MaxJobExecutionConnections

Maximum connections in the pool for the job execution thread group. 

  • In BMC Server Automation versions 8.1 and later, this connection pool is used by NSH script jobs to log output and all other job types.
  • In versions 8.0 and earlier, the NSH script jobs do not use this connection pool.

BMC recommends a value that is twice the number of work item threads (MaxWorkItemThreads).

Database

MaxGeneralConnections

Maximum connections in the pool for the general thread group.

  • In BMC Server Automation versions 8.1 and later, the default setting for this parameter should be sufficient.
  • In BMC Server Automation versions 8.0 and earlier, this connection pool is used by NSH script jobs to log output. In those versions,  BMC recommends a value of 2 × MaxWorkItemThreads, especially for environments that depend heavily on NSH script jobs.

Database

MinGeneralConnections

Controls the minimum number of database connections created on startup for the General Connection pool. Because connections are created on demand, this has no impact on performance.

DatabaseMaxClientConnections

Maximum connections in the pool for the general thread group.

  • In BMC Server Automation versions 8.1 and later, the default setting for this parameter should be sufficient.
  • In BMC Server Automation versions 8.0 and earlier, this connection pool is used by NSH script jobs to log output. In those versions,  BMC recommends a value of 2 × MaxWorkItemThreads, especially for environments that depend heavily on NSH script jobs.

Database

MinClientConnections

Controls the minimum number of database connections created on startup for the Client Connection pool. Because connections are created on demand, this has no impact on performance. The default is 0.

Database

MaxIdleTime

Maximum idle time, in seconds, for database connections. The Application Server closes database connections that are idle longer than the specified timeout. If there is a network device between the Application Server and database with a lower idle timeout, then this setting should be adjusted. The default is 600 seconds (10 minutes).

Database

MinTimeToLog

Controls logging of long running database queries. Leave at 0 to disable this functionality.

Database

FetchSize

Number of rows fetched simultaneously from the database. The default is 100 rows.

Database

IdleConnectionTestPeriod

Number of seconds that connections are idle before being tested. The default is 600 seconds (10 minutes).

Process Spawner considerations

As memory size increases, so does the cost of spawning a new process directly from the Application Server. For NSH script jobs, a job server can be configured to use a Process Spawner to spawn subprocesses, rather than spawning them directly. The Process Spawner is simply a process with a small memory footprint that can spawn new processes without the penalty of the Application Server's large memory footprint. As the configured size of the Application Server grows, the benefit of using the Process Spawner increases.

Component

Parameter

Description and recommended value

ProcessSpawner

SpawnExternally

Controls whether processes are spawned outside the Application Server

For BMC Server Automation versions 8.1 and later, BMC recommends using the Process Spawner for all job servers.

Was this page helpful? Yes No Submitting... Thank you

Comments