Recommendations for Application Servers of type Job
This topic provides general recommendations for sizing Application Servers defined as type Job. The term job server refers to Application Servers of type Job.
- Number of Job Servers
- Work item thread pool settings
- Lightweight work item thread pool settings
- BlExec service and thread pool settings
- Other parameters related to thread pools and connections
- Database connection settings
- Process Spawner considerations
- Load balancer considerations
Number of Job Servers
Determining the number of Job Servers (deployments) can be done in a few different ways. As a starting point for a new environment, you should have one Job Server (with 100 Work Item Threads) per 2500 RSCD agents. After the initial usage of the environment, you should consider several factors for scaling the number of Job Servers. Depending on the job usage profile of the environment, you might find that you can have a higher or lower ratio of Job Servers per RSCD agents.
- If there are certain jobs that must run in a specific amount of time, more job servers can be added to scale to that run time (for example, a Compliance Job must run across 5,000 servers in three hours).
- Because Job run time can be dependent on network and database performance, those items should also be considered.
- If there are certain jobs that are resource intensive on the Application Server, having more job servers will distribute the load across and decrease the run time of the job.
- Using the Jconsole, determine whether the Work Item Threads are being maxed out during Job runs.
- Consider also the typical number of targets in a job, the desired run time, and the frequency of job runs.
Work item thread pool settings
The work item thread pool is the thread pool whose configuration has the greatest effect on overall job performance. However, selecting the best size for this thread pool involves trade-offs, and the best size is different for different environments.
Most jobs generate one or more work items per target host, so a job targeted at thousands of servers can result in thousands of work items being queued for processing. Furthermore, the work items themselves tend not to be CPU intensive. In light of these considerations, you should usually allocate a generous number of work item threads for a job server.
For 32-bit Application Servers, BMC recommends a setting of 50 work item threads. Configuring a larger number of work item threads risks an OutOfMemory error under the process size limitations of 32-bit processes.
For 64-bit Application Servers, larger available process spaces make it possible to use a larger number of work item threads. The number of work item threads to configure is primarily determined by the effects of contention between work item threads.
The following table shows the blasadmin information required to make these sizing adjustments:
Component | Parameter | Description and recommended value |
---|---|---|
AppServer | MaxWorkItemThreads | Number of threads that can be used to execute job parts, for parallelism. BMC recommends 100 work item threads for 64-bit Application Servers. |
Lightweight work item thread pool settings
Lightweight work item threads are of benefit primarily for Deploy Jobs.
Component | Parameter | Description and recommended value |
---|---|---|
AppServer | MaxLightweightWorkItemThreads | Number of threads that can be used to execute lightweight job parts, for parallelism. If Deploy Jobs represent a significant percentage of the workload, BMC recommends using the default value of 200 in most cases. If the value is set to 0, then zero threads for lightweight work items uses ordinary work item threads for the execution of all work items, lightweight or not. |
BlExec service and thread pool settings
An Application Server's BlExec service maintains a pool of threads for the execution of asynchronous tasks involving communication with remote targets. You do not normally need to change the BlExec service's configuration settings; the default values produce good results in most cases.
Component | Parameter | Description and recommended value |
---|---|---|
AppServer | EnableAsyncExecution | Enables or disables the asynchronous execution framework for jobs that allow it (for example, Deploy Jobs). This is always set to true. |
BlExec | MaxSocketConnections | Maximum simultaneous sockets opened by the BlExec service |
BlExec | NumWorkerThreads | Number of worker threads used by the BlExec service |
Other parameters related to thread pools and connections
For completeness, this section describes some additional configuration parameters related to thread pool sizes for job servers. These configuration parameters do not normally require adjustment from their default values.
Component | Parameter | Description and recommended value |
---|---|---|
AppServer | MaxApprovalThreads | Number of approval threads |
AppServer | {{id name="RecommendationsforApplicationServersoftypeJob-maxjobs"/}} MaxJobs | Maximum number of jobs that the Application Server can execute simultaneously, regardless of the availability of resources to execute the jobs. Note that this limit applies to one Application Server only, and not to the whole deployed environment. The default is 20. If you run many single target jobs (such at patch-generated Deploy Jobs), increasing this value can remove a bottleneck condition where MaxJobs is hit but there are available WorkItem Threads. Note: If you are running batch jobs, the actual number of jobs running might exceed the MaxJobs setting. Number of jobs running is one of the statistics included in Application Server information in the Infrastructure Management window or in the detailed report that you can export from that window, as well as in the Application Server status report when you generate data for Support. The reason for this phenomenon is that only the batch job itself is counted towards the MaxJobs limit, while all jobs — the batch job and all its member jobs — are included in the count of number of jobs running. |
AppServer | MaxJobThreads | Maximum number of threads that can be used to execute a job, for parallelism |
AppServer | MaxClientContexts | Number of maximum client connections to the Application Server BMC recommends estimating peak client connection demand, as described in Estimating client connections in Recommendations-for-Application-Servers-of-type-Configuration, and setting MaxClientContexts to this value. Note that in the case of Application Servers of type Job, client connections are established for BLCLI commands in NSH scripts that run on the Job server. In the absence of sufficient information about peak client connection demand, BMC recommends using the default value of 200. |
AppServer | MaxWorkerThreads | Number of client connection worker threads. While the Application Server is defined as a JOB server type, any NSH Script Jobs that use BLCLI commands that the server runs will need a client connection and they will connect to whatever is defined in the blasadmin Auth | AppServiceURL setting. By default this is blank (and should be left as such) and the server will connect to itself. |
AppServer | MaxHeapSize | Maximum heap size for this Application Server. |
AppServer | MaxJMXConnections | These connections are used for communication between Application Servers. In environments with many Application Server instances, this setting may need to be increased from the default of 20. |
AppServer | MinPort,MaxPort | These settings specify a range of available ports used for communication between Application Servers. The default range provides for 50 ports (BasePort+50-BasePort+9899) which is sufficient for most environments. |
AppServer | MaxTimeForCancelToFinish | Use the default value, which is 10. |
JobFactory | GlobalDefaultJobParallelism | Global default value for job parallelism made available to the user The recommended value is 30. In a large environment, you might want to set a lower number to constrain how many WorkItemThreads a single job can consume by default. Note: This setting is not applied to File Deploy Jobs and Network Shell Script Jobs. |
Database connection settings
The number of available database connections can affect Job Server performance. The database connection settings should be set in conjunction with the setting for MaxWorkItemThreads in the AppServer module.
Component | Parameter | Description and recommended value |
---|---|---|
Database | MaxJobExecutionConnections | Maximum connections in the pool for the job execution thread group. This connection pool is used by NSH script jobs to log output and all other job types. BMC recommends a value that is twice the number of work item threads (MaxWorkItemThreads). |
Database | MaxGeneralConnections | Maximum connections in the pool for the general thread group. The default setting (200) for this parameter should be sufficient. |
Database | MinGeneralConnections | Controls the minimum number of database connections created on startup for the General Connection pool. Because connections are created on demand, this has no impact on performance. |
Database | MaxClientConnections | Maximum connections in the pool for client connections. Use the following rules of thumb:
|
Database | MinClientConnections | Controls the minimum number of database connections created on startup for the Client Connection pool. Because connections are created on demand, this has no impact on performance. The default is 0. |
Database | MaxIdleTime | Maximum idle time, in seconds, for database connections. The Application Server closes database connections that are idle longer than the specified timeout. If there is a network device between the Application Server and database with a lower idle timeout, then this setting should be adjusted. The default is 600 seconds (10 minutes). |
Database | MinTimeToLog | Controls logging of long running database queries. Leave at 0 to disable this functionality. |
Database | FetchSize | Number of rows fetched simultaneously from the database. The default is 100 rows. |
Database | IdleConnectionTestPeriod | Number of seconds that connections are idle before being tested. The default is 600 seconds (10 minutes). |
Database | BootStrapConnectionPoolMaxSize | Maximum number of database connections created for the General Connection pool before starting the Application Server. BMC recommends to keep the same value as ConnectionPoolMaxSize. |
Process Spawner considerations
As memory size increases, so does the cost of spawning a new process directly from the Application Server. For NSH script jobs, a job server can be configured to use a Process Spawner to spawn subprocesses, rather than spawning them directly. The Process Spawner is simply a process with a small memory footprint that can spawn new processes without the penalty of the Application Server's large memory footprint. As the configured size of the Application Server grows, the benefit of using the Process Spawner increases.
Load balancer considerations
When a load balancer is used to share the load across multiple Application Servers of type Configuration or All, the following adjustments to the Application Server's parameter settings are recommended. For more information, see Considerations for load balancing.
Component | Parameter | Description and recommended value |
---|---|---|
AuthServer | AppServiceURLs | Application service URLs distributed in the session credentials issued by the Authentication Service For a Job instance, the setting should remain blank. This will ensure that any NSH Script Jobs that contain BLCLI calls will direct requests to the instance that is running the Work Item Thread associated with the Job and nowhere else. If the instance is an ALL instance, then an alias for the load balancer should be added to the local hosts file on the Application Server, so that the load balancer URL resolves to the local system. This prevents network communication issues that might result from a node behind a load balancer trying to talk to itself through the same VIP that it sits behind. The load balancer vendor might have specific configuration settings to handle this situation, as well. |
AuthServer | ProxyServiceURLs | NSH Proxy service URLs distributed in the session credentials issued by the Authentication Service. For a Job instance, the setting should remain blank if the instance is a combined JOB+NSH_PROXY. This will ensure that any Jobs that use an NSH Proxy will direct requests to the instance that is running the Work Item Thread associated with the Job and nowhere else. If the instance is an ALL instance, then an alias for the load balancer should be added to the local hosts file on the Application Server, so that the load balancer URL resolves to the local system. This prevents network communication issues might result from a node behind a load balancer trying to talk to itself through the same VIP that it sits behind. The load balancer vendor might have specific configuration settings to handle this situation, as well. If the NSH_PROXY instance is a standalone instance, or you want to direct the NSH Proxy communication to another instance that runs the proxy, the network path must be considered. For example, would the communication from the Job instance to the NSH Proxy instance result in the same situation as the above where a node behind a load balancer tries to communicate with itself through the VIP? If so, the hosts alias should be used here, as well. |