Recommendations for Application Servers of type Job


This topic provides general recommendations for sizing Application Servers defined as type Job. The term job server refers to Application Servers of type Job.

Number of Job Servers

Determining the number of Job Servers (deployments) can be done in a few different ways. As a starting point for a new environment, you should have one Job Server (with 100 Work Item Threads) per 2500 RSCD agents. After the initial usage of the environment, you should consider several factors for scaling the number of Job Servers. Depending on the job usage profile of the environment, you might find that you can have a higher or lower ratio of Job Servers per RSCD agents.

  • If there are certain jobs that must run in a specific amount of time, more job servers can be added to scale to that run time (for example, a Compliance Job must run across 5,000 servers in three hours).
  • Because Job run time can be dependent on network and database performance, those items should also be considered.
  • If there are certain jobs that are resource intensive on the Application Server, having more job servers will distribute the load across and decrease the run time of the job.
  • Using the Jconsole, determine whether the Work Item Threads are being maxed out during Job runs.
  • Consider also the typical number of targets in a job, the desired run time, and the frequency of job runs.

Work item thread pool settings

The work item thread pool is the thread pool whose configuration has the greatest effect on overall job performance. However, selecting the best size for this thread pool involves trade-offs, and the best size is different for different environments.

Most jobs generate one or more work items per target host, so a job targeted at thousands of servers can result in thousands of work items being queued for processing. Furthermore, the work items themselves tend not to be CPU intensive. In light of these considerations, you should usually allocate a generous number of work item threads for a job server.

For 32-bit Application Servers, BMC recommends a setting of 50 work item threads. Configuring a larger number of work item threads risks an OutOfMemory error under the process size limitations of 32-bit processes.

For 64-bit Application Servers, larger available process spaces make it possible to use a larger number of work item threads. The number of work item threads to configure is primarily determined by the effects of contention between work item threads.

The following table shows the blasadmin information required to make these sizing adjustments:

Component

Parameter

Description and recommended value

AppServer
MaxWorkItemThreads

Number of threads that can be used to execute job parts, for parallelism. BMC recommends 100 work item threads for 64-bit Application Servers.

Lightweight work item thread pool settings

Lightweight work item threads are of benefit primarily for Deploy Jobs.

Component

Parameter

Description and recommended value

AppServer
MaxLightweightWorkItemThreads

Number of threads that can be used to execute lightweight job parts, for parallelism.

If Deploy Jobs represent a significant percentage of the workload, BMC recommends using the default value of 200 in most cases.

If the value is set to 0, then zero threads for lightweight work items uses ordinary work item threads for the execution of all work items, lightweight or not.

BlExec service and thread pool settings

An Application Server's BlExec service maintains a pool of threads for the execution of asynchronous tasks involving communication with remote targets. You do not normally need to change the BlExec service's configuration settings; the default values produce good results in most cases.

Component

Parameter

Description and recommended value

AppServer
EnableAsyncExecution

Enables or disables the asynchronous execution framework for jobs that allow it (for example, Deploy Jobs). This is always set to true.
BMC recommends leaving asynchronous execution enabled.

BlExec
MaxSocketConnections

Maximum simultaneous sockets opened by the BlExec service

BMC recommends using the default value of 500 in most cases. A higher value allows more simultaneous connections; a lower value reduces the demand for file descriptors.

BlExec
NumWorkerThreads

Number of worker threads used by the BlExec service

BMC recommends using the default value of 20.

Other parameters related to thread pools and connections

For completeness, this section describes some additional configuration parameters related to thread pool sizes for job servers. These configuration parameters do not normally require adjustment from their default values.

Component

Parameter

Description and recommended value

AppServer
MaxApprovalThreads

Number of approval threads

This parameter governs a small pool of threads used to communicate with TrueSight Orchestration.

AppServer
{{id name="RecommendationsforApplicationServersoftypeJob-maxjobs"/}}

MaxJobs

Maximum number of jobs that the Application Server can execute simultaneously, regardless of the availability of resources to execute the jobs. Note that this limit applies to one Application Server only, and not to the whole deployed environment.

The default is 20. If you run many single target jobs (such at patch-generated Deploy Jobs), increasing this value can remove a bottleneck condition where MaxJobs is hit but there are available WorkItem Threads.

Note: If you are running batch jobs, the actual number of jobs running might exceed the MaxJobs setting. Number of jobs running is one of the statistics included in Application Server information in the Infrastructure Management window or in the detailed report that you can export from that window, as well as in the Application Server status report when you generate data for Support. The reason for this phenomenon is that only the batch job itself is counted towards the MaxJobs limit, while all jobs — the batch job and all its member jobs — are included in the count of number of jobs running.

Click here for examples...

Example 1:

  • MaxJobs is set to 20 jobs.
  • Number of jobs running was 15, but then a batch job with 10 member jobs started executing.
  • The new number of jobs running is now 26:
    15 jobs from before + 1 parent batch job + 10 child (member) jobs

Example 2:

  • MaxJobs is set to 20 jobs.
  • Number of jobs running was 0, and then a batch job started executing.
  • In this batch job, the Execute by server option was selected, and the job is targeted against 28 servers. This option is identical to running multiple batch jobs simultaneously, with each batch job configured to have only one target and to execute individual jobs sequentially.
  • The new number of jobs running is now 57:
    1 parent batch job + 28 child batch jobs (1 per target) + 28 child (member) jobs, one under each child batch job


AppServer
MaxJobThreads

Maximum number of threads that can be used to execute a job, for parallelism

This parameter specifies the maximum size for the job execution pool. BMC recommends a value of 5. The job execution pool is distinct from the work item thread pool. Although most of the work involved in executing a job is delegated to work items, some of the work (such as creating the work items) is the responsibility of the job and is carried out by the job execution pool.

AppServer
MaxClientContexts

Number of maximum client connections to the Application Server

BMC recommends estimating peak client connection demand, as described in Estimating client connections in Recommendations-for-Application-Servers-of-type-Configuration, and setting MaxClientContexts to this value. Note that in the case of Application Servers of type Job, client connections are established for BLCLI commands in NSH scripts that run on the Job server.

In the absence of sufficient information about peak client connection demand, BMC recommends using the default value of 200.

AppServer
MaxWorkerThreads

Number of client connection worker threads. While the Application Server is defined as a JOB server type, any NSH Script Jobs that use BLCLI commands that the server runs will need a client connection and they will connect to whatever is defined in the blasadmin Auth | AppServiceURL setting. By default this is blank (and should be left as such) and the server will connect to itself.

BMC recommends using the default value of 10, or approximately 5 percent of the value of MaxClientContexts.

AppServer
MaxHeapSize

Maximum heap size for this Application Server.

Click here to review recommendations for the Application Server maximum Java heap size (MaxHeapSize).

This section describes recommended Java heap sizes for Application Servers running under different operating systems. These are recommendations only and must be adjusted in light of observed conditions, especially out-of-memory errors.

  • For 32-bit processes, BMC recommends operating system-specific Java heap size values according to the table below. Due to memory constraints, job servers using 32-bit processes should be configured to use no more than 50 work item threads.
  • For 64-bit processes, BMC recommends that the Java heap size be increased as indicated in the table, if there is sufficient physical memory to support this setting.

The following table provides details about the relevant blasadmin setting:

Module

Setting

Description and Recommendation

AppServer

MaxHeapSize

Specifies the maximum heap size for this Application Server.
See operating system-specific recommendations for this value summarized in the table below.



Windows:

  • 32-bit: 1024 MB
  • 64-bit: 6-8 GB



Linux:

  • 32-bit: 1536 MB
  • 64-bit: 6-8 GB



Solaris:

  • 32-bit: 2048
  • 64-bit: Not applicable
AppServer
MaxJMXConnections

These connections are used for communication between Application Servers. In environments with many Application Server instances, this setting may need to be increased from the default of 20.

AppServer
MinPort,MaxPort

These settings specify a range of available ports used for communication between Application Servers. The default range provides for 50 ports (BasePort+50-BasePort+9899) which is sufficient for most environments.

AppServer
MaxTimeForCancelToFinish

Use the default value, which is 10.

JobFactory
GlobalDefaultJobParallelism

Global default value for job parallelism made available to the user

This parameter has no direct effect on the operation of the Application Server. Rather, it is the default value that appears in the UI for a job's maximum parallelism option, when that option is selected.

The recommended value is 30. In a large environment, you might want to set a lower number to constrain how many WorkItemThreads a single job can consume by default.

Note: This setting is not applied to File Deploy Jobs and Network Shell Script Jobs.

Database connection settings

The number of available database connections can affect Job Server performance. The database connection settings should be set in conjunction with the setting for  MaxWorkItemThreads in the AppServer module.

Component

Parameter

Description and recommended value

Database
MaxJobExecutionConnections

Maximum connections in the pool for the job execution thread group. This connection pool is used by NSH script jobs to log output and all other job types.

BMC recommends a value that is twice the number of work item threads (MaxWorkItemThreads).

Database
MaxGeneralConnections

Maximum connections in the pool for the general thread group. The default setting (200) for this parameter should be sufficient.

Note

In the case of an upgrade from a pre-8.7 version of TrueSight Server Automation, the default is 100 (that is, the pre-upgrade default remains and is not updated).

After the upgrade, verify that the resources in your environment can support a higher number of connections and that the database has sufficient connections, and only then change this setting to the new default of 200 connections.

Database
MinGeneralConnections

Controls the minimum number of database connections created on startup for the General Connection pool. Because connections are created on demand, this has no impact on performance.

Database
MaxClientConnections

Maximum connections in the pool for client connections. Use the following rules of thumb:

  • For an Application Server configured to act exclusively as a Job server, this value should be 2 times the value of the MaxWorkerThreads setting.
  • For an Application Server configured to act as both a Job server and an NSH proxy server, this value should be 2 times the sum of the values for MaxWorkerThreads + MaxNshProxyThreads.
Database
MinClientConnections

Controls the minimum number of database connections created on startup for the Client Connection pool. Because connections are created on demand, this has no impact on performance. The default is 0.

Database
MaxIdleTime

Maximum idle time, in seconds, for database connections. The Application Server closes database connections that are idle longer than the specified timeout. If there is a network device between the Application Server and database with a lower idle timeout, then this setting should be adjusted. The default is 600 seconds (10 minutes).

Database
MinTimeToLog

Controls logging of long running database queries. Leave at 0 to disable this functionality.

Database
FetchSize

Number of rows fetched simultaneously from the database. The default is 100 rows.

Database
IdleConnectionTestPeriod

Number of seconds that connections are idle before being tested. The default is 600 seconds (10 minutes).

Database
BootStrapConnectionPoolMaxSize

Maximum number of database connections created for the General Connection pool before starting the Application Server. BMC recommends to keep the same value as ConnectionPoolMaxSize.

Process Spawner considerations

As memory size increases, so does the cost of spawning a new process directly from the Application Server. For NSH script jobs, a job server can be configured to use a Process Spawner to spawn subprocesses, rather than spawning them directly. The Process Spawner is simply a process with a small memory footprint that can spawn new processes without the penalty of the Application Server's large memory footprint. As the configured size of the Application Server grows, the benefit of using the Process Spawner increases.

Load balancer considerations

When a load balancer is used to share the load across multiple Application Servers of type Configuration or All, the following adjustments to the Application Server's parameter settings are recommended. For more information, see Considerations for load balancing.

Component

Parameter

Description and recommended value

AuthServer
AppServiceURLs

Application service URLs distributed in the session credentials issued by the Authentication Service

For a Job instance, the setting should remain blank. This will ensure that any NSH Script Jobs that contain BLCLI calls will direct requests to the instance that is running the Work Item Thread associated with the Job and nowhere else. 

If the instance is an ALL instance, then an alias for the load balancer should be added to the local hosts file on the Application Server, so that the load balancer URL resolves to the local system. This prevents network communication issues that might result from a node behind a load balancer trying to talk to itself through the same VIP that it sits behind. The load balancer vendor might have specific configuration settings to handle this situation, as well.

AuthServer
ProxyServiceURLs

NSH Proxy service URLs distributed in the session credentials issued by the Authentication Service.

For a Job instance, the setting should remain blank if the instance is a combined JOB+NSH_PROXY. This will ensure that any Jobs that use an NSH Proxy will direct requests to the instance that is running the Work Item Thread associated with the Job and nowhere else. 

If the instance is an ALL instance, then an alias for the load balancer should be added to the local hosts file on the Application Server, so that the load balancer URL resolves to the local system. This prevents network communication issues might result from a node behind a load balancer trying to talk to itself through the same VIP that it sits behind. The load balancer vendor might have specific configuration settings to handle this situation, as well.

If the NSH_PROXY instance is a standalone instance, or you want to direct the NSH Proxy communication to another instance that runs the proxy, the network path must be considered. For example, would the communication from the Job instance to the NSH Proxy instance result in the same situation as the above where a node behind a load balancer tries to communicate with itself through the VIP? If so, the hosts alias should be used here, as well.

 

Tip: For faster searching, add an asterisk to the end of your partial query. Example: cert*