Managing the Application Server

The Application Server is the core of the middle tier in a BMC Server Automation installation. Not only does the Application Server control communication between clients and servers, it also regulates activity between the client and the database, file, and mail servers. The Application Server provides many adjustable parameters that enable you to scale a BMC Server Automation system to virtually any size. The following topics describe some of those parameters:

• To scale the Application Server
• To set up job distribution between multiple Application Servers
• Considerations for troubleshooting jobs in a MAS environment
• Job distribution and job priority in a multiple Application Server environment
• To synchronize keystore files of multiple Application Servers
• To specify a maximum time for canceling a job part
• To set limits for client connections
• To set time-out behavior for work item threads
• To enable automatic restart of provisioning jobs after Application Server restart
• To set the outcome of Update Server Properties Jobs when target agents are unreachable, not licensed, or not accessible by user
• To filter out informational messages from Update Server Properties Job logs
• To set a maximum number of Compliance Results displayed
• To set behavior for past due jobs
• To set the number of database connections
• To set up HTTP proxy server support
• To bind the Application Server to an IP address
• To enable and disable Network Shell proxy inspection
• To control Network Shell proxy sessions
• To ensure version compatibility between Application Server and client
• To set a maximum cache size for file system objects
• To set the maximum number of items per page

To scale the Application Server

The Application Server provides several options that you can adjust to accommodate increased activity.
An Application Server should be configured so that even when all of its work item threads are busy, the Application Server still has additional resource capacity.

1. Start the Application Server Administration console, as described in Starting-the-Application-Server-Administration-console.
2. To specify the maximum number of worker threads, enter the following command:
   set appserver maxworkerthreads #
   where # is the maximum number of threads that can process requests from clients. For example, you might set this value to 10, which means that only 10 client connections can be serviced at a time even though many more users might actually be connected to the Application Server. Worker threads should not be confused with work item threads, which process BMC Server Automation Console jobs (see step 5).
3. To specify the maximum number of client connections that the Application Server can manage, enter the following command:
   set appserver MaxClientContexts #
   where # is the maximum number of connections to clients.
4. To specify the maximum number of jobs, enter the following command:
   set appserver MaxJobs #
   where # is the maximum number of jobs. By controlling the number of jobs that are processed simultaneously, you can avoid overtaxing Application Server resources.
5. To specify a maximum size for the pool of threads that can be used to process BMC Server Automation Console jobs, enter the following command:
   set appserver MaxWorkItemThreads #
   where # is a number of work item threads.
   All BMC Server Automation jobs let you specify how many targets to process in parallel. You can set a value from 1 to 10 or allow an unlimited number of targets to be processed in parallel. The MaxWorkItemThreads and MaxLightweightWorkItemThreads (see step 6) also can control how many targets can be processed in parallel. If your system uses one Application Server, the maximum number of targets that can be processed is based on the Application Server available work item threads. If your system uses multiple Application Servers, the maximum number of targets that can be processed in parallel is based on the sum of all available work item threads on all Application Servers. When processing Deploy Jobs, work item threads often sit idle while target servers process deployment tasks. To avoid this kind of inefficiency, the Application Server can use a pool of lightweight work item threads to process phases of a Deploy Job that access target servers.
   For more information about the role of work item threads, see Work item and job execution threads.
6. To specify a maximum size for the pool of lightweight work item threads that can be used for Deploy Jobs, enter the following command:
   set appserver MaxLightweightWorkItemThreads #
   where # is a number of lightweight work item threads. By default this value is set to 0.
   An Application Server can optionally provide a secondary pool of lightweight work item threads. These threads are only used during the Simulate and Commit phases of a Deploy Job. Lightweight work item threads primarily perform tasks on target servers and consequently consume almost no memory on an Application Server. Using lightweight work item threads helps you run more Deploy Jobs in parallel more efficiently. Other than being limited to particular types of tasks, lightweight work item threads behave exactly like work item threads.
7. Restart the Application Server.

To set up job distribution between multiple Application Servers

When Application Servers are configured to access the same database, they automatically attempt to cooperate by balancing their job processing workloads. They accomplish this balance by distributing the processing of entire jobs or work items for large individual jobs to other Application Servers.
For Application Servers to cooperate, they must know which Application Servers are in service. Each Application Server periodically updates its time stamp, which functions as its heartbeat. Application Servers that are cooperating monitor each other's heartbeat to determine which Application Servers are in service.

1. Start the Application Server Administration console, as described in Starting-the-Application-Server-Administration-console.
2. To specify a time span that indicates a remote Application Server has timed out, enter the following command:
   set appserver RemoteServerTimeout #
   where # is number of seconds between heartbeats before a remote Application Server is considered out of service. For example, if RemoteServerTimeout is set to 5, and 10 seconds elapse between heartbeats, the Application Server is considered out of service.
3. To specify an interval between heartbeats for an Application Server, enter the following command:
   set appserver ServerMonitorInterval #
   where # is the frequency with which an Application Server updates its own time stamp (that is, its heartbeat). When an Application Server updates its heartbeat, it also checks for the heartbeat of any remote Application Servers.
4. To specify a time-out for connecting to a remote Application Server, enter the following command:
   set appserver SocketConnectTimeout #
   where # is the maximum number of seconds for obtaining an initial socket connection to a remote Application Server. After that maximum is exceeded, the connection times out.
5. To specify a time-out for responses from a remote Application Server, enter the following command:
   set appserver SocketTimeout #
   where # is the maximum number of seconds to wait for a response from an Application Server after the initial connection has already been established. After the maximum is exceeded, the connection times out.
6. To specify that a socket connection use SSL, enter the following command:
   set appserver UseSSLSockets true
   where true indicates that connections to this Application Server must be encrypted using SSL.
7. To specify that remote Application Servers contacting the Application Server must authenticate, enter the following command:
   set appserver RequireClientAuthentication true
   where true instructs the Application Server to require authentication from remote Application Servers. Generally, connections encrypted with SSL also require client authentication.
8. To specify a port used for communication between Application Servers, enter the following command:
   set appserver RegistryPort #
   where # is a port number. By default the RegistryPort is set to 9836.
9. Restart the Application Server.

Considerations for troubleshooting jobs in a MAS environment

Each Application Server has a log file that contains information about what is being executed on that Application Server. When Application Servers are configured to access the same database, they automatically attempt to cooperate by balancing their job processing workloads, which means that the log information is also distributed.

For example, you might have two physical Application Servers (appserver1 and appserver2), with one job server running on each, and the Distribution Manager is dynamically allocating resources and running jobs on both Application Servers as needed. In this example, the logging information for the job is actually distributed between the log files on both appserver1 and appserver2. Therefore you would need to review the log files on both Application Servers.
By default, the Application Server log file, appserver.log, is located in /opt/bmc/bladeLogic/NSH/br (on UNIX) or installationDirectory\BladeLogic\NSH\br (on Windows). Additional log files are generated for each individual Application Server deployment (deploymentProfileName.log) in the same location.

Job distribution and job priority in a multiple Application Server environment

You can use the PRIORITY* property to mark a job, or a class of jobs, with a relatively higher priority to ensure they are executed first in case of resource contention. You can assign one of any of the following priorities: Critical, High, Normal, Low, and Lowest. By default, all job types have a priority of Normal. Note that these priority levels are meaningful only in relation to each other.
For a list of the permissions and authorizations required to modify a job priority, see Setting-up-authorizations-for-changing-job-priority and Setting-job-priority.
If you have implemented a multiple Application Server environment, note the following considerations about job priority:

• While queuing work items across all jobs, the Distribution Manager queues work items in respect to priority. For example, if two concurrent jobs are competing for resources, the individual work items of the higher priority job are queued to be processed before the work items of the lower priority job.
• There is no guarantee about the order of completion of each job (which is dependent on various extraneous factors including the actions performed in each job, its target vector, the responsiveness of the target space, and so on).
• The parallelism configuration of a job can significantly impact the appearance of the effectiveness of the job's priority level, as it controls the maximum number of simultaneous work items that can be allocated for a given job. For example, consider the case of a job with a priority of Critical, but with a low maximum parallelism level. Such a job would appear to relinquish resources to a lower priority job with a high parallelism level, after the initial work item assignment quota for that Critical priority job is reached.

To synchronize keystore files of multiple Application Servers

Multiple Application Servers on different hosts can be set up to cooperate on processing jobs. (For information, see Setting up job distribution between multiple Application Servers.) For this cooperation to take place, all Application Servers must have the same bladelogic.keystore file.
To synchronize keystore files of cooperating Application Servers, perform the following steps on each cooperating Application Server:

1. Stop the cooperating Application Server.
2. Copy the bladelogic.keystore file from the <installationDirectory>/br/deployments directory of the central Application Server to each deployment directory of the cooperating Application Server.
3. Using the blasadmin utility, ensure that the passwords match for bladelogic.keystore files of all deployments of the cooperating Application Server. Updating the password for one deployment will update it for all deployments. If the new bladelogic.keystore file that you copied into a deployment has a different password from that of the old bladelogic.keystore file, change the keystore password for that deployment. (If keystore passwords match, you can skip this step.)
4. Restart the cooperating Application Servers.

To change the password needed for the bladelogic.keystore file for all Application Server deployments:

1. On the cooperating Application Server, start the Application Server Administration console to configure all Application Servers on the host:
   blasadmin -a
   This command starts the blasadmin utility and you can enter set and show commands. All commands that you enter during the session affect all additional Application Servers configured on the same host.
2. At the blasadmin prompt, enter:
   set appserver CertPasswd <password>
3. Restart the cooperating Application Server services.

To specify a maximum time for canceling a job part

You can specify a maximum period of time that can elapse for a job part to be canceled. If cancellation of a job part exceeds this maximum, the job part is classified as stuck and the job part is aborted. This prevents situations where cancellation of a job is not performing as expected and the act of canceling the job can potentially hang the job.

1. Start the Application Server Administration console, as described in Starting-the-Application-Server-Administration-console.
2. To set a maximum period for job cancellation, enter the following command:
   set appserver MaxTimeForCancelToFinish #
   where # is the maximum amount of time in minutes that should elapse for job cancellation.
3. Restart the Application Server.

To set limits for client connections

The Application Server lets you specify certain limits for connections to the Application Server, such as a prune time for idle connections or the maximum amount of time a client can perform read operations from the Application Server. If the client exceeds these maximums, the connection is closed.

1. Start the Application Server Administration console, as described in Starting-the-Application-Server-Administration-console.
2. To specify an idle prune time, enter the following command:
   set appserver IdleConnectionPruneTime #
   where # is a value in minutes. When there is no traffic over the connection between a client and the Application Server for this period of time, the connection is considered expired. When a new incoming connection is made, idle connections with non-zero IdleConnectionPruneTime values are checked, and the connections that have expired are pruned.
   By default this value is set to 0, which means the connection never expired.
3. To specify a time-out for client socket read operations from the Application Server, enter the following command:
   set appserver SocketTimeout #
   where # is the maximum number of seconds for client socket reads before the socket times out.
4. Restart the Application Server.

To set time-out behavior for work item threads

The Application Server lets you specify time-out behavior for work item threads. By default, if you assign job part time-outs, a work item thread is canceled when it exceeds the time period you have defined in the JOB_PART_TIMEOUT property. In addition, all other work item threads acting on the same server are also canceled. This prevents situations where multiple work item threads time out serially on the same unresponsive server. If necessary, you can use this procedure to override the default behavior so that only one work item thread times out automatically.

1. Start the Application Server Administration console, as described in Starting-the-Application-Server-Administration-console.
2. To set time-out behavior, enter the following command:
   set appserver PropagateWorkItemTimeout true|false
   Setting this value to true means all work item threads acting on the same server are canceled when one work item thread times out. Setting it to false means only the one work item thread is canceled.
3. Restart the Application Server.

To enable automatic restart of provisioning jobs after Application Server restart

A restart of an Application Server cancels provisioning jobs that have been submitted but are waiting idle (for example, a job waiting for the PXE client to boot). To ensure that these jobs are automatically restarted when the Application Server restarts, use this procedure.

1. Start the Application Server Administration console, as described in Starting-the-Application-Server-Administration-console.
2. Enable automatic restart of Provision Jobs by entering the following command:
   set appserver restartIdleProvisionJobs true
3. Restart the Application Server.

To set the outcome of Update Server Properties Jobs when target agents are unreachable, not licensed, or not accessible by user

By default, an Update Server Properties Job always ends successfully, even if the RSCD agents on the remote target servers are unreachable, not licensed, or not accessible by the user because the AGENT_STATUS property for the target servers is updated in any case. You can, however, set the Update Server Properties Job to end in failed status whenever agents do not respond or when the user does not have access to agent.

If a server has the IS_ONLINE property set to false, the Update Server Properties Job result for that server shows the status to be Offline.

1. Start the Application Server Administration console, as described in Starting-the-Application-Server-Administration-console.
2. Set the outcome of Update Server Properties Jobs by entering the following command:
   set JobFactory UspJobSucceedsWhenAgentDown true|false
   The default value is true. Setting the value to false means that the Update Server Properties Job is reported (in the log and in the display of job results) as having ended in failed status if the agent on the remote target is unreachable, not licensed, or not accessible by the user.

For more information, see Creating-or-modifying-Update-Server-Properties-Jobs.

To filter out informational messages from Update Server Properties Job logs

The job log for an Update Server Properties Job can include many INFO-type messages, in addition to error and warning messages. As of product version 8.6.01, you can choose whether to include such informational messages or filter them out

1. Start the Application Server Administration console, as described in Starting-the-Application-Server-Administration-console.
2. Set the filtering of Update Server Properties Job logs by entering the following command:
   set JobFactory LogOnlyErrorsOrWarningsOnUSPJob true|false
   The default value is true. A value of true means that Update Server Properties Job logs do not include most INFO-type messages; the logs include error and warning messages, as well as messages about job start and job completion.
   If you want to receive all INFO-type messages, set the value to false.

For more information, see Creating-or-modifying-Update-Server-Properties-Jobs.

To set a maximum number of Compliance Results displayed

The Application Server lets you specify the maximum number of compliance results that are displayed for any failed condition in a compliance rule. A Compliance Job examines a component and compares its parts to conditions defined in compliance rules for a component template. Parts that do not comply are shown in Compliance Job results. If you are running a Compliance Job that examines many server objects that fail a compliance condition, you might tax your system resources, particularly disk space. If results for a Compliance Job exceed the limits you set in this procedure, the job run is marked with a warning and the job log includes a message saying that job results have been truncated.

1. Start the Application Server Administration console, as described in Starting-the-Application-Server-Administration-console.
2. To set a maximum for compliance results displayed, enter the following command:
   set appserver ComplianceResultMaxNumberOfAssets #
   where # is a the maximum number of server objects displayed per failed condition in a compliance rule.
3. Restart the Application Server.

To set behavior for past due jobs

The Application Server lets you specify a period of time that a newly created job can remain in a queue while the Application Server is down or too busy to process the job. The period of time is measured from the scheduled occurrence of the job to the time the Application Server starts. Setting a value for this option specifies that if you restart the Application Server and the specified period has:

• Not elapsed, the scheduled occurrence of a one-time-only job is not executed but the scheduled occurrence of a recurring job is executed

• Elapsed, the scheduled occurrence of a job is not executed

  Note

  This procedure only defines behavior for new jobs. No matter how you define this value, all existing jobs remain in the job queue for the default amount of time (60 minutes).

1. Start the Application Server Administration console, as described in Starting-the-Application-Server-Administration-console.
2. To set past due job execution behavior, enter the following command:
   set scheduler MaxJobTimeInSchedulerQ #
   where # is a value in minutes. By default this value is set to 60. Setting this value to 0 means that all past due jobs are executed when the Application Server starts.
3. Restart the Application Server.

To set the number of database connections

Use this procedure to set maximums and minimums for database connections. You can set the maximum and minimum number of database connections that jobs use. You can also set the maximum and minimum number of database connections used for general, non-job-related purposes, such as client connections to the database. By providing separate settings for job-related and non-job-related activity, you can help to prevent situations where client connections seem to hang because large jobs are using all available database connections.

1. Start the Application Server Administration console, as described in Starting-the-Application-Server-Administration-console.

2. To set a maximum and minimum number of job-related database connections, use either of the following commands:
   set database MaxJobExecutionConnections #
   set database MinJobExecutionConnections #
   where #is a number of database connections. Note that the job-related database connection is set to 1 even if you set MaxJobExecutionConnections=0.

   Note

   Values that you set for job-related database connections are actually divided across two job connection pools, with 75% of the value that you set applied to a primary pool and 25% to a secondary pool that is reserved for nested jobs. This means that you might run out of job-related connections sooner than expected, as discussed in Troubleshooting the Application Server.

3. To set a maximum and minimum number of non-job-related database connections, use either of the following commands:
   set database MaxGeneralConnections #
   set database MinGeneralConnections #
   where # is a number of database connections.
4. Restart the Application Server.

To set up HTTP proxy server support

This procedure describes how to set up a user name and password for authentication on an HTTP proxy server. Many organizations provide Internet access through a proxy server. If the HTTP proxy server authenticates users, it requires a user name and password, which the Application Server can provide if you perform the following procedure.

1. Start the Application Server Administration console, as described in Starting-the-Application-Server-Administration-console.
2. To identify a proxy server, enter the following command:
   set appserver HTTPProxyName <serverName>
   where <serverName> is the name of the HTTP proxy server.
3. To specify a listening port for the HTTP proxy server, enter the following command:
   set appserver HTTPProxyPort #
   where # is the port used to contact the proxy server.
4. To specify a user name provided to the HTTP proxy server, enter the following command:
   set appserver HTTPProxyUser <userName>
   where <userName> is the name of a valid user on the proxy server.
5. To specify a password, enter the following command:
   set appserver HTTPProxyPassword <password>
   where <password> is the password assigned to the proxy user you identified in the previous step.
6. Restart the Application Server.

To bind the Application Server to an IP address

Use this procedure to specify an IP address to which an Application Server should listen. You can also instruct the Application Server to listen for connections on all of its IP addresses. This procedure is primarily useful when an Application Server has more than one network interface and you want the Application Server to listen for connections on only one.

1. Start the Application Server Administration console, as described in Starting-the-Application-Server-Administration-console.
2. To specify an IP address to which the Application Server should listen, enter the following command:
   set appserver SocketsBindAddress <IP_address>|all
   In the command shown above <IP_address> is the IP address (IPv4 or IPv6) or host name to which the Application Server should listen. If you do not specify an IP address or host name, the Application Server listens on all of its IP addresses. Similarly, if you enter all in the command shown above, the Application Server listens on all IP addresses. If you have previously instructed an Application Server to listen for a specific IP address, you must use all in this command to change those instructions so the Application Server listens on all IP addresses.
3. Restart the Application Server.

To enable and disable Network Shell proxy inspection

To ensure data integrity, BMC Server Automation inspects data packets traveling between Network Shell clients and proxy servers. When this inspection reveals a problem, the packet is not delivered. You can use this procedure to turn off packet inspection.

1. Start the Application Server Administration console, as described in Starting-the-Application-Server-Administration-console.
2. Enable or disable packet inspection by entering the following command:
   set appserver EnableProxyInspection true | false
   where false turns off proxy inspection and true turns it on. By default proxy inspection is turned on.
3. Restart the Application Server.

To control Network Shell proxy sessions

To ensure proper performance of connections between clients and Application Servers of type NSH_Proxy, you can set a connection session timeout for the NSH proxy. Such session timeouts minimize the number of handshakes that take place during the connection between the client and NSH proxy.

1. Start the Application Server Administration console, as described in Starting-the-Application-Server-Administration-console.
2. Enter the following command:
   set appserver NshProxyApplicationSessionTimeOut # 
   where # is the number of seconds to allow the NSH proxy session to remain active based on the last authentication credentials that were provided, before timing out. 
3. Restart the Application Server.

As of patch 1 for version 8.7 (that is, version 8.7.00.001), you can, instead, control this timeout using a configuration file. Create a configuration file, openssl_sessions.cnf in <install-dir>/rscd. In this configuration file, include the following two parameters:

• session_reuse: The number of times an NSH session can be reused. To disable the reuse of sessions, assign a value of 0; to enable it, assign a value of 1 or more.
• session_timeout: Amount of time in seconds after which the NSH session will time out. Note that the default timeout is 86400 seconds (24 hours).

To ensure version compatibility between Application Server and client

To ensure that a connection does not take place when an Application Server and client are at different versions, you can set up a version compatibility check. This check compares the version numbers of the client and the Application Server, at a level of detail you specify. If the version numbers are not the same, the Application Server refuses the connection.

1. Start the Application Server Administration console, as described in Starting-the-Application-Server-Administration-console.
2. Enter the following command:
   set appserver VersionCompatibilityCheck major|minor|micro|build
   (In the version number 8.1.0.125, 8 is the major part, 1 is the minor part, and so on.)
   major — Sets the check to compare only the major part of the version numbers.
   minor — Sets the check to compare the major and minor parts of the version numbers. This setting is the default.
   micro — Sets the check to compare the major, minor, and micro parts of the version numbers.
   build — Sets the check to compare all four parts of the version numbers.
   For example, suppose the Application Server version is 8.1.0.125 and the client version is 8.1.0.123. If you specify:
   set appserver VersionCompatibilityCheck micro
   The check would find that the version numbers are the same and allow the connection.
   If you specify:
   set appserver VersionCompatibilityCheck build
   The check would find that the version numbers differ and would refuse the connection.
3. Restart the Application Server.

To set a maximum cache size for file system objects

The Application Server lets you specify the maximum number of file system objects that are stored in the cache. You can use this setting to improve database performance. For example, if you take a snapshot of a directory structure that contains multiple instances of the same object, the Job does not have to write the same file to the database multiple times, as the file system object is stored in the cache and can be reused.

1. Start the Application Server Administration console, as described in Starting-the-Application-Server-Administration-console.
2. To set a maximum cache size for file system objects, enter the following command:
   set appserver FileSystemObjectCacheMaxSize #
   where # is a the maximum number of file system objects that are stored in the cache. The default value is 5000.
3. Restart the Application Server.

To set the maximum number of items per page

The Application Server lets you specify a maximum number of records retrieved from a managed server, per page. These records are used when you are working with groups, smart groups, folders, search groups, database assets, custom objects, and server objects in the Configuration Object Dictionary, for example.

1. Start the Application Server Administration console, as described in Starting-the-Application-Server-Administration-console.
2. To set a maximum page size, enter the following command:
   set appserver MaxPageSize #
   where # is a the maximum number of items retrieved per page. The default value is 1000.

3. Restart the Application Server.

   Tip

   In the BMC Server Automation Console, large numbers of objects are presented in groups of 50, by default. You can adjust this display number by selecting Window > Preferences. Expand BMC and Paging Options to change the default. You can then page through these groups to make working with large numbers of objects more manageable. You must select File > Refresh after changing the default to have the change take effect.