Monitoring Application Server performance through monitoring tools

You can obtain various Application Server performance statistics through a pair of monitoring tools that are available at any TrueSight Sever Automation Console installation — the BladeLogic JConsole (a GUI) and the JMXCLI (a command line interface). These monitoring tools retrieve data from the Application Server using the Java Management Extensions (JMX) protocol, by listening to the JMX port defined for the Application Server. Alternatively, you can obtain Application Server performance statistics by monitoring the Application Server log file.

You can direct the retrieved performance statistics to more advanced monitoring systems for further analysis, as performed by BMC products BMC PATROL for TrueSight Server Automation and BMC ProactiveNet Automation Server Monitoring, which poll the JMX port of an Application Server instance at periodic intervals. Such monitoring systems enable you to configure threshold settings for the various performance statistics, so that you can receive warnings and alerts when measured statistics fall beyond acceptable performance values for a certain length of time. (See, for example, the list of statistics monitored by BMC ProactiveNet Automation Server Monitoring.)

Before you begin

Ensure that the user that you plan to use with the JConsole or the JMXCLI is associated with a role that has the BL_Administration.read authorization.

Using the TrueSight Sever Automation JConsole

The TrueSight Sever Automation JConsole (bljconsole) is a performance monitoring tool that you can use to monitor the Java process of your TrueSight Server Automation Application Server. The TrueSight Sever Automation JConsole is based on the JConsole tool provided by Oracle, which uses Java Management Extensions (JMX) technology to monitor performance and resource consumption of applications running on the Java platform.

To execute the TrueSight Sever Automation JConsole

  1. Run the bljconsole.bat or bljconsole script found within the bin directory in the TrueSight Sever Automation installation directory (typically C:\Program Files\BMC Software\BladeLogic\NSH\bin or /opt/bmc/bladeLogic/NSH/bin). 
  2. Select Connection > New Connection.
  3. Select the Authentication Profile for the environment to which you want to connect.  Note that the RMI Registry port (typically base port + 36) must be open from the system running the bljconsole to the application server.

For more information about the display of performance data in the JConsole, see the Oracle documentation for the JConsole tool of Java 1.5.

Using the JMXCLI utility

The JMXCLI is a command line interface that retrieves performance statistics from the TrueSight Server Automation Application Server and displays them through the command line as a comma delimited list. Retrieval of statistics is controlled by an associated properties file, which regulates the frequency and total length of data polling, as well as the exact statistics to be retrieved. You can save the output to various formats; for example, as a CSV file, which can then be consumed by a spreadsheet application and presented in tabular format.

To prepare the properties file

A default properties file named bl_jmx_monitor.propertiesis provided together with the JMXCLI utility, in the the br directory in the TrueSight Sever Automation installation directory (typically C:\Program Files\BMC Software\BladeLogic\NSH\br or /opt/bmc/bladeLogic/NSH/br). You can edit the contents of this file according to your needs. The properties file contains the following types of settings:

  • Data polling settings —  control the frequency and total length of data polling
  • Default statistics — performance metrics to retrieve; each defined with 4 lines of attributes. You can comment out any statistics that you are not interested in, by inserting a pound (hash) character at the beginning of each line.

The following table lists the settings in the default properties file:

Data polling

Setting

Description

refresh_interval_millis

Length of data polling interval, in milliseconds. That is, the frequency of refreshing the data from the Application Server.

The default value is 60000 milliseconds (1 minute).

refresh_count

The number of intervals, that is, the total number of times to poll the Application Server for data (at the frequency specified by the previous setting).

The default value is 2000 times. Default values for these two settings result in the JMXCLI running for 2000 minutes (almost a day and a half).

Performance metrics

Number

Statistic

Description

1

CurrentTime

Current time at the Application Server.

2

ResidentSetSize

Resident set size (RSS) of the Java process in the main memory. Total physical (non-virtual) memory used.

3

VirtualSize

Total memory used by the Java process (virtual + resident).

4

TotalJvmMemory

Total amount of memory (used + free) reserved from the operating system for the JVM.

5

UsedJvmMemory

Amount of memory currently used on the java heap.

6

FreeJvmMemory

Amount of memory available (not currently used) on the java heap.

7

NumberOfRunningJobs

Number of running jobs on the Application Server.

8

NumOfWorkItemsCached

Number of work items that were cached.

9

ClientConnectionPool : NumAvailableConnections

Number of available database connections created for clients.

10

ClientConnectionPool : NumConnections

Number of database connections currently in use by clients.

11

GeneralConnectionPool : NumAvailableConnections

Number of available database connections created in the general connection pool, a default pool used by various Application Server activities (not including job-related or client-related activities).

12

GeneralConnectionPool : NumConnections

Number of database connections from the general connection pool that are currently in use.

13

JobConnectionPool : NumAvailableConnections

Number of available database connections created for jobs.

14

JobConnectionPool : NumConnections

Number of database connections currently in use by jobs.

15

NumIdleClientWorkerThreads

Number of idle work item threads that are available. This represents the available capacity to do work when physical resources are free.

16

ActiveConnections

Number of connections that are open and are active.

To execute the JMXCLI

  1. Before running the JMXCLI, use the blcred command line utility to acquire a session credential. For more information about using blcred, see Using-the-blcred-utility.
    Base your session on any profile that is authorized to connect to the Application Server and run BLCLI commands. If necessary, prepare such a profile before you acquire the session credential, as described in Setting-up-an-authentication-profile.

  2. Run the JMXCLI from within the bin directory in the TrueSight Sever Automation installation directory (typically C:\Program Files\BMC Software\BladeLogic\NSH\bin or /opt/bmc/bladeLogic/NSH/bin).
    For a typical case, use the following command syntax:

    jmxcli -v <profile name> -d <RMI registry port> -r <role> <properties file name>

    For example:

    jmxcli -v defaultProfile -d 9836 -r BLAdmins bl_jmx_monitor.properties

    Note

    In order to connect to a system that does not run authentication services (for example, a server of type JOB), specify a default profile that authenticates you to a server of type CONFIG or ALL in the environment, and then use the RMI registry port (specified by the -d option) for the particular instance in the environment and if the instance is on a separate host use the -a option to specify the hostname of the other system.

    For information about additional options, run JMXCLI with the -h option (or without any options).

    The JMXCLI starts running and the output is displayed as a comma-delimited list.

Using the Application Server log file

The Application Server log file contains much of the information that is available through the other monitoring tools described on this page. This information appears in the log within the Memory Monitor line. You can direct this information to a monitoring agent or feed it into a monitoring system for analysis. Note that the contents of the log file cannot be customized (as opposed to the other monitoring tools described here).

By default, the Application Server log file, appserver.log, is located in /opt/bmc/bladeLogic/NSH/br (on UNIX) or installationDirectory\BladeLogic\NSH\br (on Windows).

This log file contains various memory usage statistics, including statistics for the following items:

  • JVM memory
  • File descriptors
  • Work item threads
  • NSH proxy threads
  • Client connections
  • Database connections — from the client-connection pool, job-connection pool, and general-connection pool
Example

Below is an example of the Memory Monitor entries in the log file:

[20 Oct 2014 15:18:52,140] [Scheduled-System-Tasks-Thread-18] [INFO] [System:System:]
[Memory Monitor]
Total JVM (B): 328269824
Free JVM (B): 156032944
Used JVM (B): 172236880
VSize (B): 613498880
RSS (B): 469307392
Used File Descriptors: 2403
Used Work Item Threads: 0/50
Used NSH Proxy Threads: 0/15
Used Client Connections: 2/200
DB Client-Connection-Pool: 1/1/0/100/75/25
DB Job-Connection-Pool: 0/0/0/100/75/25
DB General-Connection-Pool: 1/1/0/100/75/25
Host/Appserver/Version: LEX-ENG-01/lex-eng-01/8.9.01.00


Note that the numbers of used threads and used client connections are presented as number of used out of total number of items available. For database connections from the pools, the numbers are current number of connections in primary pool (high water mark) / available connections in primary pool / used connections in primary pool / maximum connections for primary + secondary pools / maximum connections in primary pool / maximum connections in secondary pool.

The last line, with basic information about the Application Server (host name, deployment name, and version), is available as of version 8.9.01 of TrueSight Server Automation.


 

Tip: For faster searching, add an asterisk to the end of your partial query. Example: cert*