Operations dashboard

The Operations dashboard of TrueSight Server Automation provides a consolidated status of job servers, jobs, Application Server memory usage, and database cleanup in your environment. You can use this dashboard to review and analyze the following details:

Status of jobs and other parameters related to jobs, such as work item threads and light weight work item threads
Infrastructure details of the Application Servers that are deployed in your environment
Status of database cleanups for the various database modules

The Operations dashboard has the following characteristics:

Runs on the HTTPS port of the Maintenance Server
Uses the REST APIs to retrieve data that is stored in the SQLite database (flat file database) of the Maintenance Server. For information about the REST APIs used, see Rest APIs used by the Operations dashboard.

You can access this dashboard directly or using the TrueSight Server Automation console.

Work item threads

Work items are separately-schedulable units of work performed as part of the execution of a job. It is in the execution of work items that the job carries out its responsibilities. Each job server maintains a pool of threads, known as work item threads for the execution of work items.

Light weight work item threads

Some work items are designated as lightweight work items because their execution consumes significantly fewer server resources than the execution of normal work items. In addition to the work item thread pool, each job server maintains a separate thread pool for lightweight work items, which is called the lightweight work item thread pool.

For more information, see Job-execution-framework.

Prerequisites

Do the following to view the Operations dashboard:

Deploy the Maintenance Server. For details, see Deploying-a-Maintenance-Server.
Enable the display of the Operations dashboard and start data collection for the dashboard.
Do the following:
1. Log in to the system where the Maintenance Server is deployed.
2. (Linux) Change to the following directory:
  /opt/bmc/bladelogic/NSH/br/
3. Start the blasadmin utility.
  ./blasadmin
4. Run the following command to view the name of the Maintenance Server:
  list deployments
5. Change to the Maintenance Server.
  switch <maint_server>
6. Run this command to view all the parameters of the Maintenance Server:
  show maintenance all
7. Change the value of the EnableOpsDashboard parameter to true. The default value is false.
  set maintenance enableopsdashboard true

(Optional) Configure the data collection interval and data retention period using the blasadmin settings:

Log in to the system where the Maintenance Server is deployed.
(Linux) Change to the following directory:
/opt/bmc/bladelogic/NSH/br/
Start the blasadmin utility.
./blasadmin
Run the following command to view the Maintenance Server name:
list deployments
Change to the Maintenance Server.
switch <maint_server>
Run this command to view all the parameters of the Maintenance Server:
show maintenance all

Change the value of the following parameters:

Parameter	Description
OpsDataCollectionInterval	The interval at which data is collected by the REST API from the database. The default interval is 30 seconds. Example: To change the interval to 15 minutes, specify the value as PT15M.
OpsDataRetentionPeriod	The duration up to which the collected data is stored in the database. The default value is 10, which indicates that data before 10 days from the current date is removed from the database.

Restart the Maintenance Server.
Do the following:
1. In the
  TrueSight Server Automation
  console, select Configuration > Infrastructure Management.
2. Expand the Application Servers node.
3. Right-click the Maintenance Server and select Restart.
4. Click OK.
Before launching the dashboard, ensure that the Maintenance Server status is Ready.
Do the following:
1. In the
  TrueSight Server Automation
  console, select Configuration > Infrastructure Management.
2. Expand the Application Servers node.
3. Double-click the Maintenance Server.
  The right pane shows the status of the Maintenance Server.

Analyzing jobs using the Operations dashboard

To use the Operations dashboard for analyzing jobs, do the following:

Access the Operations dashboard using one of the following ways:
- Enter the following URL in the address bar of the browser, and log in as a BLAdmin user:
  https://<ip_address>:<maintenance_server_https_port>/ops/
  For example, https://<ip_address>:12343/ops/
- Log in to the
  TrueSight Server Automation
  console, click Health dashboard under Configuration, and then click Ops dashboard.
Important
You can access the dashboard only when you log in as a member of the BLAdmins role using the SRP authentication.
(Optional) Configure the auto refresh interval for the data being displayed on the dashboard on the Configuration tab. By default, data displayed on the dashboard is refreshed every five minutes. Minimum refresh interval is 60 seconds and maximum refresh interval is 3600 seconds (1 hour).
Review and analyze the information on the dashboard.

The Operations dashboard provides the following information.

The TrueSight Server Automation tab displays the following information about jobs:

Average percentage utilization in the last 2 minutes
Last 24 hours history

Last 2 Minutes Average Utilization in %

This section displays the average utilization percentage of jobs, work item threads, and light work item threads in the last two minutes.

	Displays the following details of running jobs: Last 2 mins Job Runs: The number of jobs that were run during the last two minutes. Max Concurrent Job Runs: The maximum number of jobs that ran simultaneously on the Job Servers during the last two minutes. % Dial Indicator: The utilization of Job Servers (in percentage) during the last two minutes. The dial position indicates whether the capacity utilization of Job Servers is within normal limits. If the dial position starts moving towards the red zone, it indicates that the Job Server capacity is about to be used up. In this scenario, you might need to provision more Job Servers or balance the load between the existing Job Servers.
	Displays the following details of work item threads: Last 2 mints WITs: The number of work item threads that were used for jobs during the last two minutes. Max WITs Available: The maximum work item threads that were available for jobs during the last two minutes. % Dial Indicator: The percentage of work item threads that were used during the last two minutes. The dial position indicates whether the utilization of work item threads is within normal limits. If the dial position starts moving towards the red zone, it indicates that the maximum allocation limit of work items is approaching.
	Displays the following details of lightweight work item threads: Last 2 mins LWWITs: The number of lightweight work item threads that were used for jobs during the last two minutes. Max LWWITs Available: The maximum lightweight work item threads that were available for jobs during the last two minutes. % Dial Indicator: The percentage of lightweight work item threads that were used during the last two minutes. The dial position indicates whether the utilization of work item threads is within normal limits. If the dial position starts moving towards the red zone, it indicates that the maximum allocation limit of lightweight work items is approaching.
	Displays a list of Job Servers with these details: Allocated work item threads out of available maximum work item threads Number of jobs that are run

Last 24 Hours History

This section displays the following job-related data for the last 24 hours:

Running jobs and jobs that are waiting to be run
Work item threads and lightweight work item threads

Waiting to run Vs Active Running Jobs Count The Waiting to run Vs Active Running Jobs Count chart shows the following data for the last 24 hours: Date-wise count of the actively running jobs and jobs that were waiting to be run. Threshold value that indicates the maximum number of jobs that are processed by the Application Servers. Click Threshold to show or hide the threshold line on the graph. Select a value on the X axis and drag it to select a specific duration for an enlarged view. Click to restore the original view.
Running Jobs History The Running Jobs History chart shows the following data for the last 24 hours: Date-wise count of jobs that were running on each Job Server Threshold value that indicates the maximum number of jobs that are processed by the Application Server Click Threshold to show or hide the threshold line on the graph. Select a value on the X axis and drag it to select a specific duration for an enlarged view. Click to restore the original view.
	The Work Item Threads chart shows the following historical data for the last 24 hours: Date-wise count of work item threads that were used for each Application Server deployment Threshold value that indicates the count of maximum available work item threads Click Threshold to show or hide the threshold line on the graph. Select a value on the X axis and drag it to select a specific duration for an enlarged view. Click to restore the original view.
	The Light Weight Work Item Threads chart shows the following historical data for the last 24 hours: Date-wise count of lightweight work item threads that were used for each Application Server deployment. Threshold value that indicates the count of maximum available lightweight work item threads. Click Threshold to show or hide the threshold line on the graph. Select a value on the X axis and drag it to select a specific duration for an enlarged view. Click to restore the original view.
JVM utilization metrics The JVM Utilization Metrics chart shows a comparative analysis of JVM utilization for all the Application Server deployments in the last 24 hours. Select a value on the X axis and drag it to select a specific duration for an enlarged view. Click to restore the original view. Hover your mouse over to view the following details:

The Deployments tab displays the infrastructure details of each Application Server that is deployed in your environment.

Click an Application Server from the list to view its details, such as OS name, version, JRE name, version, free and used memory.

NEW IN 21.02

The Database tab shows the following information:

Statistics for the usage of database modules
Status of the historical database cleanups
The blasadmin settings configured on the Maintenance Server for the auto cleanup

Statistics

Click Statistics to view the number of records for each module and the growth trend for each module.

DB Records Per DB Module

The pie chart shows the distribution of records according to database modules. The table shows the count of records for each database module. Use Search to quickly locate the required module and its records.

DB Growth Trend

The area chart shows the module-wise growth of database records. To view the records growth only for the selected modules, clear the check boxes corresponding to the other modules. Select a value on the X axis and drag it to select a specific duration for an enlarged view. Click to restore the original view.

Cleanup

Click Cleanup to view the historical trend of cleanups that were completed in your environment, the module-wise status of cleanup jobs, and daily and weekly cleanup history of cleanup modules.

Calculated DB Cleanup Modules Growth Trend

This area chart shows the module-wise growth trend of records that are available in the database after the daily cleanup operation. The records count that is displayed on the chart is calculated by subtracting the cleaned-up records from the total records for each module. You can use this information to make the appropriate changes to the daily cleanup schedules. The chart data is refreshed once a day.

To view the records growth only for the selected modules, clear the check boxes corresponding to the other modules. Select a value on the X axis and drag it to select a specific duration for an enlarged view. Click to restore the original view.

Daily Cleanup Runs

This table shows the module-wise status of cleanup jobs that were run during the last six months. The status includes the number of deleted rows, time taken for the cleanup, and timestamp when the cleanup was run. The modules are arranged in the descending order of deleted records. The module with the highest deleted records is placed at the top of the table. You can use the Search filter to quickly locate the required module from the list.

Daily Cleanup History

This area chart shows the trend of daily cleanup execution for each database cleanup module. To view the cleanup history only for the selected modules, clear the check boxes corresponding to the other modules.

Weekly Cleanup History

This table shows the module-wise status of weekly cleanup jobs. The status includes the number of deleted rows, time taken for the cleanup, and timestamp when the cleanup was run. The modules are arranged in the descending order of the number of deleted records. The module with the highest time for the cleanup is placed at the top of the table. You can use the Search filter to quickly locate the required module from the list.

AutoCleanup Configurations

Click AutoCleanup Configurations to view the blasadmin settings related to auto cleanup and Operations dashboard that are configured on the Maintenance Server.

REST APIs used by the Operations dashboard

The Operations dashboard uses the following REST APIs to retrieve data. Make sure that you acquire the session before accessing any of these APIs. You can access these APIs from the following Swagger URL:

<ip_address_of_maintenance_server_host>:<maint_server_https_port>/rest/swagger-ui.html#/metrics

Module: Metric APIs
Method	API
GET	/api/v1/metrics/{param}
GET	/api/v1/metrics/{param}/{deploymentID}
GET	/api/v1/metrics/24 hours/{param}
GET	/api/v1/metrics/all
GET	/api/v1/metrics/deployment/{deploymentID}
GET	/api/v1/metrics/last1krecs
GET	/api/v1/metrics/listalldeployments
GET	/api/v1/metrics/listallparameters
GET	/api/v1/metrics/listDbservicesDetails
GET	/api/v1/metrics/listInfraDetails
GET	/api/v1/metrics/listJobManagerServiceDetails
GET	/api/v1/metrics/db/statistics
GET	/api/v1/metrics/dbcleanup/configs
GET	/api/v1/metrics/dbcleanup/dailycleanupinfo
GET	/api/v1/metrics/dbcleanup/history
GET	/api/v1/metrics/dbcleanup/weeklycleanupinfo