Using the Operations Overview dashboard

The Operations Overview dashboard provides a consolidated status of jobs and agent activity, and TrueSight Server Automation database performance. The dashboard is organized into the following tabs:

System
Database

You can sort the data according to your requirements. Also, you can export individual sections from the dashboard in CSV or PDF format to meet your reporting and compliance needs.

Supported browsers

Access the Live Report dashboard by using the latest version of one of the following browsers:

Chrome
Firefox
Microsoft Edge

Before you begin

Make sure that you have been assigned the BLAdmins role.

To view the dashboard

Enter the following URL in the browser:

https://<AppServerHost>:<Port>/consolidated/#/operationsOverview

Example:

https://aus-bsa-004:9843/consolidated/#/operationsOverview

Variable	Description
<AppServerHost>	Fully qualified name or IP address of the Application Server on which the Tomcat service has been enabled.
<Port>	The port on the Application Server on which the Tomcat service is running. The default port is 9843.

Enter the user name and password.

System tab

The System tab helps you to monitor infrastructure health, troubleshoot performance issues, and optimize resource usage.

The following table describes the panels in the System tab:

Panel	Description
Jobs	Displays the following job-level statistics with actionable hints for troubleshooting the job reliability and performance issues: Number of jobs with high failure rates Number of servers failing across multiple jobs Jobs running over 8 hours Example:
Agents	Displays the agent health and activity: Number of agents that are alive Latest agent version count Agents active in the last 30 days Example:
Memory Utilization	Displays the system-level memory usage and availability by using various parameters, including: Total physical OS Memory (in MB) Available physical OS memory (in MB) Physical OS memory used (in MB) System memory state description Example:

Database tab

The Database tab provides a comprehensive view of the TrueSight Server Automation database, such as its growth trends, session information, and CPU utilization.

Zoomed Databse.png

The following table describes the panels in the Database tab:

Panel	Description
Growth Per Month Information	Displays the monthly growth trend of the TrueSight Server Automation database: X-axis: Month Y-axis: Database size (in GB) It helps you track monthly changes in the total database size. Example:
Data Distribution by Module-in million	Displays the distribution of records according to database modules. Example:
File Size	Displays the following metrics for the database files: Storage type File size (in MB) Used space (in MB) Unused space (in MB) File type Example:
Session Information	Displays real-time metrics on database sessions: Transaction mode Max connections allowed Number of connection sessions in the sleeping, running, or inactive status Example:
Top 10 Large Tables	Displays the following information about the top ten largest tables: Number of rows Table size Example:
CPU Util Information	Displays the CPU usage across the following processes: SQL process Other processes System Idle process

Click here to view information about the existing Operations dashboard.

The Operations dashboard of TrueSight Server Automation provides a consolidated status of job servers, jobs, Application Server memory usage, and database cleanup in your environment. You can use this dashboard to review and analyze the following details:

Status of jobs and other parameters related to jobs, such as work item threads and light weight work item threads
Infrastructure details of the Application Servers that are deployed in your environment
Status of database cleanups for the various database modules

The Operations dashboard has the following characteristics:

Runs on the HTTPS port of the Maintenance Server
Uses the REST APIs to retrieve data that is stored in the SQLite database (flat file database) of the Maintenance Server. For information about the REST APIs used, see Rest APIs used by the Operations dashboard.

You can access this dashboard directly or using the TrueSight Server Automation console.

Work item threads

Work items are separately-schedulable units of work performed as part of the execution of a job. It is in the execution of work items that the job carries out its responsibilities. Each job server maintains a pool of threads, known as work item threads for the execution of work items.

Light weight work item threads

Some work items are designated as lightweight work items because their execution consumes significantly fewer server resources than the execution of normal work items. In addition to the work item thread pool, each job server maintains a separate thread pool for lightweight work items, which is called the lightweight work item thread pool.

For more information, see Job-execution-framework.

Prerequisites

Do the following to view the Operations dashboard:

Deploy the Maintenance Server. For details, see Deploying-a-Maintenance-Server.
Enable the display of the Operations dashboard and start data collection for the dashboard.
Do the following:
1. Log in to the system where the Maintenance Server is deployed.
2. (Linux) Change to the following directory:
  /opt/bmc/bladelogic/NSH/br/
3. Start the blasadmin utility.
  ./blasadmin
4. Run the following command to view the name of the Maintenance Server:
  list deployments
5. Change to the Maintenance Server.
  switch <maint_server>
6. Run this command to view all the parameters of the Maintenance Server:
  show maintenance all
7. Change the value of the EnableOpsDashboard parameter to true. The default value is false.
  set maintenance enableopsdashboard true

(Optional) Configure the data collection interval and data retention period using the blasadmin settings:

Log in to the system where the Maintenance Server is deployed.
(Linux) Change to the following directory:
/opt/bmc/bladelogic/NSH/br/
Start the blasadmin utility.
./blasadmin
Run the following command to view the Maintenance Server name:
list deployments
Change to the Maintenance Server.
switch <maint_server>
Run this command to view all the parameters of the Maintenance Server:
show maintenance all

Change the value of the following parameters:

Parameter	Description
OpsDataCollectionInterval	The interval at which data is collected by the REST API from the database. The default interval is 30 seconds. Example: To change the interval to 15 minutes, specify the value as PT15M.
OpsDataRetentionPeriod	The duration up to which the collected data is stored in the database. The default value is 10, which indicates that data before 10 days from the current date is removed from the database.

Restart the Maintenance Server.
Do the following:
1. In the
  TrueSight Server Automation
  console, select Configuration > Infrastructure Management.
2. Expand the Application Servers node.
3. Right-click the Maintenance Server and select Restart.
4. Click OK.
Before launching the dashboard, ensure that the Maintenance Server status is Ready.
Do the following:
1. In the
  TrueSight Server Automation
  console, select Configuration > Infrastructure Management.
2. Expand the Application Servers node.
3. Double-click the Maintenance Server.
  The right pane shows the status of the Maintenance Server.

Analyzing jobs using the Operations dashboard

To use the Operations dashboard for analyzing jobs, do the following:

Access the Operations dashboard using one of the following ways:
- Enter the following URL in the address bar of the browser, and log in as a BLAdmin user:
  https://<ip_address>:<maintenance_server_https_port>/ops/
  For example, https://<ip_address>:12343/ops/
- Log in to the
  TrueSight Server Automation
  console, click Health dashboard under Configuration, and then click Ops dashboard.
Warning
Important
You can access the dashboard only when you log in as a member of the BLAdmins role using the SRP authentication.
(Optional) Configure the auto refresh interval for the data being displayed on the dashboard on the Configuration tab. By default, data displayed on the dashboard is refreshed every five minutes. Minimum refresh interval is 60 seconds and maximum refresh interval is 3600 seconds (1 hour).
Review and analyze the information on the dashboard.

The Operations dashboard provides the following information.

The TrueSight Server Automation tab displays the following information about jobs:

Average percentage utilization in the last 2 minutes
Last 24 hours history

Last 2 Minutes Average Utilization in %

This section displays the average utilization percentage of jobs, work item threads, and light work item threads in the last two minutes.

	Displays the following details of running jobs: Last 2 mins Job Runs: The number of jobs that were run during the last two minutes. Max Concurrent Job Runs: The maximum number of jobs that ran simultaneously on the Job Servers during the last two minutes. % Dial Indicator: The utilization of Job Servers (in percentage) during the last two minutes. The dial position indicates whether the capacity utilization of Job Servers is within normal limits. If the dial position starts moving towards the red zone, it indicates that the Job Server capacity is about to be used up. In this scenario, you might need to provision more Job Servers or balance the load between the existing Job Servers.
	Displays the following details of work item threads: Last 2 mints WITs: The number of work item threads that were used for jobs during the last two minutes. Max WITs Available: The maximum work item threads that were available for jobs during the last two minutes. % Dial Indicator: The percentage of work item threads that were used during the last two minutes. The dial position indicates whether the utilization of work item threads is within normal limits. If the dial position starts moving towards the red zone, it indicates that the maximum allocation limit of work items is approaching.
	Displays the following details of lightweight work item threads: Last 2 mins LWWITs: The number of lightweight work item threads that were used for jobs during the last two minutes. Max LWWITs Available: The maximum lightweight work item threads that were available for jobs during the last two minutes. % Dial Indicator: The percentage of lightweight work item threads that were used during the last two minutes. The dial position indicates whether the utilization of work item threads is within normal limits. If the dial position starts moving towards the red zone, it indicates that the maximum allocation limit of lightweight work items is approaching.
	Displays a list of Job Servers with these details: Allocated work item threads out of available maximum work item threads Number of jobs that are run

Last 24 Hours History

This section displays the following job-related data for the last 24 hours:

Running jobs and jobs that are waiting to be run
Work item threads and lightweight work item threads

Waiting to run Vs Active Running Jobs Count The Waiting to run Vs Active Running Jobs Count chart shows the following data for the last 24 hours: Date-wise count of the actively running jobs and jobs that were waiting to be run. Threshold value that indicates the maximum number of jobs that are processed by the Application Servers. Click Threshold to show or hide the threshold line on the graph. Select a value on the X axis and drag it to select a specific duration for an enlarged view. Click to restore the original view.
Running Jobs History The Running Jobs History chart shows the following data for the last 24 hours: Date-wise count of jobs that were running on each Job Server Threshold value that indicates the maximum number of jobs that are processed by the Application Server Click Threshold to show or hide the threshold line on the graph. Select a value on the X axis and drag it to select a specific duration for an enlarged view. Click to restore the original view.
	The Work Item Threads chart shows the following historical data for the last 24 hours: Date-wise count of work item threads that were used for each Application Server deployment Threshold value that indicates the count of maximum available work item threads Click Threshold to show or hide the threshold line on the graph. Select a value on the X axis and drag it to select a specific duration for an enlarged view. Click to restore the original view.
	The Light Weight Work Item Threads chart shows the following historical data for the last 24 hours: Date-wise count of lightweight work item threads that were used for each Application Server deployment. Threshold value that indicates the count of maximum available lightweight work item threads. Click Threshold to show or hide the threshold line on the graph. Select a value on the X axis and drag it to select a specific duration for an enlarged view. Click to restore the original view.
JVM utilization metrics The JVM Utilization Metrics chart shows a comparative analysis of JVM utilization for all the Application Server deployments in the last 24 hours. Select a value on the X axis and drag it to select a specific duration for an enlarged view. Click to restore the original view. Hover your mouse over to view the following details:

The Deployments tab displays the infrastructure details of each Application Server that is deployed in your environment.

Click an Application Server from the list to view its details, such as OS name, version, JRE name, version, free and used memory.

NEW IN 21.02

The Database tab shows the following information:

Statistics for the usage of database modules
Status of the historical database cleanups
The blasadmin settings configured on the Maintenance Server for the auto cleanup

Statistics

Click Statistics to view the number of records for each module and the growth trend for each module.

DB Records Per DB Module

The pie chart shows the distribution of records according to database modules. The table shows the count of records for each database module. Use Search to quickly locate the required module and its records.

DB Growth Trend

The area chart shows the module-wise growth of database records. To view the records growth only for the selected modules, clear the check boxes corresponding to the other modules. Select a value on the X axis and drag it to select a specific duration for an enlarged view. Click to restore the original view.

Cleanup

Click Cleanup to view the historical trend of cleanups that were completed in your environment, the module-wise status of cleanup jobs, and daily and weekly cleanup history of cleanup modules.

Calculated DB Cleanup Modules Growth Trend

This area chart shows the module-wise growth trend of records that are available in the database after the daily cleanup operation. The records count that is displayed on the chart is calculated by subtracting the cleaned-up records from the total records for each module. You can use this information to make the appropriate changes to the daily cleanup schedules. The chart data is refreshed once a day.

To view the records growth only for the selected modules, clear the check boxes corresponding to the other modules. Select a value on the X axis and drag it to select a specific duration for an enlarged view. Click to restore the original view.

Daily Cleanup Runs

This table shows the module-wise status of cleanup jobs that were run during the last six months. The status includes the number of deleted rows, time taken for the cleanup, and timestamp when the cleanup was run. The modules are arranged in the descending order of deleted records. The module with the highest deleted records is placed at the top of the table. You can use the Search filter to quickly locate the required module from the list.

Daily Cleanup History

This area chart shows the trend of daily cleanup execution for each database cleanup module. To view the cleanup history only for the selected modules, clear the check boxes corresponding to the other modules.

Weekly Cleanup History

This table shows the module-wise status of weekly cleanup jobs. The status includes the number of deleted rows, time taken for the cleanup, and timestamp when the cleanup was run. The modules are arranged in the descending order of the number of deleted records. The module with the highest time for the cleanup is placed at the top of the table. You can use the Search filter to quickly locate the required module from the list.

AutoCleanup Configurations

Click AutoCleanup Configurations to view the blasadmin settings related to auto cleanup and Operations dashboard that are configured on the Maintenance Server.

REST APIs used by the Operations dashboard

The Operations dashboard uses the following REST APIs to retrieve data. Make sure that you acquire the session before accessing any of these APIs. You can access these APIs from the following Swagger URL:

<ip_address_of_maintenance_server_host>:<maint_server_https_port>/rest/swagger-ui.html#/metrics

Module: Metric APIs
Method	API
GET	/api/v1/metrics/{param}
GET	/api/v1/metrics/{param}/{deploymentID}
GET	/api/v1/metrics/24 hours/{param}
GET	/api/v1/metrics/all
GET	/api/v1/metrics/deployment/{deploymentID}
GET	/api/v1/metrics/last1krecs
GET	/api/v1/metrics/listalldeployments
GET	/api/v1/metrics/listallparameters
GET	/api/v1/metrics/listDbservicesDetails
GET	/api/v1/metrics/listInfraDetails
GET	/api/v1/metrics/listJobManagerServiceDetails
GET	/api/v1/metrics/db/statistics
GET	/api/v1/metrics/dbcleanup/configs
GET	/api/v1/metrics/dbcleanup/dailycleanupinfo
GET	/api/v1/metrics/dbcleanup/history
GET	/api/v1/metrics/dbcleanup/weeklycleanupinfo