Spark plugin

This plugin

monitors, visualizes and raises alerts on your Apache Spark Metrics Servlet sink in one-second intervals.

Prerequisites

Meter 4.2 or later must be installed.

The Apache Spark plugin 1.2.1 or later supports the following Operating System.

Linux

✅️

Plugin setup

MetricsServlet

MetricsServlet is added by default as a sink in master, worker and client driver. See the /etc/conf/metrics.properties file on your Spark installation for details.

JMV Source

Enable the jvm source (for instance master, worker, driver and executor) to get detailed metrics of the JVM uncommenting the following lines in your /etc/conf/metrics.properties

This plugin collects metrics from the master and an optional running application. You must configure the host and port for the WebUI of the master and application process.

WebUI Configuration

By default, the WebUI for the master runs on port 18080 and applications run on port 4040. These are the default values for this parameters, but you can change them based on your configuration.

To install the plugin

From the top right of the screen, perform one of the following actions:
- Click > Data Collection, select the TrueSight meter, and select the Sources tab.
- Click > Configure Sources.
Use the search box or scroll through the page to find the source system which has the TrueSight meter on which you want to install the plugin.
Click the name of the source to view source details.
Select the Plugins tab.
Use the search box or scroll through the page to find the plugin you want to install.
Click + Install to start the installation.
Refer to the following sections for the configuration details required to collect data and view the list of plugin metrics.

Configuration details

Property	Description
Host	Host of the metrics endpoint on the WebUI for Master or Application.
Port	Port of the metrics endpoint on the WebUI for Master or Application.
Instance Type	Type of the instance to monitor. It can be Application or Master.
Poll Time (ms)	How often to poll for metrics in milliseconds
Source	The source to display in the legend for this instance
Debug Level	If enabled it will show additional debug output in the Plugin Console.

Advanced configuration

To customize this plugin, see Apache Spark Documentation.

Plugin metrics

Metric Name	Description	Context
SPARK_MASTER_WORKERS_COUNT	The number of active workers on the master.	Master
SPARK_MASTER_APPLICATIONS_RUNNING_COUNT	Running application count on the master.	Master
SPARK_MASTER_APPLICATIONS_WAITING_COUNT	Waiting application count on the master.	Master
SPARK_MASTER_JVM_MEMORY_USED	Memory used by the JVM on the master.	Master
SPARK_MASTER_JVM_MEMORY_COMMITTED	Memory committed by the JVM on the master.	Master
SPARK_MASTER_JVM_HEAP_MEMORY_USED	Heap memory used by the JVM on the master.	Master
SPARK_MASTER_JVM_HEAP_MEMORY_USAGE	Percentage of heap memory used by the JVM on the master.	Master
SPARK_MASTER_JVM_NONHEAP_MEMORY_COMMITTED	Non-heap memory committed by the JVM on the master.	Master
SPARK_MASTER_JVM_NONHEAP_MEMORY_USED	Non-heap memory used by the JVM on the master.	Master
SPARK_MASTER_JVM_NONHEAP_MEMORY_USAGE	Percentage of non-heap memory usage by the JVM on the master.	Master
SPARK_APP_JOBS_ACTIVE	Jobs running on the application	App
SPARK_APP_JOBS_ALL	All jobs created by the application.	App
SPARK_APP_STAGES_FAILED	Failed stages for the application.	App
SPARK_APP_STAGES_RUNNING	Running stages for the application.	App
SPARK_APP_STAGES_WAITING	Waiting stages for the application.	App
SPARK_APP_BLKMGR_DISK_SPACE_USED	Block manager disk space used	App
SPARK_APP_BLKMGR_MEMORY_USED	Block manager memory used	App
SPARK_APP_BLKMGR_MEMORY_FREE	Block manager remaining memory	App
SPARK_APP_JVM_MEMORY_COMMITTED	Memory committed by the JVM of the app	App
SPARK_APP_JVM_MEMORY_USED	Memory used by the JVM of the app	App
SPARK_APP_JVM_HEAP_MEMORY_COMMITTED	Heap memory committed by the JVM of the app	App
SPARK_APP_JVM_HEAP_MEMORY_USED	Heap memory used by the JVM of the app	App
SPARK_APP_JVM_HEAP_MEMORY_USAGE	Percentage of heap memory in use by the JVM of the app	App
SPARK_APP_JVM_NOHEAP_MEMORY_COMMITTED	Non-heap memory committed by the JVM of the app	App
SPARK_APP_JVM_NONHEAP_MEMORY_USED	Non-heap memory used by the JVM of the app	App
SPARK_APP_JVM_NONHEAP_MEMORY_USAGE	Percentage of non-heap memory in use by the JVM of the app	App