This plugin monitors, visualizes and raises alerts on your Apache Spark Metrics Servlet sink in one-second intervals.
Meter 4.2 or later must be installed.
The Apache Spark plugin 1.2.1 or later supports the following Operating System.
By default, the WebUI for the master runs on port 18080 and applications run on port 4040. These are the default values for this parameters, but you can change them based on your configuration.
To install the plugin
- From the top right of the screen, perform one of the following actions:
Click> Data Collection, select the TrueSight meter, and select the Sources tab.
- Click > Configure Sources.
- Use the search box or scroll through the page to find the source system which has the TrueSight meter on which you want to install the plugin.
- Click the name of the source to view source details.
- Select the Plugins tab.
- Use the search box or scroll through the page to find the plugin you want to install.
Click + Install to start the installation.
|Host||Host of the metrics endpoint on the WebUI for Master or Application.|
|Port||Port of the metrics endpoint on the WebUI for Master or Application.|
|Instance Type||Type of the instance to monitor. It can be Application or Master.|
|Poll Time (ms)||How often to poll for metrics in milliseconds|
|Source||The source to display in the legend for this instance|
|Debug Level||If enabled it will show additional debug output in the Plugin Console.|
|SPARK_MASTER_WORKERS_COUNT||The number of active workers on the master.||Master|
|SPARK_MASTER_APPLICATIONS_RUNNING_COUNT||Running application count on the master.||Master|
|SPARK_MASTER_APPLICATIONS_WAITING_COUNT||Waiting application count on the master.||Master|
|SPARK_MASTER_JVM_MEMORY_USED||Memory used by the JVM on the master.||Master|
|SPARK_MASTER_JVM_MEMORY_COMMITTED||Memory committed by the JVM on the master.||Master|
|SPARK_MASTER_JVM_HEAP_MEMORY_USED||Heap memory used by the JVM on the master.||Master|
|SPARK_MASTER_JVM_HEAP_MEMORY_USAGE||Percentage of heap memory used by the JVM on the master.||Master|
|SPARK_MASTER_JVM_NONHEAP_MEMORY_COMMITTED||Non-heap memory committed by the JVM on the master.||Master|
|SPARK_MASTER_JVM_NONHEAP_MEMORY_USED||Non-heap memory used by the JVM on the master.||Master|
|SPARK_MASTER_JVM_NONHEAP_MEMORY_USAGE||Percentage of non-heap memory usage by the JVM on the master.||Master|
|SPARK_APP_JOBS_ACTIVE||Jobs running on the application||App|
|SPARK_APP_JOBS_ALL||All jobs created by the application.||App|
|SPARK_APP_STAGES_FAILED||Failed stages for the application.||App|
|SPARK_APP_STAGES_RUNNING||Running stages for the application.||App|
|SPARK_APP_STAGES_WAITING||Waiting stages for the application.||App|
|SPARK_APP_BLKMGR_DISK_SPACE_USED||Block manager disk space used||App|
|SPARK_APP_BLKMGR_MEMORY_USED||Block manager memory used||App|
|SPARK_APP_BLKMGR_MEMORY_FREE||Block manager remaining memory||App|
|SPARK_APP_JVM_MEMORY_COMMITTED||Memory committed by the JVM of the app||App|
|SPARK_APP_JVM_MEMORY_USED||Memory used by the JVM of the app||App|
|SPARK_APP_JVM_HEAP_MEMORY_COMMITTED||Heap memory committed by the JVM of the app||App|
|SPARK_APP_JVM_HEAP_MEMORY_USED||Heap memory used by the JVM of the app||App|
|SPARK_APP_JVM_HEAP_MEMORY_USAGE||Percentage of heap memory in use by the JVM of the app||App|
|SPARK_APP_JVM_NOHEAP_MEMORY_COMMITTED||Non-heap memory committed by the JVM of the app||App|
|SPARK_APP_JVM_NONHEAP_MEMORY_USED||Non-heap memory used by the JVM of the app||App|
|SPARK_APP_JVM_NONHEAP_MEMORY_USAGE||Percentage of non-heap memory in use by the JVM of the app||App|