Spark plugin

This plugin monitors, visualizes and raises alerts on your Apache Spark Metrics Servlet sink in one-second intervals.

Prerequisites

Meter 4.2 or later must be installed.

The Apache Spark plugin 1.2.1 or later supports the following Operating System.

Linux(tick)

Plugin setup

MetricsServlet

MetricsServlet is added by default as a sink in master, worker and client driver. See the /etc/conf/metrics.properties file on your Spark installation for details.

JMV Source

Enable the jvm source (for instance master, worker, driver and executor) to get detailed metrics of the JVM uncommenting the following lines in your /etc/conf/metrics.properties
This plugin collects metrics from the master and an optional running application. You must configure the host and port for the WebUI of the master and application process.

WebUI Configuration

By default, the WebUI for the master runs on port 18080 and applications run on port 4040. These are the default values for this parameters, but you can change them based on your configuration.

To install the plugin

  1. From the top right of the screen, perform one of the following actions:
    • Click Settings > Data Collection, select the TrueSight meter, and select the Sources tab.

    • Click Settings > Configure Sources.
  2. Use the search box or scroll through the page to find the source system which has the TrueSight meter on which you want to install the plugin.
  3. Click the name of the source to view source details.
  4. Select the Plugins tab.
  5. Use the search box or scroll through the page to find the plugin you want to install.
  6. Click + Install to start the installation.

    Refer to the following sections for the configuration details required to collect data and view the list of plugin metrics.

Configuration details

PropertyDescription
HostHost of the metrics endpoint on the WebUI for Master or Application.
PortPort of the metrics endpoint on the WebUI for Master or Application.
Instance TypeType of the instance to monitor. It can be Application or Master.
Poll Time (ms)How often to poll for metrics in milliseconds
SourceThe source to display in the legend for this instance
Debug LevelIf enabled it will show additional debug output in the Plugin Console.

Advanced configuration

To customize this plugin, see Apache Spark Documentation.

Plugin metrics

Metric NameDescriptionContext
SPARK_MASTER_WORKERS_COUNTThe number of active workers on the master.Master
SPARK_MASTER_APPLICATIONS_RUNNING_COUNTRunning application count on the master.Master
SPARK_MASTER_APPLICATIONS_WAITING_COUNTWaiting application count on the master.Master
SPARK_MASTER_JVM_MEMORY_USEDMemory used by the JVM on the master.Master
SPARK_MASTER_JVM_MEMORY_COMMITTEDMemory committed by the JVM on the master.Master
SPARK_MASTER_JVM_HEAP_MEMORY_USEDHeap memory used by the JVM on the master.Master
SPARK_MASTER_JVM_HEAP_MEMORY_USAGEPercentage of heap memory used by the JVM on the master.Master
SPARK_MASTER_JVM_NONHEAP_MEMORY_COMMITTEDNon-heap memory committed by the JVM on the master.Master
SPARK_MASTER_JVM_NONHEAP_MEMORY_USEDNon-heap memory used by the JVM on the master.Master
SPARK_MASTER_JVM_NONHEAP_MEMORY_USAGEPercentage of non-heap memory usage by the JVM on the master.Master
SPARK_APP_JOBS_ACTIVEJobs running on the applicationApp
SPARK_APP_JOBS_ALLAll jobs created by the application.App
SPARK_APP_STAGES_FAILEDFailed stages for the application.App
SPARK_APP_STAGES_RUNNINGRunning stages for the application.App
SPARK_APP_STAGES_WAITINGWaiting stages for the application.App
SPARK_APP_BLKMGR_DISK_SPACE_USEDBlock manager disk space usedApp
SPARK_APP_BLKMGR_MEMORY_USEDBlock manager memory usedApp
SPARK_APP_BLKMGR_MEMORY_FREEBlock manager remaining memoryApp
SPARK_APP_JVM_MEMORY_COMMITTEDMemory committed by the JVM of the appApp
SPARK_APP_JVM_MEMORY_USEDMemory used by the JVM of the appApp
SPARK_APP_JVM_HEAP_MEMORY_COMMITTEDHeap memory committed by the JVM of the appApp
SPARK_APP_JVM_HEAP_MEMORY_USEDHeap memory used by the JVM of the appApp
SPARK_APP_JVM_HEAP_MEMORY_USAGEPercentage of heap memory in use by the JVM of the appApp
SPARK_APP_JVM_NOHEAP_MEMORY_COMMITTEDNon-heap memory committed by the JVM of the appApp
SPARK_APP_JVM_NONHEAP_MEMORY_USEDNon-heap memory used by the JVM of the appApp
SPARK_APP_JVM_NONHEAP_MEMORY_USAGEPercentage of non-heap memory in use by the JVM of the appApp
Was this page helpful? Yes No Submitting... Thank you

Comments