This plugin collects metrics from Apache Hadoop nodes.

Prerequisites

Meter 4.5 or later must be installed.

The Hadoop plugin supports the following Operating System.

Linux(tick)

To install the plugin

  1. From the top right of the screen, perform one of the following actions:
    • Click Settings > Data Collection, select the TrueSight meter, and select the Sources tab.

    • Click Settings > Configure Sources.
  2. Use the search box or scroll through the page to find the source system which has the TrueSight meter on which you want to install the plugin.
  3. Click the name of the source to view source details.
  4. Select the Plugins tab.
  5. Use the search box or scroll through the page to find the plugin you want to install.
  6. Click + Install to start the installation.

    Refer to the following sections for the configuration details required to collect data and view the list of plugin metrics.

Configuration details

FieldDescription
HostThe hadoop host name
TypeType of hadoop node (single node or multi node) to monitor
RM PortResource manager hadoop port number
DataNode PortDataNode hadoop port number
NameNode PortNameNode hadoop port number
PollIntervalThe frequency (in milliseconds) used to poll for metrics. Default value is 1000 ms, we do not recommend a value smaller than the default value

Advanced configuration

To customize this plugin, see Hadoop plugin on GitHub.

Plugin metrics

Metric NameDescription
HADOOP_NAMENODE_FS_CAPACITY_REMAININGRemaining disk space left in bytes
HADOOP_NAMENODE_FS_CORRUPTED_BLOCKSNumber of corrupt blocks
HADOOP_NAMENODE_FS_MISSINGED_BLOCKSNumber of missing blocks
HADOOP_NAMENODE_FS_BLOCKSED_TOTALTotal number of blocks
HADOOP_NAMENODE_FS_FILES_TOTALTotal number of files
HADOOP_NAMENODE_FS_BLOCKED_CAPACITYTotal Blocked Capacitys
HADOOP_NAMENODE_FS_UNDER_REPLICATED_BLOCKSNumber of under replicated blocks
HADOOP_NAMENODE_FS_CAPACITYED_USEDDisk usage in bytes
HADOOP_NAMENODE_FS_TOTAL_LOADTotal load on the file system
HADOOP_NAMENODE_FSSTATE_VOLUME_FAILURES_TOTALTotal volume failures
HADOOP_NAMENODE_FSSTATE_NUMBER_OF_LIVE_DATA_NODESTotal number of live data nodes
HADOOP_NAMENODE_FSSTATE_NUMBER_OF_DEAD_DATA_NODESTotal number of dead data nodes
HADOOP_NAMENODE_FSSTATE_NUMBER_OF_STALE_DATA_NODESNumber of stale data nodes
HADOOP_MAP_REDUCED_PROCESS_CPU_TIMEPrcesss cpu time
HADOOP_MAP_REDUCED_PROCESS_CPU_LOADPrcesss cpu load
HADOOP_MAP_REDUCED_SYSTEM_CPU_LOADSystem cpu load
HADOOP_MAP_REDUCED_AVAILABLE_PROCESSORSNumber of processors available
HADOOP_MAP_REDUCED_TOTAL_SWAP_SPACE_SIZETotal swap space size
HADOOP_MAP_REDUCED_FREE_SWAP_SPACE_SIZETotal swap space free
HADOOP_MAP_REDUCED_FREE_PHYSICAL_MEMORY_SIZEPhysical memory free size
HADOOP_MAP_REDUCED_TOTAL_PHYSICAL_MEMORY_SIZETotal Physical memory size
HADOOP_YARN_APPLICATION_RUNNINGThe number of running apps
HADOOP_YARN_APPLICATION_FAILEDThe number of failed apps
HADOOP_YARN_APPLICATION_KILLEDThe number of killed apps
HADOOP_YARN_APPLICATION_PENDINGThe number of pending apps
HADOOP_YARN_AVAILABLE_MEMORYThe amount of available memory
HADOOP_YARN_AVAILABLE_VCORESThe number of available virtual cores
HADOOP_YARN_NUMMBER_OF_UNHEALTHY_NODESThe number of unhealthy nodes
HADOOP_YARN_NUMMBER_OF_ACTIVE_NODESThe number of active nodes
HADOOP_YARN_NUMMBER_OF_LOST_NODESThe number of lost nodes
HADOOP_YARN_USED_MEMORYTotal used memory
HADOOP_YARN_NUMBER_OF_CONTAINERSNumber of containers
HADOOP_DATANODE_HEAP_MEMORY_USEDHeap memory used
HADOOP_DATANODE_HEAP_MEMORY_MAXHeap memory max used
HADOOP_DATANODE_GC_COUNTGC count
HADOOP_DATANODE_GC_TIME_MILLISGC time milliseconds
HADOOP_DATANODE_GC_NUMBER_OF_WARN_THREADSHOLD_EXCEEDEDGC of warn threadshold exceeded
HADOOP_DATANODE_GC_NUMBER_OF_INFO_THREADSHOLD_EXCEEDEDGC of info threadshold exceeded
HADOOP_DATANODE_GC_TOTAL_EXTRA_SLEEP_TIMEGC extra sleep time
HADOOP_DATANODE_BLOCKS_READBlocks read
HADOOP_DATANODE_BLOCKS_WRITTENBlocks written
HADOOP_DATANODE_VOLUME_FAILURESNumber of failed volumes
HADOOP_DATANODE_NUMBER_OF_FAILED_STORAGE_VOLUMESNumber of failed storage volumes
HADOOP_DATANODE_REMAINING_CAPACITYRemaining disk space