Monitor types and attributes
Top-level containers and configuration
- Hadoop-setup-HADOOP_SETUP
- Hadoop-main-HADOOP_MAIN
- Hadoop-Environment-HADOOP_ENVIRONMENT
- Hadoop-High-Availability-cluster-NameNode-HADOOP_HA_CLUSTER_NAME_NODE
DataNode monitoring
At the DataNode level, you can use the following application classes to monitor all DataNode related activities, including DataNode file system state, remote procedure activities, Java Virtual Machines metrics, user group information metrics, and other DataNode activities.
JobHistory monitoring
The JobHistory server saves the history of all the running jobs in the Hadoop system. The following application classes enable you to monitor the JobHistory service:
JobTracker monitoring
The JobTracker is the service within Hadoop that farms out MapReduce tasks to specific nodes in the cluster, ideally the nodes that have the data, or at least are in the same rack. The JobTracker is a point of failure for the Hadoop MapReduce service. If it goes down, all running jobs are halted. The following application classes enable you to monitor the JobTracker service:
NodeManager monitoring
A NodeManager accepts instructions from the ResourceManager and manage resources available on a single node. Use the following application classes to monitor the NodeManager:
NameNode Monitoring
ResourceManagers monitoring
A ResourceManager is the master that arbitrates all the available cluster resources and thus helps manage the distributed applications running on the YARN system. It works together with the per-node NodeManagers and the per-application ApplicationMasters. Use the following application classes to monitor the ResourceManagers:
Secondary NameNode monitoring
When the NameNode goes down and the file system goes offline, there is an optional SecondaryNameNode that can be hosted on a separate machine. It only creates checkpoints of the namespace by merging the edits file into the fsimage file and does not provide any real redundancy. Use the following application classes to monitor the secondary NameNode:
TaskTracker monitoring
A TaskTracker is a node in the cluster that accepts tasks - Map, Reduce and Shuffle operations - from a JobTracker. Use the following application classes to monitor the TaskTracker node:
JounalNodes monitoring
In a typical HA cluster, two separate machines are configured as NameNodes. At any point in time, exactly one of the NameNodes is in an Active state, and the other is in a Standby state. The Active NameNode is responsible for all client operations in the cluster, while the Standby is simply acting as a slave, maintaining enough state to provide a fast failover if necessary.
In order for the Standby node to keep its state synchronized with the Active node, both nodes communicate with a group of separate daemons called "JournalNodes".