_Monitor_types_and_application_classes

The following monitor types/application classes are available in BMC PATROL for Hadoop.

• • Top-level containers and configuration
  • DataNode monitoring
    
  • JobHistory monitoring
  • JobTracker monitoring
  • NodeManager monitoring
  • NameNode monitoring
  • ResourceManagers monitoring
  •  Secondary NameNode monitoring
  • TaskTracker monitoring

Top-level containers and configuration

• Hadoop Environment (HADOOP_ENVIRONMENT)
• Hadoop JournalNode RPC detailed activity (HADOOP_JOURNALNODE_RPCDETAILEDACTIVITY)
• Hadoop JournalNode RPC detailed activity container (HADOOP_JOURNALNODE_RPCDETAILEDACTIVITY_CONTAINER)
• Hadoop main (HADOOP_MAIN)
• Hadoop setup (HADOOP_SETUP)

DataNode monitoring

At the DataNode level, you can use the following application classes to monitor all DataNode related activities, including DataNode file system state, remote procedure activities, Java Virtual Machines metrics, user group information metrics, and other DataNode activities. 

JobHistory monitoring

The JobHistory server saves the history of all the running jobs in the Hadoop system. The current version of Hadoop KM monitors only the availability of these history servers.

• Hadoop JobHistory Node (HADOOP_JOBHISTORY_NODE)
• Hadoop JobHistoryServer JVM metrics (HADOOP_JOBHISTORYSERVER_JVMMETRICS)
• Hadoop JobHistoryServer metrics system Stats (HADOOP_JOBHISTORYSERVER_METRICSSYSTEM_STATS)
• Hadoop JobHistoryServer RPC activity (HADOOP_JOBHISTORYSERVER_RPCACTIVITY)
• Hadoop JobHistoryServer RPC activity container (HADOOP_JOBHISTORYSERVER_RPCACTIVITY_CONTAINER)
• Hadoop JobHistoryServer UGI metrics (HADOOP_JOBHISTORYSERVER_UGIMETRICS)

JobTracker monitoring

The JobTracker is the service within Hadoop that farms out MapReduce tasks to specific nodes in the cluster, ideally the nodes that have the data, or at least are in the same rack. The JobTracker is a point of failure for the Hadoop MapReduce service. If it goes down, all running jobs are halted. The following application classes enable you to monitor the JobTracker service:

NodeManager monitoring

A NodeManager accepts instructions from the ResourceManager and manage resources available on a single node. Use the following application classes to monitor the NodeManager:

NameNode monitoring

The NameNode is the centerpiece of an HDFS file system. It keeps the directory tree of all files in the file system, and tracks where across the cluster the file data is kept. It does not store the data of these files itself. Client applications talk to the NameNode whenever they wish to locate a file, or when they want to add/copy/move/delete a file. The NameNode responds the successful requests by returning a list of relevant DataNode servers where the data lives. The following application classes enable you to monitor the NameNode:

ResourceManagers monitoring

A ResourceManager is the master that arbitrates all the available cluster resources and thus helps manage the distributed applications running on the YARN system. It works together with the per-node NodeManagers and the per-application ApplicationMasters. Use the following application classes to monitor the ResourceManagers:

 Secondary NameNode monitoring

When the NameNode goes down and the file system goes offline, there is an optional SecondaryNameNode that can be hosted on a separate machine. It only creates checkpoints of the namespace by merging the edits file into the fsimage file and does not provide any real redundancy. Use the following application classes to monitor the secondary NameNode:

TaskTracker monitoring

A TaskTracker is a node in the cluster that accepts tasks - Map, Reduce and Shuffle operations - from a JobTracker. Use the following application classes to monitor the TaskTracker node: