Moviri Integrator for TrueSight Capacity Optimization - Cloudera

“Moviri Integrator for TrueSight Capacity Optimization – Cloudera” is an additional component of BMC TrueSight Capacity Optimization product. It allows extracting data from Cloudera Enterprise, which is Cloudera Hadoop distribution composed of CDH (Cloudera Data Hub) and Cloudera Manager.  Relevant capacity metrics are loaded into BMC TrueSight Capacity Optimization, which provides advanced analytics over the extracted data in the form of an interactive dashboard, the Hadoop View.

The integration supports the extraction of both performance and configuration data across different component of CDH and can be configured via parameters that allow entity filtering and many other settings. Furthermore the connector is able to replicate relationships and logical dependencies among entities such as clusters, resource pools, services and nodes.

The documentation is targeted at BMC TrueSight Capacity Optimization administrators, in charge of configuring and monitoring the integration between BMC TrueSight Capacity Optimization and Cloudera.

Requirements

Supported versions of data source software

  • Supported Cloudera Data Hub and Cloudera Manager versions: 5.1 to 5.101
  • The integration supports both Cloudera Manager bundled in Cloudera Enterprise and Cloudera Express products

1 - Cloudera Data Hub and Cloudera Manager versions 5.9 and 5.10 are s upported only if you apply the Feature Pack 1 (10.7.01) of the TrueSight Capacity Optimization 10.7.

Supported configurations of data source software

Moviri – Cloudera Extractor requires Cloudera Manager is continuously and correctly monitoring the various entities supported by the integration, full list available below. Any lack in meeting this requirement will cause lack in data coverage.

Installation

Downloading the additional package

ETL Module is made available in the form of an additional component, which you may download from BMC electronic distribution site (EPD) or retrieve from your content media.

Installing the additional package

To install the connector in the form of a TrueSight Capacity Optimization additional package, refer to Performing system maintenance tasks instructions.

Datasource Check and Configuration

Preparing to connect to the data source software

The connector included in "Moviri Integrator for TrueSight Capacity Optimization – Cloudera" use the Cloudera Java API v6 to communicate with Cloudera Manager. This is always enabled and no additional configuration is required.
Please note that only SELECT statements are used by the connector, preventing any accidental change to the environments.
The connector requires a read-only user with permissions on all the clusters that should be accessed.

Connector configuration attributes

The following table shows specific properties of the connector, all the other generic properties are documented here.


Property Name

Value Type

Required?

Default

Description

Cloudera Manager Connection

Cloudera Hostname         

String

Yes

 

Cloudera server hostname

Cloudera PortNumberYes7180Cloudera connection port
Spark HostnameStringNo
Spark server hostname
Spark PortNumberYes18080Spark connection port
UserStringYes Username
PasswordStringYes Password
Connection TimeoutNumberNo20Advanced - Connection timeout in seconds
Use Encryption (TLS)BooleanYesfalseAdvanced - Use encryption
Ignore certificate validationBooleanYesfalseAdvanced - Ignore validation of TLS certificate
Ignore common name validationBooleanYesfalseAdvanced - Ignore validation of TLS common name
Warn if version is unsupportedBooleanYesfalseAdvanced - Warn in the event the Cloudera Manager version is unsupported

Data Selection

Data GranularityMultipleYes10 minutes

Granularity of data to be imported, it is possible to choose between

  • Raw
  • 10 minutes
  • 1 hour
  • 6 hours
  • 1 day

Import nodes

Boolean

Yes

yes

Import data at node level

Import pools

Boolean

Yes

yes

Import data at pool level

Import hbaseBooleanYesyesImport data about HBASE service
Import sparkBooleanYesyesImport data about Spark service
Import HDFS usage reportBooleanYesnoImport data about HDFS usage by user (requires cluster admin premission)
Substitute any dot char in pools names with this charCharNo-Advanced - Because of the dot is a special char for the Loader component, it's suggested to change it

Time Interval Settings

Default Last Counter (YYYY-MM-DD HH24:MI:SS Z)DateYes Default last counter value
Relocate data to timezone (e.g. America/New_York, leave empty to use BCO timezone)StringNo Advanced - Timezone to which relocate any imported sample

Limit extraction to date (YYYY-MM-DD HH24:MI:SS)

Date

No

 

Advanced - Maximum date to be considered while extracting data

Max days to import in a single run (0 for no limit)

Number

No

 

Maximum days to collect in a single ETL run

Use cluster displayname for lookup instead of cluster name (default) BooleanYesnoAdvanced - Use cluster displayname as internal lookup name - useful to avoid system overwrite in TSCO if different cloudera clusters have the same cluster name and the lookup is shared between their ETL
The following image shows the list of options in the ETL configuration menu, with advanced properties.


Supported entities

The following entities are supported:

  • Hadoop Cluster
  • Hadoop Resource Pool
  • Hadoop Node

In addition to standard system performance metrics, data related to the following Hadoop specific services is gathered:

  • HDFS
  • SPARK
  • YARN
  • HBASE
  • MAP REDUCE

Hierarchy

The connector is able to replicate relationships and logical dependencies among these entities. In particular all the available Clusters are attached to the root of the hierarchy and each Cluster contains its own Nodes and Pools.









Services' data is available among the above entities' metrics, according to the following table.


HDFS

YARNHBASEMAP REDUCESPARK

Cluster

X

XXXX
Pool X   
NodeX    

Troubleshooting

For ETL troubleshooting, please refer to official BMC documentation available here.

Configuration and performance metrics mapping


Cloudera entityTSCO entityPERF/CONFCloudera metricTSCO Metricconversion factor
ClusterHadoop ClusterCONFclusterNameALIAS_NAME
ClusterHadoop ClusterCONFtotal_cores_across_hostsCPU_NUM
ClusterHadoop ClusterCONFversion + fullversionDESCRIPTION
ClusterHadoop ClusterCONF"Cloudera"HADOOP_DISTRIBUTION
ClusterHadoop ClusterCONFtotal_swap_total_across_hostsSWAP_SPACE_TOT
ClusterHadoop ClusterCONF

total_capacity_used_across_filesystems

/total_capacity_across_filesystems

TOTAL_FS_UTIL
ClusterHadoop ClusterCONFtotal_physical_memory_total_across_hostsTOTAL_REAL_MEM
ClusterHadoop ClusterPERFtotal_load_1_across_hostsCPU_RUN_QUEUE
ClusterHadoop ClusterPERF

(total_cpu_system_rate_across_hosts

+total_cpu_user_rate_across_hosts)

/total_cores_across_hosts

CPU_UTIL
ClusterHadoop ClusterPERFtotal_cpu_system_rate_across_hosts/total_cores_across_hostsCPU_UTIL_SYSTEM 
ClusterHadoop ClusterPERFtotal_cpu_user_rate_across_hosts/total_cores_across_hostsCPU_UTIL_USER
ClusterHadoop ClusterPERFtotal_cpu_iowait_rate_across_hosts/total_cores_across_hostsCPU_UTIL_WAIO
ClusterHadoop ClusterPERFtotal_physical_memory_cached_across_hostsMEM_CACHED
ClusterHadoop ClusterPERFtotal_swap_out_rate_across_hostsMEM_SWAP_OUT_RATE
ClusterHadoop ClusterPERFtotal_physical_memory_used_across_hostsMEM_USED
ClusterHadoop ClusterPERF

total_physical_memory_used_across_hosts

/total_physical_memory_total_across_hosts

MEM_UTIL
ClusterHadoop ClusterPERF

total_physical_memory_cached_across_hosts

/total_physical_memory_total_across_hosts

MEM_UTIL_CACHED
ClusterHadoop ClusterPERF

total_bytes_receive_rate_across_network_interfaces

+total_bytes_transmit_rate_across_network_interfaces

NET_BYTE_RATE
ClusterHadoop ClusterPERFtotal_swap_free_across_hostsSWAP_SPACE_FREE
ClusterHadoop ClusterPERFtotal_swap_used_across_hostsSWAP_SPACE_USED
ClusterHadoop ClusterPERFtotal_swap_used_across_hosts/swap_total_across_hostsSWAP_SPACE_UTIL
ClusterHadoop ClusterPERFtotal_capacity_across_filesystems+(-1*total_capacity_used_across_filesystems)TOTAL_FS_FREE
ClusterHadoop ClusterPERFtotal_capacity_across_filesystemsTOTAL_FS_SIZE
ClusterHadoop ClusterPERFtotal_capacity_used_across_filesystemsTOTAL_FS_USED
HBASEHadoop HBASE ServicePERFtotal_compaction_queue_size_across_regionserversCOMPACTION_QUEUE_SIZE
HBASEHadoop HBASE ServicePERFtotal_events_critical_rate_across_regionserversCRIT_EVENT_RATE
HBASEHadoop HBASE ServicePERFtotal_requests_rate_across_regionserversDISK_IO_RATE
HBASEHadoop HBASE ServicePERFtotal_read_requests_rate_across_regionserversDISK_IO_READ_RATE
HBASEHadoop HBASE ServicePERFtotal_write_requests_rate_across_regionserversDISK_IO_WRITE_RATE
HBASEHadoop HBASE ServicePERFtotal_read_bytes_rate_across_regionserversDISK_READ_RATE
HBASEHadoop HBASE ServicePERF

total_write_bytes_rate_across_regionservers

+total_read_bytes_rate_across_regionservers

DISK_TRANSFER_RATE
HBASEHadoop HBASE ServicePERFtotal_write_bytes_rate_across_regionserversDISK_WRITE_RATE
HBASEHadoop HBASE ServicePERFtotal_jvm_heap_committed_mb_across_regionserversHEAPMEM_COMMITTED1024*1024
HBASEHadoop HBASE ServicePERFtotal_jvm_max_memory_mb_across_regionserversHEAPMEM_MAX1024*1024
HBASEHadoop HBASE ServicePERFtotal_jvm_heap_used_mb_across_regionserversHEAPMEM_USED1024*1024
HBASEHadoop HBASE ServicePERF

total_jvm_heap_used_mb_across_regionservers

/total_jvm_max_memory_mb_across_regionservers

HEAPMEM_UTIL
HBASEHadoop HBASE ServicePERFtotal_jvm_non_heap_committed_mb_across_regionserversNONHEAPMEM_COMMITTED1024*1024
HBASEHadoop HBASE ServicePERFtotal_jvm_non_heap_used_mb_across_regionserversNONHEAPMEM_USED1024*1024
HBASEHadoop HBASE ServicePERFtotal_stores_across_regionserversSTORE_COUNT
HBASEHadoop HBASE ServicePERFtotal_storefiles_across_regionserversSTOREFILE_COUNT
HBASEHadoop HBASE ServicePERFtotal_storefile_index_size_across_regionserversSTOREFILE_IDX_SIZE
HBASEHadoop HBASE ServicePERFtotal_storefiles_size_across_regionserversSTOREFILE_SIZE
HDFSHadoop HDFS Resource ManagerPERF<HDFS usage report>BYUSER_HDFS_FILE_COUNT
HDFSHadoop HDFS Resource ManagerPERF<HDFS usage report>BYUSER_HDFS_TOTAL_FILE_SIZE
HDFSHadoop HDFS Resource ManagerPERFtotal_bytes_read_rate_across_datanodesDISK_READ_RATE
HDFSHadoop HDFS Resource ManagerPERFtotal_bytes_written_rate_across_datanodesDISK_WRITE_RATE
HDFSHadoop HDFS Resource ManagerPERFfiles_totalHDFS_FILES_COUNT
HDFSHadoop HDFS Resource ManagerPERFdfs_capacityHDFS_TOTAL_SIZE
HDFSHadoop HDFS Resource ManagerPERFdfs_capacity_usedHDFS_USED_SIZE
ClusterHadoop NodeCONFclusterNameCLUSTER_NAME
ClusterHadoop NodeCONF"Cloudera"HADOOP_DISTRIBUTION
HOST_JVMHadoop NodeCONFjvm_max_memory_mbBYVM_HEAPMEM_MAX1024*1024
HOST_PERFHadoop NodeCONFcoresCPU_NUM
HOST_PERFHadoop NodeCONFswap_totalSWAP_SPACE_TOT
HOST_PERFHadoop NodeCONFtotal_capacity_across_filesystemsTOTAL_FS_SIZE
HOST_PERFHadoop NodeCONFphysical_memory_totalTOTAL_REAL_MEM
HOST_HDFSHadoop NodePERFfiles_totalHDFS_FILES_COUNT
HOST_HDFSHadoop NodePERFdfs_capacityHDFS_TOTAL_SIZE
HOST_HDFSHadoop NodePERFdfs_capacity_usedHDFS_USED_SIZE
HOST_JVMHadoop NodePERFjvm_gc_rateBYVM_GC_EVENTS_RATE
HOST_JVMHadoop NodePERFjvm_gc_time_ms_rateBYVM_GC_TIME_PCT0.001
HOST_JVMHadoop NodePERFjvm_heap_committed_mbBYVM_HEAPMEM_COMMITTED1024*1024
HOST_JVMHadoop NodePERFjvm_heap_used_mbBYVM_HEAPMEM_USED1024*1024
HOST_PERFHadoop NodePERFload_1CPU_RUN_QUEUE
HOST_PERFHadoop NodePERF

(cpu_system_rate+cpu_user_rate)

/getHostFact(numCores,1)

CPU_UTIL
HOST_PERFHadoop NodePERFcpu_idle_rate/getHostFact(numCores,1)CPU_UTIL_IDLE
HOST_PERFHadoop NodePERFcpu_system_rate/getHostFact(numCores,1)CPU_UTIL_SYSTEM
HOST_PERFHadoop NodePERFcpu_user_rate/getHostFact(numCores,1)CPU_UTIL_USER
HOST_PERFHadoop NodePERFcpu_iowait_rate/getHostFact(numCores,1)CPU_UTIL_WAIO
HOST_PERFHadoop NodePERFtotal_read_bytes_rate_across_disksDISK_READ_RATE
HOST_PERFHadoop NodePERFtotal_write_bytes_rate_across_disksDISK_WRITE_RATE
HOST_PERFHadoop NodePERFphysical_memory_cachedMEM_CACHED
HOST_PERFHadoop NodePERF

physical_memory_total

+(-1*physical_memory_used)

MEM_FREE
HOST_PERFHadoop NodePERFswap_out_rateMEM_SWAP_OUT_RATE
HOST_PERFHadoop NodePERFphysical_memory_usedMEM_USED
HOST_PERFHadoop NodePERFphysical_memory_used/physical_memory_totalMEM_UTIL
HOST_PERFHadoop NodePERFphysical_memory_cached/physical_memory_totalMEM_UTIL_CACHED
HOST_PERFHadoop NodePERF

total_bytes_receive_rate_across_network_interfaces

+total_bytes_transmit_rate_across_network_interfaces

NET_BYTE_RATE
HOST_PERFHadoop NodePERFtotal_bytes_receive_rate_across_network_interfacesNET_IN_BYTE_RATE
HOST_PERFHadoop NodePERFtotal_bytes_transmit_rate_across_network_interfacesNET_OUT_BYTE_RATE
HOST_PERFHadoop NodePERFswap_freeSWAP_SPACE_FREE
HOST_PERFHadoop NodePERFswap_usedSWAP_SPACE_USED
HOST_PERFHadoop NodePERFswap_used/swap_totalSWAP_SPACE_UTIL
HOST_PERFHadoop NodePERF

total_capacity_across_filesystems

+(-1*total_capacity_used_across_filesystems)

TOTAL_FS_FREE
HOST_PERFHadoop NodePERFtotal_capacity_used_across_filesystemsTOTAL_FS_USED
HOST_PERFHadoop NodePERF

total_capacity_used_across_filesystems

/total_capacity_across_filesystems

TOTAL_FS_UTIL
YARN_POOLHadoop Resource Pool (YARN)CONF

allocated_vcores_cumulative

+available_vcores

CPU_NUM
YARN_POOLHadoop Resource Pool (YARN)CONF

allocated_memory_mb_cumulative

+available_memory_mb

TOTAL_REAL_MEM
YARN_APPHadoop Resource Pool (YARN)PERFcpu_millisecondsBYAPP_CPU_TIME0.001
YARN_APPHadoop Resource Pool (YARN)PERFapplication_durationBYAPP_DURATION
YARN_APPHadoop Resource Pool (YARN)PERFfile_bytes_readBYAPP_FILE_BYTES_READ
YARN_APPHadoop Resource Pool (YARN)PERFfile_bytes_writtenBYAPP_FILE_BYTES_WRITE
YARN_APPHadoop Resource Pool (YARN)PERFhdfs_bytes_readBYAPP_HDFS_BYTES_READ
YARN_APPHadoop Resource Pool (YARN)PERFhdfs_bytes_writtenBYAPP_HDFS_BYTES_WRITE
YARN_APPHadoop Resource Pool (YARN)PERFmap_input_bytesBYAPP_MAP_IN_BYTES
YARN_APPHadoop Resource Pool (YARN)PERFmap_input_recordsBYAPP_MAP_IN_RECORDS
YARN_APPHadoop Resource Pool (YARN)PERFmap_output_bytesBYAPP_MAP_OUT_BYTES
YARN_APPHadoop Resource Pool (YARN)PERFmap_output_recordsBYAPP_MAP_OUT_RECORDS
YARN_APPHadoop Resource Pool (YARN)PERFreduce_input_recordsBYAPP_RED_IN_RECORDS
YARN_APPHadoop Resource Pool (YARN)PERFreduce_output_recordsBYAPP_RED_OUT_RECORDS
YARN_POOLHadoop Resource Pool (YARN)PERFapps_completed_rateAPP_COMPLETION_RATE
YARN_POOLHadoop Resource Pool (YARN)PERFapps_failed_rateAPP_FAILED_RATE
YARN_POOLHadoop Resource Pool (YARN)PERFyarn_application_file_bytes_read_rateAPP_FILE_READ_RATE
YARN_POOLHadoop Resource Pool (YARN)PERFyarn_application_file_bytes_written_rateAPP_FILE_WRITE_RATE
YARN_POOLHadoop Resource Pool (YARN)PERFyarn_application_hdfs_bytes_read_rateAPP_HDFS_BYTES_READ
YARN_POOLHadoop Resource Pool (YARN)PERFyarn_application_hdfs_bytes_written_rateAPP_HDFS_BYTES_WRITE
YARN_POOLHadoop Resource Pool (YARN)PERFapps_killed_rateAPP_KILLED_RATE
YARN_POOLHadoop Resource Pool (YARN)PERFyarn_application_maps_rateAPP_MAP_LAUNCH_RATE
YARN_POOLHadoop Resource Pool (YARN)PERFapps_pendingAPP_PENDING
YARN_POOLHadoop Resource Pool (YARN)PERFyarn_application_reduces_rateAPP_RED_LAUNCH_RATE
YARN_POOLHadoop Resource Pool (YARN)PERFapps_runningAPP_RUNNING
YARN_POOLHadoop Resource Pool (YARN)PERFapps_submitted_rateAPP_SUBMITTED_RATE
YARN_POOLHadoop Resource Pool (YARN)PERFallocated_memory_mbMEM_USED
YARN_POOLHadoop Resource Pool (YARN)allocated_vcoresVCORES_USED
YARNHadoop YARN Resource ManagerCONF

total_allocated_vcores_across_yarn_pools

+total_available_vcores_across_yarn_pools

CPU_NUM
YARNHadoop YARN Resource ManagerCONF

total_allocated_memory_mb_across_yarn_pools

+total_available_memory_mb_across_yarn_pools

TOTAL_REAL_MEM1024*1024
SparkHadoop YARN Resource ManagerPERFtotalCoresAPP_SPARK_CORES
SparkHadoop YARN Resource ManagerPERFdiskUsedAPP_SPARK_DISK_BYTES
SparkHadoop YARN Resource ManagerPERFtotalGCTimeAPP_SPARK_GC_TIME
SparkHadoop YARN Resource ManagerPERFtotalInputBytesAPP_SPARK_INPUT_BYTES
SparkHadoop YARN Resource ManagerPERFmaxMemoryAPP_SPARK_MEM_TOTAL_BYTES
SparkHadoop YARN Resource ManagerPERFmemoryUsedAPP_SPARK_MEM_USED_BYTES
SparkHadoop YARN Resource ManagerPERFrddBlocksAPP_SPARK_RDD_BLOCKS
SparkHadoop YARN Resource ManagerPERFtotalShuffleReadAPP_SPARK_SHUFFLE_READ_BYTES
SparkHadoop YARN Resource ManagerPERFtotalShuffleWriteAPP_SPARK_SHUFFLE_WRITE_BYTES
SparkHadoop YARN Resource ManagerPERFtotalDurationAPP_SPARK_TASK_TIME
SparkHadoop YARN Resource ManagerPERFcompletedTasksAPP_SPARK_TASKS_COMPLETED
SparkHadoop YARN Resource ManagerPERFfailedTasksAPP_SPARK_TASKS_FAILED
SparkHadoop YARN Resource ManagerPERFtotalTasksAPP_SPARK_TASKS_TOTAL
SparkHadoop YARN Resource ManagerPERFtotalCoresBYAPP_SPARK_CORES
SparkHadoop YARN Resource ManagerPERFdiskUsedBYAPP_SPARK_DISK_BYTES
SparkHadoop YARN Resource ManagerPERFtotalGCTimeBYAPP_SPARK_GC_TIME
SparkHadoop YARN Resource ManagerPERFtotalInputBytesBYAPP_SPARK_INPUT_BYTES
SparkHadoop YARN Resource ManagerPERFmaxMemoryBYAPP_SPARK_MEM_TOTAL_BYTES
SparkHadoop YARN Resource ManagerPERFmemoryUsedBYAPP_SPARK_MEM_USED_BYTES
SparkHadoop YARN Resource ManagerPERFrddBlocksBYAPP_SPARK_RDD_BLOCKS
SparkHadoop YARN Resource ManagerPERFtotalShuffleReadBYAPP_SPARK_SHUFFLE_READ_BYTES
SparkHadoop YARN Resource ManagerPERFtotalShuffleWriteBYAPP_SPARK_SHUFFLE_WRITE_BYTES
SparkHadoop YARN Resource ManagerPERFtotalDurationBYAPP_SPARK_TASK_TIME
SparkHadoop YARN Resource ManagerPERFcompletedTasksBYAPP_SPARK_TASKS_COMPLETED
SparkHadoop YARN Resource ManagerPERFfailedTasksBYAPP_SPARK_TASKS_FAILED
SparkHadoop YARN Resource ManagerPERFtotalTasksBYAPP_SPARK_TASKS_TOTAL
YARNHadoop YARN Resource ManagerPERFyarn_application_file_bytes_read_rateAPP_FILE_READ_RATE
YARNHadoop YARN Resource ManagerPERFyarn_application_file_bytes_written_rateAPP_FILE_WRITE_RATE
YARNHadoop YARN Resource ManagerPERFyarn_application_hdfs_bytes_read_rateAPP_HDFS_BYTES_READ
YARNHadoop YARN Resource ManagerPERFyarn_application_hdfs_bytes_written_rateAPP_HDFS_BYTES_WRITE
YARNHadoop YARN Resource ManagerPERFyarn_application_maps_rateAPP_MAP_LAUNCH_RATE
YARNHadoop YARN Resource ManagerPERFyarn_application_reduces_rateAPP_RED_LAUNCH_RATE
YARNHadoop YARN Resource ManagerPERF

total_allocated_vcores_across_yarn_pools

/(total_allocated_vcores_across_yarn_pools

+total_available_vcores_across_yarn_pools)

CPU_UTIL
YARNHadoop YARN Resource ManagerPERFtotal_available_memory_mb_across_yarn_poolsMEM_FREE1024*1024
YARNHadoop YARN Resource ManagerPERFtotal_allocated_memory_mb_across_yarn_poolsMEM_USED1024*1024
YARNHadoop YARN Resource ManagerPERF

total_allocated_memory_mb_across_yarn_pools

/(total_allocated_memory_mb_across_yarn_pools

+total_available_memory_mb_across_yarn_pools)

MEM_UTIL
YARN_ALL_POOLSHadoop YARN Resource ManagerPERFapps_completed_cumulative_rateAPP_COMPLETION_RATE
YARN_ALL_POOLSHadoop YARN Resource ManagerPERFapps_failed_cumulative_rateAPP_FAILED_RATE
YARN_ALL_POOLSHadoop YARN Resource ManagerPERFapps_killed_cumulative_rateAPP_KILLED_RATE
YARN_ALL_POOLSHadoop YARN Resource ManagerPERFapps_pending_cumulativeAPP_PENDING
YARN_ALL_POOLSHadoop YARN Resource ManagerPERFapps_running_cumulativeAPP_RUNNING
YARN_ALL_POOLSHadoop YARN Resource ManagerPERFapps_submitted_cumulative_rateAPP_SUBMITTED_RATE


 

Was this page helpful? Yes No Submitting... Thank you

Comments