Kafka plugin

This plugin collects metrics from the Kafka.

Prerequisites

Meter version 4.2 or later must be installed.

The Kafka plugin 0.8.2 or later supports the following Operating Systems.

Linux(tick)
Windows(tick)
OS X(tick)

Plugin Setup

To collect statistics from a Kafka broker, enable access to the JMX remote endpoint.

Run the server specifying the JMX_PORT.

For instance, $ JMX_PORT=9990 ./bin/kafka-server-start.sh config/server.properties

To install the plugin

  1. From the top right of the screen, perform one of the following actions:
    • Click Settings > Data Collection, select the TrueSight meter, and select the Sources tab.

    • Click Settings > Configure Sources.
  2. Use the search box or scroll through the page to find the source system which has the TrueSight meter on which you want to install the plugin.
  3. Click the name of the source to view source details.
  4. Select the Plugins tab.
  5. Use the search box or scroll through the page to find the plugin you want to install.
  6. Click + Install to start the installation.

    Refer to the following sections for the configuration details required to collect data and view the list of plugin metrics.

Configuration details

Field NameDescription
HostHost of the Kafka JMX endpoint.
PortPort of the Kafka JMX endpoint. Defaults to 9990.
UsernameUsername of the Kafka JMX endpoint.
PasswordPassword of the Kafka JMX endpoint.
SourceThe Source to display in the legend for the metrics data. It will default to the hostname of the server.
Poll IntervalHow often should the plugin poll for metrics.

Advanced configuration

To customize this plugin, see Kafka plugin on GitHub.

Plugin metrics

Metric NameDescription
KAFKA_JVM_HEAPMEMORY_USEDHeap Memory Used by the JVM.
KAFKA_TOPIC_MESSAGE_IN_RATENumber of Message In on Topics per second.
KAFKA_TOPIC_BYTE_IN_RATENumber of Bytes In on Topics per second.
KAFKA_TOPIC_BYTE_OUT_RATENumber of Bytes Out from Topics per second.
KAFKA_PRODUCE_REQUEST_RATENumber of Requests by Producers per second.
KAFKA_CONSUME_REQUEST_RATENumber of requests by Fetch Consumers per second.
KAFKA_FOLLOW_REQUEST_RATENumber of requests by Fetch Followers per second.
KAFKA_PRODUCE_REQUEST_FAILSTotal number of failed Produce Requests.
KAFKA_FETCH_REQUEST_FAILSTotal number of failed Fetch Requests.
KAFKA_ISR_SHRINK_RATEIf a brokers goes down, ISR for some of the partitions will shrink.
KAFKA_ISR_EXPAND_RATENumber of ISR expansions per second.
KAFKA_PRODUCE_REQUEST_TOTAL_TIMETotal time of Producers requests.
KAFKA_CONSUME_REQUEST_TOTAL_TIMETotal time of Fetch Consumers requests.
KAFKA_FOLLOW_REQUEST_TOTAL_TIMETotal time of Fetch Followers requests.
KAFKA_PRODUCE_REQUEST_RESPONSE_TIMETotal time to send the response to the Producer.
KAFKA_CONSUME_REQUEST_RESPONSE_TIMETotal time to send the response to a Consumer.
KAFKA_FOLLOW_REQUEST_RESPONSE_TIMETotal time to send the reponse to a Follower.
KAFKA_NETWORK_PROCESSOR_IDLE_TIMEThe average fraction of time the network processors are idle.
KAFKA_REQUEST_HANDLER_THREADS_IDLE_TIMEThe average fraction of time the request handler threads are idle.
KAFKA_UNDER_REPLICATED_PARTITIONSNumber of under replicated partitions.
KAFKA_LEADER_ELECTION_RATELeader election rate. A non-zero value means broker failures.
KAFKA_UNCLEAN_LEADER_ELECTION_RATEUnclean leader election rate.
KAFKA_MAX_LAG_MESSAGES_FOLLOWER_AND_REPLICASMax lag in message between follower and leader replicas.
KAFKA_PRODUCER_PURGATORY_REQUESTS_WAITINGNumber of request waiting in the producer purgatory
KAFKA_PRODUCER_PURGATORY_REQUESTS_DELAYEDNumber of delayed request in producer purgatory
KAFKA_FETCH_PURGATORY_REQUESTS_WAITINGNumber of requests waiting in the fetch purgatory
KAFKA_FETCH_PURGATORY_REQUESTS_DELAYEDNumber of delayed request in fetch purgatory
KAFKA_LEADER_COUNTNumber of leaders.
KAFKA_PARTITIONS_COUNTNumber of partitions.
KAFKA_OFFLINE_PARTITIONS_COUNTNumber of offline partitions.
KAFKA_ACTIVE_CONTROLLER_COUNTNumber of active controllers.
Was this page helpful? Yes No Submitting... Thank you

Comments