Product Description

Apache Kafka is a distributed, partitioned, replicated commit log service. It provides the functionality of a messaging system.

Software Pattern Summary

Platforms Supported by the Pattern

The pattern discovers Apache Kafka Server deployed on both UNIX and Windows systems.

Identification

Software Instance Triggers

Apache Kafka pattern

The Apache Kafka pattern triggers on a java process which arguments match at least one of the following:

• libs/kafka_[^/\s]+\.jar
• Kafka
• confluent.support.metrics.SupportedKafka

If the java process contains org.apache.zookeeper.server in the arguments, the pattern is not executed.

Apache Kafka Cluster pattern

The pattern triggers on a SoftwareInstance that matches the following criteria:

• Type is one of “Apache Kafka Server”, “Confluent Kafka Server” or “Cloudera Kafka Server”
• Zoo_family attribute matches the regular expression \S+\:\d+

Simple Identification Mappings

Versioning

Find installation root

The pattern finds the installation root by parsing the trigger process arguments with one of the following regular expressions:

The pattern also checks the path to the trigger process command for the install root.

If multiple candidates are found, the pattern runs active versioning against all of the candidates and selects the first successful installation root. If no install roots are successful, the pattern selects the first installation root.

Active Versioning

To obtain version information, the pattern attempts to run the following commands: 

• (UNIX only) <install root>/bin/kafka-topics.sh –version 
• (UNIX only) <install root>/bin/kafka-topics –version
• (Windows only) <install root>\bin\windows\kafka-topics.bat –version 

Path Versioning

The pattern checks for path versioning by parsing the trigger process arguments with the following regular expressions:

• \bkafka_\d(?:\.\d+){1,2}\-(\d+(?:\.\d+)*)
• \bkafka_\d(?:\.\d+|){1,2}\-(\d+(?:\.\d+)*)

Package Versioning

The pattern attempts to find versioning from all packages which name starts with confluent-kafka-

File name Versioning

If the process arguments refer to “cloudera”, the file name versioning is not attempted. 

The pattern lists the following searches for filenames that match the common\-config\-(\d+(?:\.\d+)*)\.jar  regular expression in the following directories:

• <installation root>/current/share/java/confluent-common
• <installation root>/share/java/confluent-common

 If the pattern finds the filenames, it obtains the full version.

Additional Attributes

Finding the configuration file

(Windows) The pattern finds the path to the configuration file by parsing the trigger process arguments with the following regular expression:

• kafka\.Kafka\s*"?(\w\:.+[^'"])

(UNIX) The pattern attempts to find the path to the configuration file by parsing the trigger process argument with the following regular expression:

• (?i)(?:kafka\.Kafka|io\.confluent\.support\.metrics\.SupportedKafka)\s(/\S+)

If this approach fails, the pattern attempts to find the configuration filename with the following regular expression:

• (?:kafka\.Kafka|io\.confluent\.support\.metrics\.SupportedKafka)\s(?:\S+/)*(\S+)

In the third approach, the path to the file is assumed to be <installation root>/config/<config file name>

Instance

The pattern obtains the instance information by parsing the configuration file with the following regular expression:

• Broker.id=(\d+)

Port

The pattern obtains the port information by parsing the configuration file with one of the following regular expressions:

• (?m)^\s*port=(\d+)
• (?m)^\s*listeners\s*=.*?PLAINTEXT://.*?:(\d+)

If the port information is still missing, the default port is 9092.

Publisher

The pattern sets the following publisher attributes:

• If the process arguments refer to Cloudera, — "Cloudera".
• If the process arguments refer to Confluent or file name versioning is successful, — "Confluent". 
• If the process arguments refer to none of Cloudera or Confluent, — "Apache".

Application Model Produced By Pattern

The pattern produces an instance-based SoftwareInstance with a key based on a configuration file path, type, and host key.

Relationship Creation

The pattern creates a dependency relationship with the Apache Zookeeper Server. The pattern detects if the configuration file contains zookeeper.connect=(\S+). If this attribute is detected, its information is used to find what Zookeeper server to connect to.

Clustering

The pattern creates a SoftwareCluster node that consists of all Kafka SoftwareInstances that share the same Apache Zookeeper Server.

Topics

The pattern models Kafka Topics if the relevant option is enabled. By default it is

The pattern gets details of the Topics by running one of the following commands

This command lists the names of all the Topics, along with details of their partition count and replication factor. 

The pattern gets details of what consumer groups are running on each Topic by running the following command:

• %install_root%\\bin\\windows\\kafka-consumer-groups --bootstrap-server localhost:%port% --describe –all-groups
• %install_root%/bin/kafka-consumer-groups.sh --bootstrap-server localhost:%port% --describe –all-groups

The pattern creates a Detail node for each topic and links it to either a standalone SI or a cluster.

Example

The following image illustrates a typical installation of Apache Kafka:

Subject Matter Expertise

Subject Matter Expert input is welcome on any approaches towards improvement of the Apache Kafka model.

Information Sources

• Apache Kafka documentation
• How to deploy and configure Apache Kafka in Cloudera Enterprise
• Running a Multi-Broker Apache Kafka 0.8 Cluster on a Single Node

Open Issues

There are no known open issues with this pattern.