Apache Kafka
- Product name
- Publisher page
-
- [Apache|Apache]
- Category
- Distibuted Messaging System
- Release
- TKU 2024-Mar-1
- More information
- Publisher link
Product Description
Apache Kafka is a distributed, partitioned, replicated commit log service. It provides the functionality of a messaging system.
Software Pattern Summary
Product Component | OS Type | Versioning | Pattern Depth |
---|---|---|---|
Apache Kafka Server | UNIX/Windows | Active, Path, Package, File name | Instance-based |
Apache Kafka Cluster | UNIX/Windows | Software Instance | Instance-based |
Platforms Supported by the Pattern
The pattern discovers Apache Kafka Server deployed on both UNIX and Windows systems.
Identification
Software Instance Triggers
Apache Kafka pattern
The Apache Kafka pattern triggers on a java process which arguments match at least one of the following:
- libs/kafka_[^/\s]+\.jar
- Kafka
- confluent.support.metrics.SupportedKafka
If the java process contains org.apache.zookeeper.server
in the arguments, the pattern is not executed.
Apache Kafka Cluster pattern
The pattern triggers on a SoftwareInstance that matches the following criteria:
- Type is one of “Apache Kafka Server”, “Confluent Kafka Server” or “Cloudera Kafka Server”
Zoo_family
attribute matches the regular expression\S+\:\d+
Simple Identification Mappings
Name | Command | Arguments |
---|---|---|
Apache Kafka | unix_cmd 'java' | regex '/libs/kafka_[^/\s]+\.jar' |
Apache Kafka | windows_cmd 'java' or unix_cmd ‘java’ | regex 'kafka\.Kafka' |
Confluent Kafka Server | unix_cmd 'java' | regex 'io\.confluent\.support\.metrics\.SupportedKafka' |
Versioning
Find installation root
The pattern finds the installation root by parsing the trigger process arguments with one of the following regular expressions:
Regular expression | Notes |
---|---|
kafka\.Kafka\s*"?(.+)[\\/]config[\\/] | N/A |
-Xloggc:(/\S+)/logs/kafkaServer-gc\.log | UNIX only |
(?i)(.+)(?:[\\\/]libs[\\\/].+\.jar|\*) | Classpath only |
(?i)(.+)(?:[\\\/]bin[\\\/]) | Classpath only |
The pattern also checks the path to the trigger process command for the install root.
If multiple candidates are found, the pattern runs active versioning against all of the candidates and selects the first successful installation root. If no install roots are successful, the pattern selects the first installation root.
Active Versioning
To obtain version information, the pattern attempts to run the following commands:
- (UNIX only)
<install root>/bin/kafka-topics.sh –version
- (UNIX only)
<install root>/bin/kafka-topics –version
- (Windows only)
<install root>\bin\windows\kafka-topics.bat –version
Path Versioning
The pattern checks for path versioning by parsing the trigger process arguments with the following regular expressions:
\bkafka_\d(?:\.\d+){1,2}\-(\d+(?:\.\d+)*)
\bkafka_\d(?:\.\d+|){1,2}\-(\d+(?:\.\d+)*)
Package Versioning
The pattern attempts to find versioning from all packages which name starts with confluent-kafka-
File name Versioning
If the process arguments refer to “cloudera”, the file name versioning is not attempted.
The pattern lists the following searches for filenames that match the common\-config\-(\d+(?:\.\d+)*)\.jar
regular expression in the following directories:
- <installation root>/current/share/java/confluent-common
- <installation root>/share/java/confluent-common
If the pattern finds the filenames, it obtains the full version.
Additional Attributes
Finding the configuration file
(Windows) The pattern finds the path to the configuration file by parsing the trigger process arguments with the following regular expression:
kafka\.Kafka\s*"?(\w\:.+[^'"])
(UNIX) The pattern attempts to find the path to the configuration file by parsing the trigger process argument with the following regular expression:
(?i)(?:kafka\.Kafka|io\.confluent\.support\.metrics\.SupportedKafka)\s(/\S+)
If this approach fails, the pattern attempts to find the configuration filename with the following regular expression:
(?:kafka\.Kafka|io\.confluent\.support\.metrics\.SupportedKafka)\s(?:\S+/)*(\S+)
In the third approach, the path to the file is assumed to be <installation root>/config/<config file name>
Instance
The pattern obtains the instance information by parsing the configuration file with the following regular expression:
Broker.id=(\d+)
Port
The pattern obtains the port information by parsing the configuration file with one of the following regular expressions:
(?m)^\s*port=(\d+)
(?m)^\s*listeners\s*=.*?PLAINTEXT://.*?:(\d+)
If the port information is still missing, the default port is 9092.
Publisher
The pattern sets the following publisher attributes:
- If the process arguments refer to Cloudera, — "Cloudera".
- If the process arguments refer to Confluent or file name versioning is successful, — "Confluent".
- If the process arguments refer to none of Cloudera or Confluent, — "Apache".
Application Model Produced By Pattern
The pattern produces an instance-based SoftwareInstance with a key based on a configuration file path, type, and host key.
Relationship Creation
The pattern creates a dependency relationship with the Apache Zookeeper Server. The pattern detects if the configuration file contains zookeeper.connect=(\S+)
. If this attribute is detected, its information is used to find what Zookeeper server to connect to.
Clustering
The pattern creates a SoftwareCluster node that consists of all Kafka SoftwareInstances that share the same Apache Zookeeper Server.
Topics
The pattern models Kafka Topics if the relevant option is enabled. By default it is
The pattern gets details of the Topics by running one of the following commands
Windows | <install root>\bin\windows\kafka-topics.bat –zookeeper <zookeeper address> --describe |
<install root>\bin\windows\kafka-topics.bat –bootstrap-server 127.0.0.1:<port> --describe | |
UNIX | <install root>/bin/kafka-topics.sh –zookeeper <zookeeper address> --describe |
<install root>/bin/kafka-topics.sh –bootstrap-server 127.0.0.1:<port> --describe |
This command lists the names of all the Topics, along with details of their partition count and replication factor.
The pattern gets details of what consumer groups are running on each Topic by running the following command:
- %install_root%\\bin\\windows\\kafka-consumer-groups --bootstrap-server localhost:%port% --describe –all-groups
- %install_root%/bin/kafka-consumer-groups.sh --bootstrap-server localhost:%port% --describe –all-groups
The pattern creates a Detail node for each topic and links it to either a standalone SI or a cluster.
Example
The following image illustrates a typical installation of Apache Kafka:
Subject Matter Expertise
Subject Matter Expert input is welcome on any approaches towards improvement of the Apache Kafka model.
Information Sources
- Apache Kafka documentation
- How to deploy and configure Apache Kafka in Cloudera Enterprise
- Running a Multi-Broker Apache Kafka 0.8 Cluster on a Single Node
Open Issues
There are no known open issues with this pattern.
Comments
Log in or register to comment.