Configuring monitoring of Apache Kafka environment


As an administrator, use BMC Helix Operations Management TrueSight Operations Management to ensure that your environment stays operational. Monitor and gather performance and metrics information about your environment, and then use this information to resolve issues proactively. After installing the PATROL Agent and knowledge modules (KMs), create a monitor policy. After the policy is enabled, data collection starts. For more information about monitor policies, see Defining monitor policies.


To configure monitoring of Apache Kafka environment

  1. Perform one of the following actions:
    1. In BMC Helix Operations Managementselect Configuration > Monitor Policies > Create.
    2. In TrueSight Operations Management, select Configuration > Infrastructure Policies > Create Policy.
  2. Click Add Monitoring Configuration.
  3. From the Monitoring Solution list, select Kafka.
    The system automatically sets the values in the Version (latest), Monitor Profile, and Monitor type fields.
  4. In the Environment Configurations section, click Add to configure a new environment for monitoring.
  5. In the Environment Configurations window, enter the following details:

Field

Description

Environment Configuration

Environment name

Enter a unique name for the environment where you discover Apache Kafka instances.

Use alphanumeric characters in the name (A-Z, a-z, 0-9).

Do not use any of the following characters: ![@#$%?{}^\\\/|+=\&*();']

Connection Details

Bootstrap Server(s) (Server1:port,Server2:port)

Enter the bootstrap server names in the format: servername:port.

Separate multiple server names by a comma.

For example, server1:9092, server2:9092.

Device Mapping

Broker Device

Select one of the following options to enable the device mapping of the nodes.

Host name: The device is created by using the host name, which is returned from the API.
DNS lookup: The device is created by resolving the DNS name.
None: The device is not created.

When you select one of the device mapping options, all the database instances are discovered under the database host device.

Metric Collection Details

Endpoint Details

Select one of the following methods for metric collection:

  • JMX Endpoint
  • Aiven Endpoint

JMX port to collect metrics

Enter the JMX port number. The JMX port is used to collect the performance metrics.

For example, 9999.

Important

Make sure that the JMX port number you use is open, available, and the same for all brokers.

Remote JMX authentication

User name

Enter the user name to connect to JMX.

Password

Enter the password to connect to JMX.

Aiven Endpoint Details

Aiven Endpoint with port

Enter the Aiven endpoint with port. Ensure that Jolokia integration is enabled on Aiven to expose JMX metrics.

For more information, refer to Aiven Documentation: https://aiven.io/docs/platform/howto/integrations/access-jmx-metrics-jolokia.

Aiven User name

Enter the Jolokia endpoint username to connect to Aiven.

Aiven Password

Enter the Jolokia endpoint password to connect to Aiven.

CA Cert file (pem)

Download the Kafka CA certificate from Aiven and specify the path

For Aiven Kafka, generate the trust store file (client.truststore.jks) and key store file (client.keystore.p12) by using the access key, access certificate, and CA certificate, and then specify the path to the trust store file.

For more information, refer to Aiven Documentation: https://aiven.io/docs/products/kafka/howto/keystore-truststore.

SASL Authentication

Use SASL

Select the checkbox to enable SASL authentication.

SASL authentication type

Select the SASL authentication type. In this version, only SASL_PLAINTEXT is supported for authentication.

SASL user name 

Enter the user name to configure the SASL connection.

SASL password 

Enter the password for SASL authentication.

SSL Certificate Details

Use SSL

Select this check box to enable the SSL connection.

Trust store file

Enter the path of the trust store file.

For example, for Windows, enter C:\\apachekafka\\jks\\truststore.jks and for Linux, enter /opt/apachekafka/jks/truststore.jks.

Trust store password

Enter the password for the trust store.

Key store file

Enter the path of the key store file. This field is optional for the client and can be used for two-way authentication for the client.

For example, for Windows, enter C:\\apachekafka\\jks\\keystore.jks and for Linux /opt/apachekafka/jks/keystore.jks.

Key store password

Enter the password for the key store file. This field is optional for the client and is required only if the SSLssl.keystore.location is configured.

Key password

Enter the password for the private key in the key store. This field is optional for the client.

Filtering Options

Filter Kafka Topics

Enter the Kafka topics you want to include or exclude from the monitoring.

You can enter the exact Kafka topic name or a regular expression matching one or more topics.

For example, Metric.*, Events.*|Device.*.

Kafka Topics Filter Type

Select the include or exclude option to filter the Kafka topics.

Filter Kafka Consumer Groups

Enter the Kafka consumer groups you want to include or exclude from the monitoring.

You can enter the exact consumer group name or regular expression matching one or more consumer groups.

For example, Notif.*, Config.*|Deploy.*.

Kafka Consumer Groups Filter Type

Select the include or exclude option to filter the Kafka consumer groups.

KM Administration

Optional JVM args

Enter JVM arguments for the Java collector.

Logging

Select this option to enable PSL and Java debug.

Administration

Java home

Enter the path of the Java home directory.

For example, if your Java executable exists in the /usr/java/jdk1.8.0_45/jre/bin/java path, enter /usr/java/jdk1.8.0_45/jre as the value in this field.

 

Tip: For faster searching, add an asterisk to the end of your partial query. Example: cert*