“Moviri Integrator for TrueSight Capacity Optimization – k8s (Kubernetes) Prometheus” is an additional component of BMC TrueSight Capacity Optimization product. It allows extracting data from the Kubernetes cluster management system, a leading solution to manage cloud-native containerized environments.  Relevant capacity metrics are loaded into BMC TrueSight Capacity Optimization, which provides advanced analytics over the extracted data in the form of an interactive dashboard, the Kubernetes View.

The integration supports the extraction of both performance and configuration data across different component of the Kubernetes system and can be configured via parameters that allow entity filtering and many other settings. Furthermore, the connector is able to replicate relationships and logical dependencies among entities such as clusters, nodes, namespaces, deployments and pods.

The documentation is targeted at BMC TrueSight Capacity Optimization administrators, in charge of configuring and monitoring the integration between BMC TrueSight Capacity Optimization and Kubernetes.

• Requirements
  • Supported Versions of Data Source Software
  • Supported Versions of BMC TrueSight Capacity Optimization
  • Supported Versions of Data Source Configuration
  • Downloading the Additional Package
  • Installing the Additional Package
• Data Source Checks and Configuration
  • Preparing to Connect to the Data Source Software
    • Kubernetes API
    • Prometheus API
  • Connector Configuration Attributes
• Integration Results
  • Supported Entity Types
  • Supported Metrics
    • Data Source Metrics
    • 
      Derived Source Metrics
  • 
    Hierarchy
  • Lookup Consideration
  • Labels
• k8s Heapster to k8s Prometheus Migration

Requirements

Supported Versions of Data Source Software

• Kubernetes versions 1.5 to 1.12

• Rancher 1.6 to 2.3 (when managing a kubernetes cluster version 1.5 to 1.7)

• Openshift 3.11 to 3.12

• Prometheus 2.7 and onward

Supported Versions of BMC TrueSight Capacity Optimization

• BMC TrueSight Capacity Optimization v.11
• BMC TrueSight Capacity Optimization v.11.3.01
• BMC TrueSight Capacity Optimization v.11.5

Supported Versions of Data Source Configuration

The integrator requires the Prometheus monitoring component to be correctly monitoring the various entities supported by the integration via the kube-state-metrics service.

The connector has also the option to access to the Kubernetes API. Access to Kubernetes API is not mandatory to configure the ETL, but it is strongly suggested. Without the access to Kubernetes API, the integrator will not be able to import the following information

• Persistent volumes capacity-relevant metrics
• Kubernetes Labels for replicaset and replicasetcontroller

Downloading the Additional Package

“Moviri Integrator for TrueSight Capacity Optimization – k8s Prometheus” is available as additional component, distributed via BMC electronic distribution site (EPD).

Installing the Additional Package

To install the “Moviri Integrator for TrueSight Capacity Optimization – k8s Prometheus” as additional package, please refer to the official BMC documentation (“Performing system maintenance tasks”)

Data Source Checks and Configuration

Preparing to Connect to the Data Source Software

The Kubernetes connector adopts a specific integration strategy to be able to extract all of the key metrics of Kubernetes environments for capacity management in a scalable fashion. In particular, the connector integrates with two data sources:

• Kubernetes API: to extract entity catalogs, relationships and relevant configuration properties for selected entities
• Prometheus: to extract key entity catalogs, relationships and metrics related to the managed entities (e.g. nodes, pods, etc.)

The next sections outline the configuration required for the two data sources.

Kubernetes API

To access the Kubernetes API, the Kubernetes connector uses a Service Account. The authentication will be performed using the service account token. Additionally, in order to prevent accidental changes, the integrator service account will be granted read-only privileges and will be allowed to query a set of specific API endpoints. Here follows an example procedure to create the service account in a Kubernetes cluster using the kubectl CLI.

Create a Service Account

First of all, create the service account to be used by the Kubernetes connector:

Then, describe the service account to discover which secret is associated to it:

Now, describe the secret to get the corresponding token:

The token data ("eyJhb ... YygBQ") will be used by the Kubernetes integrator to authenticate against the API. Save the token as it will be required at the connector creation time.

Grant the Service Account Read-only Privileges

The following section outlines an example configuration on the Kubernetes cluster that is suggested in order to allow API access to the service account used by the integrator. We provide example configurations for the two most common authorization schemes used in Kubernetes clusters, namely RBAC (Role-Based Access Control) and ABAC (Attribute-Based Access Control). To identify which mode is configured in your Kubernetes cluster, please refer to the official project documentation: https://kubernetes.io/docs/admin/authorization/

RBAC Authorization

RBAC is the authorization mode enabled by default from Kubernetes 1.6 onward. To grant read-only privileges to the connector service account, a new cluster role is created. The new cluster role allows to grant specific read-only privileges to a set of API operations and entities to the connector.

Here is an example policy file that can be used for this purpose:

Now, create the cluster role and associate it to the connector service account:

ABAC Authorization

ABAC authorization grants access rights to users and service accounts via policies that are configured in a policy file. Such file is then used by the Kubernetes API server via the startup parameter --authorization-policy-file.

In order to allow read-only access to the integrator service account, the following policy line need to be appended to the aforementioned policy file:

The apiserver will need to be restarted to pick up the new policy lines.

Test the Configuration

After having performed the above configuration, it is useful to verify that the service account can successfully connects to the Kubernetes API and execute the intended operations.

To verify, execute the curl Linux command from one of the TrueSight Capacity Optimization servers:

You should get a JSON document describing the nodes comprising the Kubernetes cluster.

Prometheus API

The access to Prometheus API depends on the Kubernetes distribution and the Prometheus configuration. The following sections describe the standard procedure for the supported platforms.

OpenShift

OpenShift Container Platform Monitoring ships with a Prometheus instance for cluster monitoring and a central Alertmanager cluster. You can get the addresses for accessing Prometheus, Alertmanager, and Grafana web UIs by running:

Make sure to append https:// to these addresses. You cannot access web UIs using unencrypted connection.

Authentication is performed against the OpenShift Container Platform identity and uses the same credentials or means of authentication as is used elsewhere in OpenShift Container Platform. You need to use a role that has read access to all namespaces, such as the cluster-monitoring-view cluster role. 

Rancher

When installing Prometheus from the Catalog Apps, the default configuration sets up a Layer 7 ingress using xip.io. From the Load Balancing tab, you can see the endpoint to access Prometheus.

Other Distributions

Prometheus does not directly support authentication for connections to the Prometheus expression browser and HTTP API. If you would like to enforce basic authentication for those connections, Prometheus documentation recommends using Prometheus in conjunction with a reverse proxy and applying authentication at the proxy layer. Please refer to the official Prometheus documentation for configuring a NGINX reverse proxy with basic authentication.

Test the Configuration

After having obtained the required information, it is useful to verify the access to the Prometheus API. To verify, execute the following command from one of the TrueSight Capacity Optimization servers:

When authentication is not required

When basic authentication is required

When bear token authentication is required

Connector Configuration Attributes

The following image shows a Run Configuration example for the “Moviri Integrator for TrueSight Capacity Optimization – k8s Prometheus”

Integration Results

Supported Entity Types

The “Moviri Integrator for TrueSight Capacity Optimization – k8s Prometheus” supports the following entities

• Kubernetes Cluster
• Kubernetes Node
• Kubernetes Namespace
• Kubernetes Deployment
• Kubernetes Pod
• Kubernetes Persistent Volume

Supported Metrics

The following sections describe the metrics that are imported by the “Moviri Integrator for TrueSight Capacity Optimization – k8s Prometheus”.

Data Source Metrics

This section describes the metrics that are imported by the Kubernetes connector from the data source and are assigned to corresponding TrueSight Capacity Optimization entities and metrics, with minimal transformations.

Derived Source Metrics

This section describes the metrics that are derived by the Kubernetes connector from the data source metrics for the purpose of supporting a wide range of capacity management use cases and analyses.

Hierarchy

The connector is able to replicate relationships and logical dependencies among these entities as they are found configured within the Kubernetes cluster.

In particular, the following structure is applied:

• a Kubernetes Cluster is attached to the root of the hierarchy
• each Kubernetes Cluster contains its own Nodes, Namespaces and Persistent Volumes
• each Kubernetes Namespace contains its own Deployments and (standalone) Pods

The following image shows a sample hierarchy.

Lookup Consideration

The “Moviri Integrator for TrueSight Capacity Optimization – Prometheus” supports multiple lookup. The following table shows the lookup field shared for each entity type

Labels

The “Moviri Integrator for TrueSight Capacity Optimization – k8s Prometheus” supports Kubernetes Labels as Tags in TSCO TrueSight Capacity Optimization.
The keys in labels are imported as Tag types, and the values in labels are imported as Tag values. The integrator will replace special word delimiter appears in Kubernetes label key as underscore (_) in Tag type.
In particular, the entities we import labels are:

Here’s a snapshot for what tag looks like:

k8s Heapster to k8s Prometheus Migration

The “Moviri Integrator for TrueSight Capacity Optimization – k8s Prometheus” supports a seamless transition from entities and metrics imported by the “Moviri Integrator for TrueSight Capacity Optimization – k8s Heapster”. Please follow these steps to migrate between the two integrators:

1. Stop “Moviri Integrator for TrueSight Capacity Optimization – k8s Heapster” ETL task.
2. Install and configure the “Moviri Integrator for TrueSight Capacity Optimization – k8s Prometheus”, ensuring that the lookup is shared with the “Moviri Integrator for TrueSight Capacity Optimization – k8s Heapster” ETL task.
3. Start “Moviri Integrator for TrueSight Capacity Optimization – k8s Prometheus” ETL task.