Moviri – K8s (Kubernetes) Prometheus Extractor

“Moviri Integrator for BMC Helix Continuous Optimization – k8s (Kubernetes) Prometheus” is an additional component of BMC BMC Helix Continuous Optimization product. It allows extracting data from the Kubernetes cluster management system, a leading solution to manage cloud-native containerized environments.  Relevant capacity metrics are loaded into BMC Helix Continuous Optimization, which provides advanced analytics over the extracted data in the form of an interactive dashboard, the Kubernetes View. 

The integration supports the extraction of both performance and configuration data across different component of the Kubernetes system and can be configured via parameters that allow entity filtering and many other settings. Furthermore, the connector is able to replicate relationships and logical dependencies among entities such as clusters, nodes, namespaces, deployments, and pods.

The documentation is targeted at BMC BMC Helix Continuous Optimization administrators, in charge of configuring and monitoring the integration between BMC BMC Helix Continuous Optimization and Kubernetes.

• • Step I. Complete the pre-configuration tasks
    • • Check supported versions of data source configuration
      • Generate access to Prometheus API
      • Verify Prometheus API access
      • Generate Access to Kubernetes API (Optional)
      • Verify Kubernetes API access
  • Step II. Configure the ETL
    • A. Configuring the basic properties
    • (Optional) B. Configuring the advanced properties
  • Step III. Run the ETL
    • A. Running the ETL in simulation mode
    • B. Running the ETL in the production mode
      • Running the ETL manually
      • Scheduling the ETL run
  • Step IV. Verify data collection
    • K8s Prometheus Workplace Entities
    • Entity Relationship
    • Hierarchy
    • Lookup Field Considerations
    • Tag Mapping (Optional)
  • k8s Heapster to k8s Prometheus Migration
  • Pod Optimization - Pod Workloads replace Pods
  • Common Issues
  • Data Verification
    • Verify Prometheus Build information
    • Verify the Status of Prometheus Target Services
    • Verify data availability in Prometheus Tables

###############################################################################
# 1) Service Account
###############################################################################
apiVersion: v1
kind: ServiceAccount
metadata:
  name: bhco-viewer
  namespace: default
---
###############################################################################
# 2) Secret for the Service Account
###############################################################################
apiVersion: v1
kind: Secret
metadata:
  name: bhco-viewer-secret
  namespace: default
  annotations:
    kubernetes.io/service-account.name: bhco-viewer
type: kubernetes.io/service-account-token
---
###############################################################################
# 3) Cluster role new
###############################################################################
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: bhco-viewer-role
rules:
  - apiGroups:
    - ""
    resources:
    - limitranges
    - namespaces
    - nodes
    - persistentvolumeclaims
    - persistentvolumes
    - pods
    - replicationcontrollers
    - resourcequotas
    verbs:
    - get
    - list
  - apiGroups:
    - apps
    resources:
    - deployments
    - replicasets
    - statefulsets
    verbs:
    - get
    - list
  - apiGroups:
    - autoscaling
    resources:
    - HorizontalPodAutoscalers
    verbs:
    - get
    - list
  - apiGroups:
    - monitoring.coreos.com
    resources:
    - prometheuses
    - prometheuses/api
    verbs:
    - get
    - list
---
###############################################################################
# 4) Cluster role binding
###############################################################################
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: bhco-viewer-binding
subjects:
  - kind: ServiceAccount
    name: bhco-viewer
    namespace: default
roleRef:
  kind: ClusterRole
  name: bhco-viewer-role
  apiGroup: rbac.authorization.k8s.io

Step II. Configure the ETL

A. Configuring the basic properties

Some of the basic properties display default values. You can modify these values if required.

To configure the basic properties:

1. In the console, navigate to Administration > ETL & System Tasks, and select ETL tasks.
2. On the ETL tasks page, click Add > Add ETL. The Add ETL page displays the configuration properties. You must configure properties in the following tabs: Run configuration, Entity catalog, and Amazon Web Services Connection
3. On the Run Configuration tab, select Moviri - k8s Prometheus Extractor from the ETL Module list. The name of the ETL is displayed in the ETL task name field. You can edit this field to customize the name.

1. Click the Entity catalog tab, and select one of the following options:

   1. Shared Entity Catalog:

      1. From the Sharing with Entity Catalog list, select the entity catalog name that is shared between ETLs.
   2. Private Entity Catalog: Select if this is the only ETL that extracts data from the k8s Prometheus resources.
2. Click the Connection tab, and configure the following properties:

1. Click the Kubernetes Extraction tab, and configure the following properties:

The following image shows a Run Configuration example for the “Moviri Integrator for BMC Helix Continuous Optimization – k8s Prometheus":

1. Click on the Import Filters tab, and configure the following property, if needed:

1. (Optional) Override the default values of the properties:

(Optional) B. Configuring the advanced properties

You can configure the advanced properties to change the way the ETL works or to collect additional metrics

To configure the advanced properties:

1. On the Add ETL page, click Advanced.
2. Configure the following properties:

1. Click Save.

The ETL tasks page shows the details of the newly configured Prometheus ETL:

Step III. Run the ETL

After you configure the ETL, you can run it to collect data. You can run the ETL in the following modes:

A. Simulation mode: Only validates connection to the data source, does not collect data. Use this mode when you want to run the ETL for the first time or after you make any changes to the ETL configuration.

B. Production mode: Collects data from the data source.

A. Running the ETL in simulation mode

To run the ETL in the simulation mode:

1. In the console, navigate to Administration > ETL & System Tasks, and select ETL tasks.
2. On the ETL tasks page, click the ETL. The ETL details are displayed.

1. In the Run configurations table, click the pencil icon to modify the ETL configuration settings.
2. On the Run configuration tab, ensure that the Execute in simulation mode option is set to Yes, and click Save.
3. Click Run active configuration. A confirmation message about the ETL run job submission is displayed.
4. On the ETL tasks page, check the ETL run status in the Last exit column.
   OK Indicates that the ETL ran without any error. You are ready to run the ETL in the production mode.

5.  If the ETL run status is Warning, Error, or Failed:

   1. On the ETL tasks page, click the pencil icon in the last column of the ETL name row.
   2. Check the log and reconfigure the ETL if required.
   3. Run the ETL again.
   4. Repeat these steps until the ETL run status changes to OK.

B. Running the ETL in the production mode

You can run the ETL manually when required or schedule it to run at a specified time.

Running the ETL manually

1. On the ETL tasks page, click the ETL. The ETL details are displayed.
2. In the Run configurations table, click the pencil icon to modify the ETL configuration settings. The Edit run configuration page is displayed.
3. On the Run configuration tab, select No for the Execute in simulation mode option, and click Save.
4. To run the ETL immediately, click Run active configuration. A confirmation message about the ETL run job submission is displayed.
   When the ETL is run, it collects data from the source and transfers it to the database.

Scheduling the ETL run

By default, the ETL is scheduled to run daily. You can customize this schedule by changing the frequency and period of running the ETL.

To configure the ETL run schedule:

1. On the ETL tasks page, click the ETL, and click Edit Task. The ETL details are displayed.

1. On the Edit task page, do the following, and click Save:

   1. Specify a unique name and description for the ETL task.
   2. In the Maximum execution time before warning field, specify the duration for which the ETL must run before generating warnings or alerts, if any.
   3. Select a predefined or custom frequency for starting the ETL run. The default selection is Predefined.
   4. Select the task group and the scheduler to which you want to assign the ETL task.
2. Click Schedule. A message confirming the scheduling job submission is displayed.
   When the ETL runs as scheduled, it collects data from the source and transfers it to the database.

Step IV. Verify data collection

Verify that the ETL ran successfully and check whether the k8s Prometheus data is refreshed in the Workspace.

To verify whether the ETL ran successfully:

1. In the console, click Administration > ETL and System Tasks > ETL tasks.
2. In the Last exec time column corresponding to the ETL name, verify that the current date and time are displayed.

To verify that the k8s Prometheus data is refreshed:

1. In the console, click Workspace.
2. Expand (Domain name) > Systems > k8s Prometheus > Instances.
3. In the left pane, verify that the hierarchy displays the new and updated Prometheus instances.
4. Click a k8s Prometheus entity, and click the Metrics tab in the right pane.
5. Check if the Last Activity column in the Configuration metrics and Performance metrics tables displays the current date.

K8s Prometheus Workplace Entities

Entity Relationship

Hierarchy

The connector is able to replicate relationships and logical dependencies among these entities as they are found configured within the Kubernetes cluster.

In particular, the following structure is applied:

• a Kubernetes Cluster is attached to the root of the hierarchy
• each Kubernetes Cluster contains its own Nodes, Namespaces and Persistent Volumes
• each Kubernetes Namespace contains its own Controllers and Stand along pod workloads
• each Kubernetes Namespace contains persistent volume via persistent volume claim relationship
• each Kubernetes Controller contains it pod workloads, which is an aggregated entitiy that contains a group of pods that are running on the same controller

Hierarchy showing Cluster > Namespace > Deployments > Pod Workload

Hierarchy Showing Cluster > PVs & Nodes

Lookup Field Considerations

Tag Mapping (Optional)

Here’s an example for what tag looks like:

k8s Heapster to k8s Prometheus Migration

The “Moviri Integrator for BMC Helix Continuous Optimization – k8s Prometheus” supports a seamless transition from entities and metrics imported by the “Moviri Integrator forBMC Helix Continuous Optimization – k8s Heapster”. Please follow these steps to migrate between the two integrators:

1. Stop “Moviri Integrator for BMC Helix Continuous Optimization – k8s Heapster” ETL task.
2. Install and configure the “Moviri Integrator for BMC Helix Continuous Optimization – k8s Prometheus”, ensuring that the lookup is shared with the “Moviri Integrator for BMC Helix Continuous Optimization – k8s Heapster” ETL task.
3. Start “Moviri Integrator for BMC Helix Continuous Optimization – k8s Prometheus” ETL task.

Pod Optimization - Pod Workloads replace Pods

The “Moviri Integrator for BMC Helix Continuous Optimization – k8s Prometheus” introduce a new entity "pod workload" from v20.02.01. Pod Workload is an aggregated entity that aggregate a group of pods that are running on the same controller. pod workload is the direct child of the controller that the pods are running on. Pod workload will use the same name as the parent controller. Pods at the same time will be dropped.

Common Issues

Cause

Solve

Data Verification

The following sections provide some indications on how to verify on Prometheus if all the pre-requisites are in place before starting collecting data

Verify Prometheus Build information

Verify the Prometheus "Last successful configuration reload" (from Prometheus UI, check "Status > Runtime & Build Information")
If the "Last successful configuration reload" is reporting less then 3 days, ask the customer to evaluate the status of the integration in the next 2/3 days

Verify the Status of Prometheus Target Services

Verify the status of Prometheus Target (from the Prometheus UI, check "Status > Targets"

• Check the status of "node-exporter" (there should be 1 instance running for each node in the cluster)
• Check the status of "kube-state-metrics" (there should be at least 1 instance running)
• Check the status of "kubelet" (there should be at least 1 instance running for each node in the cluster)

Verify data availability in Prometheus Tables

Verify if the following Prometheus tables contain data (from the Prometheus Ul)

• "kube_pod_container_info" when missing Pod Workload, Controller, Namespace (but also Cluster and Node for Requests and Limits metrics)
• "kube_node info" when missing Node and Cluster metrics.