Moviri Integrator for BMC Helix Capacity Optimization – k8s (Kubernetes) Prometheus



“Moviri Integrator for BMC Helix Capacity Optimization – k8s (Kubernetes) Prometheus” is an additional component of BMC Helix Capacity Optimization product. It allows extracting data from the Kubernetes cluster management system, a leading solution to manage cloud-native containerized environments.  Relevant capacity metrics are loaded into  BMC Helix Capacity Optimization, which provides advanced analytics over the extracted data in the form of an interactive dashboard, the Kubernetes View.

The integration supports the extraction of both performance and configuration data across different component of the Kubernetes system and can be configured via parameters that allow entity filtering and many other settings. Furthermore, the connector is able to replicate relationships and logical dependencies among entities such as clusters, nodes, namespaces, deployments, and pods.

The documentation is targeted at BMC Helix Capacity Optimization administrators, in charge of configuring and monitoring the integration between BMC Helix Capacity Optimization and Kubernetes.


This version of the connector compatible with BMC Helix Capacity Optimization 19.11 and onward.


If you used the  “Moviri Integrator for  BMC Helix Capacity Optimization– k8s Prometheus” before, please expect the following change from version 20.02.01:

  1. A new entity type "pod workload" will be imported to replace pods. If you imported pods before, all the Kubernetes pods will be gone and left in All systems and business drivers → "Unassigned".
  2. A new naming convention is implemented for Kubernetes Controller replicasets (the suffix of the name will be removed, eg: replicaset-abcd will be named as replicaset). All the old controller will be replaced by the new controller with the new name, and the old ones will be left in All systems and business drivers "Unassigned".



Step I. Complete the preconfiguration tasks

Step II. Configure the ETL

Step III. Run the ETL

Step IV. Verify the data collection

k8s Heapster to k8s Prometheus Migration

Pod Optimization - Pod workloads replace pods

Common issues

Step I. Complete the preconfiguration tasks

StepsDetails
Check the required API version is supported
 versions
  • Kubernetes versions 1.5 to 1.17
    • Kubernetes versions 1.16+ use TLS 1.3 by default and there is a known issue with the Java implementation. Please contact BMC support for help setting up connections to these versions.
  • Rancher 1.6 to 2.3 (when managing a Kubernetes cluster version 1.5 to 1.7)
  • Openshift 3.11 to 4.4
  • Google Kubernetes Engine (GKE)
  • Amazon Kubernetes Service (EKS)
  • Azure Kubernetes Service (AKS)
  • Pivotal Kubernetes Service (PKS)
  • Prometheus 2.7 and onward
Check supported versions of data source configuration
 data source version

The integrator requires the Prometheus monitoring component to be correctly monitoring the various entities supported by the integration via the following services:

  • kube-state-metrics
  • node-exporter
  • kubelet
  • (if import jvm metrics) JMX exporter

The connector has also the option to access to the Kubernetes API. Access to Kubernetes API is not mandatory to configure the ETL, but it is strongly suggested. Without the access to Kubernetes API, the integrator will not be able to import the following information

  • Persistent volumes capacity-relevant metrics

  • Kubernetes Labels for replicaset and replicasetcontroller

Generate access to Prometheus API

The access to Prometheus API depends on the Kubernetes distribution and the Prometheus configuration. The following sections describe the standard procedure for the supported platforms.

 Generate Credentials

OpenShift

OpenShift Container Platform Monitoring ships with a Prometheus instance for cluster monitoring and a central Alertmanager cluster. You can get the addresses for accessing Prometheus, Alertmanager, and Grafana web UIs by running:

$ oc -n openshift-monitoring get routes


NAME                HOST/PORT                                          ...
alertmanager-main   alertmanager-main-openshift-monitoring.apps.url.openshift.com ...

grafana             grafana-openshift-monitoring.apps.url.openshift.com           ...

prometheus-k8s      prometheus-k8s-openshift-monitoring.apps.url.openshift.com    ...

Or you can access from Openshift Interface under "Networking" → "Routes" tab or "Networking" → "Services" tab from the cluster console.

Make sure to append https:// to these addresses. You cannot access web UIs using unencrypted connection.

Authentication is performed against the OpenShift Container Platform identity and uses the same credentials or means of authentication as is used elsewhere in OpenShift Container Platform. You need to use a role that has read access to all namespaces, such as the cluster-monitoring-view cluster role. 

Rancher

When installing Prometheus from the Catalog Apps, the default configuration sets up a Layer 7 ingress using xip.io. From the Load Balancing tab, you can see the endpoint to access Prometheus.

Google Kubernetes Engine

When installing Prometheus from the Google Cloud using ingress, the Prometheus endpoint will be the type of load balancer or ingress. Access the endpoint on "Service & Ingress" type, and you can see the endpoint to access Prometheus.


Kubectl

Kubernetes commandline tool kubectl can also be used to access Prometheus API. using the following command, and replace the namespace with the namespace where Prometheus pod is running, and the external IP address will show the url of Prometheus

kubectl get service -n <namespace>


Other Distributions

Prometheus does not directly support authentication for connections to the Prometheus expression browser and HTTP API. If you would like to enforce basic authentication for those connections, Prometheus documentation recommends using Prometheus in conjunction with a reverse proxy and applying authentication at the proxy layer. Please refer to the official Prometheus documentation for configuring a NGINX reverse proxy with basic authentication.

Verify Prometheus API access
 Verify Access

To verify, execute the following command from one of the BMC Helix Capacity Optimization servers:

When authentication is not required

curl -k https://<prometheus_url>:<port>/api/v1/query?query=kube_node_info

When basic authentication is required

curl -k https://<prometheus_url>:<port>/api/v1/query?query=kube_node_info -u <username>

When bear token authentication is required

curl -k 'https://<prometheus_url>:<port>/api/v1/query?query=kube_node_info' -H 'Authorization: Bearer <token>'

When OAuth authentication is required, first you need a token from OAuth

curl -vk 'https://<oauth_url>:<port>/oauth/authorize?client_id=openshift-challenging-client&response_type=token -u <oauth_username> -H "X-CSRF-Token:xxx" -H "accept:text/plain"
*** OMITTED RESPONSE ***
< Location: https://<oauth_url>:<port>/oauth/token/implicit#access_token=BM0btgKP00TGPjLBBW-Y_Hb7A2wdSt99LAGtqh7QiJw&expires_in=86400&scope=user%3Afull&token_type=Bearer
*** OMITTED RESPONSE ***

We look at the Response Header "Location" for the access token generated. In the response example above, the token starts with BM0btg...

Then with the new access token you can follow the same steps as the bearer token and replace the "Token" with our new access token.

curl -k 'https://<prometheus_url>:<port>/api/v1/query?query=kube_node_info' -H 'Authorization: Bearer <OAuth Access Token>'
Generate Access to Kubernetes API (Optional)

Access to Kubernetes API is optional, but highly recommended. Follow the following steps for Kubernetes API access on Openshift and GKE:

 Kubernetes API

To access the Kubernetes API, the Kubernetes connector uses a Service Account. The authentication will be performed using the service account token. Additionally, in order to prevent accidental changes, the integrator service account will be granted read-only privileges and will be allowed to query a set of specific API endpoints. Here follows an example procedure to create the service account in a Kubernetes cluster using the kubectl CLI.

Create a Service Account

First of all, create the service account to be used by the Kubernetes connector:

$ kubectl create serviceaccount tsco

Then, describe the service account to discover which secret is associated to it:

$ kubectl describe serviceaccount tsco


Name: tsco
Namespace: default

Labels: <none>

Annotations: <none>

Image pull secrets: <none>

Mountable secrets: tsco-token-6x9vs

Tokens: tsco-token-6x9vs


Now, describe the secret to get the corresponding token:

kubectl describe secret tsco-token-6x9vs


Name: tsco-token-6x9vs

Namespace: default

Labels: <none>

Annotations: kubernetes.io/service-account.name=tsco

kubernetes.io/service-account.uid=07bca5e7-7c3e-11e7-87bc-42010a8e0002

Type: kubernetes.io/service-account-token

Data

====

ca.crt: 1025 bytes

namespace: 7 bytes

token: eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6InRzY28tdG9rZW4tNng5dnMiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoidHNjbyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjA3YmNhNWU3LTdjM2UtMTFlNy04N2JjLTQyMDEwYThlMDAwMiIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0OnRzY28ifQ.tA6c9AsVJ0QKD0s-g9JoBdWDfhBvClJDiZGqDW6doS0rNZ5-dwXCJTss97PXfCxd5G8Q_nxg-elB8rV805K-j8ogf5Ykr-JLsAbl9OORRzcUCShYVF1r7O_-ikGg7abtIPh_mE5eAgHkJ1P6ODvaZG_0l1fak4BxZMTVfzzelvHpVlLpJZObd7eZOEtEEEkcAhZ2ajLQoxLucReG2A25_SrVJ-6c82BWBKQHcTBL9J2d0iHBHv-zjJzXHQ07F62vpc3Q6QI_rOvaJgaK2pMJYdQymFff8OfVMDQhp9LkOkxBPuJPmNHmHJxSvCcvpNtVMz-Hd495vruZFjtwYYygBQ


The token data ("eyJhb ... YygBQ") will be used by the Kubernetes integrator to authenticate against the API. Save the token as it will be required at the connector creation time.

Grant the Service Account Read-only Privileges

The following section outlines an example configuration on the Kubernetes cluster that is suggested in order to allow API access to the service account used by the integrator. We provide example configurations for the two most common authorization schemes used in Kubernetes clusters, namely RBAC (Role-Based Access Control) and ABAC (Attribute-Based Access Control). To identify which mode is configured in your Kubernetes cluster, please refer to the official project documentation: https://kubernetes.io/docs/admin/authorization/

RBAC Authorization

RBAC is the authorization mode enabled by default from Kubernetes 1.6 onward. To grant read-only privileges to the connector service account, a new cluster role is created. The new cluster role allows to grant specific read-only privileges to a set of API operations and entities to the connector.

kubectl create -f tsco-cluster-role.yml
kubectl create clusterrolebinding tsco-view-binding --clusterrole=tsco-cluster-role --serviceaccount=default:tsco

Here is an example policy file that can be used for this purpose:

$ cat tsco-cluster-role.yml


kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1

metadata:

 namespace: default

 name: tsco-cluster-role

rules:

- apiGroups: [""]

 resources: ["pods", "nodes", "namespaces", "replicationcontrollers", "persistentvolumes", "resourcequotas", "limitranges", "persistentvolumeclaims"]

 verbs: ["get", "list"]

- apiGroups: ["apps"]

 resources: ["deployments", "replicasets", "statefulsets"]

 verbs: ["get", "list"]

- apiGroups: ["autoscaling"]

 resources: ["HorizontalPodAutoscalers"]

 verbs: ["get", "list"]


ABAC Authorization

ABAC authorization grants access rights to users and service accounts via policies that are configured in a policy file. Such file is then used by the Kubernetes API server via the startup parameter --authorization-policy-file.

In order to allow read-only access to the integrator service account, the following policy line need to be appended to the aforementioned policy file:

{"apiVersion": "abac.authorization.kubernetes.io/v1beta1", "kind": "Policy", "spec": {"user":"system:serviceaccount:default:tsco", "namespace": "*", "resource": "*", "apiGroup": "*", "readonly": true, "nonResourcePath": "*"}}

The apiserver will need to be restarted to pick up the new policy lines.


OAuth Authentication & Authorization (OpenShift Only)

When you implement OAuth Authentication, you cannot use a ServiceAccount type for authentication. You will have to set up a separate User Type and bind the ClusterRole to that User account similar to how we set up the ClusterRoleBinding for the TSCO ServiceAccount above in the RBAC Authorization section. See the tsco-cluster-role.yml file template above for an example.

kubectl create -f tsco-cluster-role.yml
kubectl create clusterrolebinding tsco-view-binding --clusterrole=tsco-cluster-role --user=tsco

Or if the tsco-view-binding exists already, then you will have to edit the tsco-view-binding ClusterRoleBinding:

kubectl edit clusterrolebinding tsco-view-binding

and add to the subjects section under the other kubernetes accounts:

- apiGroup: rbac.authorization.k8s.io
  kind: User
  name: tsco
  namespace: default

This User is from the Identity provider (LDAP/BasicAuth/etc...). If the user was newly created, you will need to log into the Openshift Console once beforehand with the new account for it to be completely registered with the system. In this example we created an user account called 'tsco'.

You can check what users are in the Kubernetes system with these commands:

kubectl get users
kubectl describe user <username>
Verify Kubernetes API access

To verify, execute the curl Linux command from one of the BMC Helix Capacity Optimization servers:

 Verify Access
$ curl -s https://<KUBERNETES_API_HOSTNAME>:<KUBERNETES_API_PORT>/api/v1/nodes -k -H "Authorization: Bearer <KUBERNETES_CONNECTOR_TOKEN>"

You should get a JSON document describing the nodes comprising the Kubernetes cluster.

{
 "kind": "NodeList",
 "apiVersion": "v1",
 "metadata": {
 "selfLink": "/api/v1/nodes",
 "resourceVersion": "37317619"
 },
 "items": [
 {
 "metadata": {

OAuth Authentication & Authorization (OpenShift Only)


When OAuth authentication is required, first you need a token from OAuth


curl -vk 'https://<oauth_url>:<port>/oauth/authorize?client_id=openshift-challenging-client&response_type=token -u <oauth_username> -H "X-CSRF-Token:xxx" -H "accept:text/plain"
*** OMITTED RESPONSE ***
> Location: https://<oauth_url>:<port>/oauth/token/implicit#access_token=BM0btgKP00TGPjLBBW-Y_Hb7A2wdSt99LAGtqh7QiJw&expires_in=86400&scope=user%3Afull&token_type=Bearer
*** OMITTED RESPONSE ***


We look at the Response Header "Location" for the access token generated. In the response example above, the token starts with BM0btg...


Then with the new access token you can follow the same steps above replacing the "Token" with our new access token.

$ curl -s https://<KUBERNETES_API_HOSTNAME>:<KUBERNETES_API_PORT>/api/v1/nodes -k -H "Authorization: Bearer <OAUTH_ACCESS_TOKEN>"
Kubernetes 1.16 and onward Workaround

For Kubernetes version 1.16 and onward, there's a known issue related to TLS v1.3 and BMC Helix Capacity Optimization Java version incompatibility. We provide a workaround to use the TLS v1.2 instead of default TLS v1.3 for connection.

We implemented a hidden property for using TLSv1.2 in order to work with Kubernetes 1.16 and onward. Add hidden property "prometheus.use.tlsv12" from ETL configuration, and set the value as "true".

If you encounter the erros as cannot connect, there's also PKIX issue need to be fixed. To fix the "PKIX path validation failed: java.security.cert.CertPathValidatorException: signature check failed" error follow these steps.

 fix the PKIX keystore error

The TrustStore file was located in the <CPIT BASE>/jre/lib/security directory and called cacerts). These steps need to be completed by the CPIT user, or whatever user installed TSCO.

Step 1: Grab the cert files needed
    Download the OAuth or Prometheus Public Cert, run this command:
    echo -n | openssl s_client -connect <Openshift OAuth / Prometheus URL>:<Port> | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > /tmp/oauth.crt

  
   Download the Kubernetes API Public Cert, run this command:
    echo -n | openssl s_client -connect <Kubernetes API URL>:<KubeAPI Port> | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > /tmp/kubeapi.crt  

Step 2: Import the certs to the TSCO EE JRE cacerts TrustStore. Keytool is a bit wonky with relative paths, so I suggest to change directories to  <CPIT BASE>/jre/lib/security
    <CPIT BASE>/jre/bin/keytool -keystore cacerts -importcert -noprompt -trustcacerts -alias oauth -file /tmp/oauth.crt
  <CPIT BASE>/jre/bin/keytool -keystore cacerts -importcert -noprompt -trustcacerts -alias kubeapi -file /tmp/kubeapi.crt


    Default keytool TrustStore password is changeit

Optional: Remove the old certs
    keytool -keystore cacerts -delete -noprompt -alias oauth
  keytool -keystore cacerts -delete -noprompt -alias kubeapi

Optional: If the above alias' does not work you can list the certs in the TrustStore using this command:
    keytool -list -v -keystore cacerts

Step II. Configure the ETL

A. Configuring the basic properties

Some of the basic properties display default values. You can modify these values if required.

To configure the basic properties:

  1. In the console, navigate to Administration ETL & System Tasks, and select ETL tasks.
  2. On the ETL tasks page, click Add > Add ETL. The Add ETL page displays the configuration properties. You must configure properties in the following tabs: Run configuration, Entity catalog, and Amazon Web Services Connection

  3. On the Run Configuration tab, select Moviri - k8s Prometheus Extractor from the ETL Module list. The name of the ETL is displayed in the ETL task name field. You can edit this field to customize the name.


  4. Click the Entity catalog tab, and select one of the following options:
    • Shared Entity Catalog:

      • From the Sharing with Entity Catalog list, select the entity catalog name that is shared between ETLs.
    • Private Entity Catalog: Select if this is the only ETL that extracts data from the k8s Prometheus resources.
  5. Click the Connection tab, and configure the following properties:


Property Name

Value Type

Required?

Default

Description

Prometheus – API URL

String

Yes


Prometheus API URL (http/https://hostname:port).

Port can be emitted.

Prometheus – API Version

String

Yes

v1

Prometheus API version, this should be the same as the Kubernetes API version if using any.

Prometheus – API Authentication Method

String

Yes


Prometheus API authentication method. There are four methods that are supported: Authentication Token (Bearer), Basic Authentication (username/password), None (no authentication), OpenShift OAuth (Username/Password).

Prometheus – Username

String

No


Prometheus API username if the Authentication method is set to Basic Authentication.

Prometheus – Password

String

No


Prometheus API password if the Authentication method is set to Basic Authentication.

Prometheus – API Authentication Token

String

No


Prometheus API Authentication Token (Bearer Token) if the Authentication method is set to Authentication Token.

OAuth - URLStringNo
URL for the OpenShift OAuth (http/https://hostname:port). Only available if Authentication Method is set to "OpenShift OAuth".

Port can be emitted.
OAuth - UsernameStringNo
OpenShift OAuth username if the Authentication Method is set to OpenShift OAuth.
OAuth - PasswordStringNo
OpenShift OAuth password if the Authentication Method is set to OpenShift OAuth.

Prometheus – Use Proxy Server

Boolean

No


If a proxy server is used when chose either Basic Authentication or None.

Proxy sever supports HTTP.

Proxy server only support Basic Authentication and None Authentication.

Prometheus - Proxy Server Host

String

No


Proxy server host name.

Prometheus - Proxy Server Port

Number

No


Proxy server port. Default 8080.

Prometheus - Proxy Username

String

No


Proxy server username

Prometheus - Proxy Password

String

No


Proxy server password

Use Kubernetes API

Boolean

Yes


If use Kubernetes API or not.

Kubernetes Host         

String

Yes

 

Kubernetes API server host name

For Openshift, use the Openshift console FQDN (e.g., console.ose.bmc.com).

Kubernetes API Port

Number

Yes

 

Kubernetes API server port

For Openshift, use the same port as the console (typically 8443).

Kubernetes API Protocol

String

Yes

 HTTPS

Kubernetes API protocol, "HTTPS" in most cases

Kubernetes Authentication Token

String

No

 

Token of the integrator service account (see data source configuration section).

If the Authentication Method is set to OpenShift OAuth, then you do not need to put a token in this field. Please make sure the User account has the right permissions specified in the data source configuration section for Kubernetes API access.

6. Click the Prometheus Extraction tab, and configure the following properties:

Property Name

Value Type

Required

Default

Description

Data Resolution

String

Yes


Data resolution for data to be pulled from Prometheus into TSCO. Default is set to 5 minutes. any value less than 5 minutes will be set to default 5 minutes.

Cluster NameStringNo
If Kubernetes API is not in use, cluster name must be specified.

Default Last Counter

String

Yes


Default earliest time the connector should be pulling data from in UTC. Format as YYYY-MM-DDTHH24:MI:SS.SSSZ, for example, 2019-01-01T19:00:00.000Z.

Maximum Hours to Extract

String

No

24

Maximum hours the connector can pull data from. If leave empty, using default 24 hours from default last counter.

Extract PODWorkload

BooleanYesNoIf wants to import podworkload
Do you want to pull cluster, namespace from API?BooleanNoYesIf pulling performance metrics from namespace, controller and cluster directly. Default is yes. This can be changed to no if the extraction from Prometheus API takes too long. If set to no: 1) the cluster performance metrics will be aggregated from nodes' performance metrics 2) If the extracting podworkload, then the namespace and container performance metrics will be aggregated from podworloads; If do not extract podworkload, namespace will be aggregated from controllers' metrics, while controllers' metrics will come from API directly.

Import by image metric on all entities?

StringNoYesIf want to import BYIMAGE metrics on cluster, namespace, controller and nodes
Import podworkload metric highmark counters?StringNoNoIf want to import Highmark metrics for CPU_USED_NUM, MEM_USED on podworkload for by container metrics

Import only the following tags (semi-colon separated list)

String

No


Import only tag types (Kubernetes Label key).

Specify the keys of the labels. They can be in the original format appears on Kubernetes API, or they can be using underscore (_) as delimiter. For example, node_role_kubernetes_io_compute and node-role.kubernetes.io/compute are equivalent and will be imported as node_role_kubernetes_io_compute.
Multiple tag types can be separated by semicolon (;).

The following image shows a Run Configuration example for the “Moviri Integrator for BMC Helix Capacity Optimization – k8s Prometheus":

7. (Optional) Enable TLS v1.2 for Kubernetes 1.16 and above

If you are using Kubernetes 1.16, or Openshift 4 and above, there's an incompatibility between Java and TLS v1.3. We are providing a workaround to use TLSv 1.2 for connection. To add the hidden property:

a. On ETL configuration page's very bottom, there's link to mamually modifying the ETL property:

b. On the manually editing property page, add the property: "prometheus.use.tlsv12"


c. Set the value as "true", and save the change

image.png

8. (Optional) Override the default values of the properties:


 Run configuration
PropertyDescription
Module selection

Select one of the following options:

  • Based on datasource: This is the default selection.
  • Based on Open ETL template: Select only if you want to collect data that is not supported by TrueSight Capacity Optimization.
Module descriptionA short description of the ETL module.
Execute in simulation modeBy default, the ETL execution in simulation mode is selected to validate connectivity with the data source, and to ensure that the ETL does not have any configuration issues. In the simulation mode, the ETL does not load data into the database. This option is useful when you want to test a new ETL task. To run the ETL in the production mode, select No.
BMC recommends that you run the ETL in the simulation mode after ETL configuration and then run it in the production mode.
 Object relationships
PropertyDescription
Associate new entities to

Specify the domain to which you want to add the entities created by the ETL.

Select one of the following options:

  • Existing domain: This option is selected by default. Select an existing domain from the Domain list. If the selected domain is already used by other hierarchy rules, select one of the following Domain conflict options:
    • Enrich domain tree: Select to create a new independent hierarchy rule for adding a new set of entities, relations, or both that are not defined by other ETLs.
    • ETL Migration: Select if the new ETL uses the same set of entities, relations, or both that are already defined by other ETLs.
  • New domain: Select a parent domain, and specify a name for your new domain.

By default, a new domain with the same ETL name is created for each ETL. When the ETL is created, a new hierarchy rule with the same name of the ETL task is automatically created in the active state. If you specify a different domain for the ETL, the hierarchy rule is updated automatically.

 ETL task properties
PropertyDescription
Task groupSelect a task group to classify the ETL.
Running on schedulerSelect one of the following schedulers for running the ETL:
  • Primary Scheduler: Runs on the Application Server.
  • Generic Scheduler: Runs on a separate computer.
  • Remote: Runs on remote computers.
Maximum execution time before warningIndicates the number of hours, minutes, or days for which the ETL must run before generating warnings or alerts, if any.
Frequency

Select one of the following frequencies to run the ETL:

  • Predefined: This is the default selection. Select a daily, weekly, or monthly frequency, and then select a time to start the ETL run accordingly.
  • Custom: Specify a custom frequency, select an appropriate unit of time, and then specify a day and a time to start the ETL run.

(Optional) B. Configuring the advanced properties


You can configure the advanced properties to change the way the ETL works or to collect additional metrics.


To configure the advanced properties:


  1. On the Add ETL page, click Advanced.
  2. Configure the following properties:

     Run configuration
    PropertyDescription
    Run configuration nameSpecify the name that you want to assign to this ETL task configuration. The default configuration name is displayed. You can use this name to differentiate between the run configuration settings of ETL tasks.
    Deploy statusSelect the deploy status for the ETL task. For example, you can initially select Test and change it to Production after verifying that the ETL run results are as expected.
    Log levelSpecify the level of details that you want to include in the ETL log file. Select one of the following options:
    • 1 - Light: Select to add the bare minimum activity logs to the log file.
    • 5 - Medium: Select to add the medium-detailed activity logs to the log file.
    • 10 - Verbose: Select to add detailed activity logs to the log file.

    Use log level 5 as a general practice. You can select log level 10 for debugging and troubleshooting purposes.

    Datasets

    Specify the datasets that you want to add to the ETL run configuration. The ETL collects data of metrics that are associated with these datasets.

    1. Click Edit.
    2. Select one (click) or more (shift+click) datasets from the Available datasets list and click >> to move them to the Selected datasets list.
    3. Click Apply.

    The ETL collects data of metrics associated with the datasets that are available in the Selected datasets list.

     Collection level
    PropertyDescription
    Metric profile selection

    Select the metric profile that the ETL must use. The ETL collects data for the group of metrics that is defined by the selected metric profile.

    • Use Global metric profile: This is selected by default. All the out-of-the-box ETLs use this profile.
    • Select a custom metric profile: Select the custom profile that you want to use from the Custom metric profile list. This list displays all the custom profiles that you have created.

    For more information about metric profiles, see Adding and managing metric profiles.
    Levels up to

    Specify the metric level that defines the number of metrics that can be imported into the database. The load on the database increases or decreases depending on the selected metric level.

    To learn more about metric levels, see Aging Class mapping.

    To learn more about metric levels, see Aging class mapping .

     Loader configuration
    PropertyDescription
    Empty dataset behaviorSpecify the action for the loader if it encounters an empty dataset:
    • Warn: Generate a warning about loading an empty dataset.
    • Ignore: Ignore the empty dataset and continue parsing.
    ETL log file nameThe name of the file that contains the ETL run log. The default value is: %BASE/log/%AYEAR%AMONTH%ADAY%AHOUR%MINUTE%TASKID
    Maximum number of rows for CSV outputA numeric value to limit the size of the output files.
    CSV loader output file nameThe name of the file that is generated by the CSV loader. The default value is: %BASE/output/%DSNAME%AYEAR%AMONTH%ADAY%AHOUR%ZPROG%DSID%TASKID
    Capacity Optimization loader output file name

    The name of the file that is generated by the BMC Helix Capacity Optimization loader. The default value is: %BASE/output/%DSNAME%AYEAR%AMONTH%ADAY%AHOUR%ZPROG%DSID%TASKID

    Detail mode
    Specify whether you want to collect raw data in addition to the standard data. Select one of the following options:
    • Standard: Data will be stored in the database in different tables at the following time granularities: Detail (configurable, by default: 5 minutes), Hourly, Daily, and Monthly.
    • Raw also: Data will be stored in the database in different tables at the following time granularities: Raw (as available from the original data source), Detail (configurable, by default: 5 minutes), Hourly, Daily, and Monthly.
    • Raw only: Data will be stored in the database in a table only at Raw granularity (as available from the original data source).
    For more information, see Accessing data using public views and Sizing and scalability considerations.

    For more information, see Accessing data using public views and Sizing and scalability considerations.

    Remove domain suffix from datasource name (Only for systems) Select True to remove the domain from the data source name. For example, server.domain.com will be saved as server. The default selection is False.
    Leave domain suffix to system name (Only for systems)Select True to keep the domain in the system name. For example: server.domain.com will be saved as is. The default selection is False.
    Update grouping object definition (Only for systems)Select True if you want the ETL to update the grouping object definition for a metric that is loaded by the ETL. The default selection is False.
    Skip entity creation (Only for ETL tasks sharing lookup with other tasks)Select True if you do not want this ETL to create an entity and discard data from its data source for entities not found in Capacity Optimization. It uses one of the other ETLs that share a lookup to create a new entity. The default selection is False.
     Scheduling options
    PropertyDescription
    Hour maskSpecify a value to run the task only during particular hours within a day. For example, 0 – 23 or 1, 3, 5 – 12.
    Day of week maskSelect the days so that the task can be run only on the selected days of the week. To avoid setting this filter, do not select any option for this field.
    Day of month maskSpecify a value to run the task only on the selected days of a month. For example, 5, 9, 18, 27 – 31.
    Apply mask validationSelect False to temporarily turn off the mask validation without removing any values. The default selection is True.
    Execute after timeSpecify a value in the hours:minutes format (for example, 05:00 or 16:00) to wait before the task is run. The task run begins only after the specified time is elapsed.
    EnqueueableSpecify whether you want to ignore the next run command or run it after the current task. Select one of the following options:
    • False: Ignores the next run command when a particular task is already running. This is the default selection.
    • True: Starts the next run command immediately after the current running task is completed.
    3.Click Save.

    The ETL tasks page shows the details of the newly configured Prometheus ETL:


Step III. Run the ETL

After you configure the ETL, you can run it to collect data. You can run the ETL in the following modes:

A. Simulation mode: Only validates connection to the data source, does not collect data. Use this mode when you want to run the ETL for the first time or after you make any changes to the ETL configuration.

B. Production mode: Collects data from the data source.

A. Running the ETL in the simulation mode

To run the ETL in the simulation mode:

  1. In the console, navigate to Administration ETL & System Tasks, and select ETL tasks.
  2. On the ETL tasks page, click the ETL. The ETL details are displayed.



  3. In the Run configurations table, click Edit  to modify the ETL configuration settings.
  4. On the Run configuration tab, ensure that the Execute in simulation mode option is set to Yes, and click Save.
  5. Click Run active configuration. A confirmation message about the ETL run job submission is displayed.
  6. On the ETL tasks page, check the ETL run status in the Last exit column.
    OK Indicates that the ETL ran without any error. You are ready to run the ETL in the production mode.
  7.  If the ETL run status is Warning, Error, or Failed:
    1. On the ETL tasks page, click  in the last column of the ETL name row.
    2. Check the log and reconfigure the ETL if required.
    3. Run the ETL again.
    4. Repeat these steps until the ETL run status changes to OK.

B. Running the ETL in the production mode

You can run the ETL manually when required or schedule it to run at a specified time.

Running the ETL manually

  1. On the ETL tasks page, click the ETL. The ETL details are displayed.
  2. In the Run configurations table, click Edit  to modify the ETL configuration settings. The Edit run configuration page is displayed.
  3. On the Run configuration tab, select No for the Execute in simulation mode option, and click Save.
  4. To run the ETL immediately, click Run active configuration. A confirmation message about the ETL run job submission is displayed.
    When the ETL is run, it collects data from the source and transfers it to the database.

Scheduling the ETL run

By default, the ETL is scheduled to run daily. You can customize this schedule by changing the frequency and period of running the ETL.

To configure the ETL run schedule:

  1. On the ETL tasks page, click the ETL, and click Edit Task . The ETL details are displayed.

  2. On the Edit task page, do the following, and click Save:

    • Specify a unique name and description for the ETL task.
    • In the Maximum execution time before warning field, specify the duration for which the ETL must run before generating warnings or alerts, if any.
    • Select a predefined or custom frequency for starting the ETL run. The default selection is Predefined.
    • Select the task group and the scheduler to which you want to assign the ETL task.
  3. Click Schedule. A message confirming the scheduling job submission is displayed.
    When the ETL runs as scheduled, it collects data from the source and transfers it to the database.

Step IV. Verify data collection

Verify that the ETL ran successfully and check whether the k8s Prometheus data is refreshed in the Workspace.

To verify whether the ETL ran successfully:

  1. In the console, click Administration > ETL and System Tasks > ETL tasks.
  2. In the Last exec time column corresponding to the ETL name, verify that the current date and time are displayed.
To verify that the k8s Prometheus data is refreshed:

  1. In the console, click Workspace.
  2. Expand (Domain name) > Systems > k8s Prometheus > Instances.
  3. In the left pane, verify that the hierarchy displays the new and updated Prometheus instances.
  4. Click a k8s Prometheus entity, and click the Metrics tab in the right pane.
  5. Check if the Last Activity column in the Configuration metrics and Performance metrics tables displays the current date.


k8s Prometheus Workspace EntityDetails

Entities

 Show Entities
TSCO EntitiesPrometheus Entity

Kubernetes Cluster

Cluster

Kubernetes Namespace

Namespace

Kubernetes Node

Node

Kubernetes Pod Workload

An aggregated group of pods running on the same controller; Stand along static pods

Kubernetes ControllerDaemonSet, ReplicaSet, StatefulSet, ReplicationController
Kubernetes Consistent VolumeConsistent Volume, Consistent Volume Claim

Hierarchy

 Show Hierarchy

The connector is able to replicate relationships and logical dependencies among these entities as they are found configured within the Kubernetes cluster.

In particular, the following structure is applied:

  • a Kubernetes Cluster is attached to the root of the hierarchy

  • each Kubernetes Cluster contains its own Nodes, Namespaces and Persistent Volumes

  • each Kubernetes Namespace contains its own Controllers and Stand along pod workloads

  • each Kubernetes Namespace contains persistent volume via persistent volume claim relationship
  • each Kubernetes Controller contains it pod workloads, which is an aggregated entitiy that contains a group of pods that are running on the same controller


Configuration and Performance metrics mapping

 Metrics Mapping

The following table lists all the metrics that are imported by the "Moviri Integrations - Kubernetes (Prometheus)" integration. If the user selects not to import POD metrics, only metrics imported at POD level are not imported.

Data Source

Data Source Entity Label

BMC BMC Helix Capacity Optimization Optimization Entity

BMC BMC Helix Capacity Optimization Metric

PrometheusClusterKubernetes - ClusterBYIMAGE_CPU_REQUEST
PrometheusClusterKubernetes - ClusterBYIMAGE_MEM_REQUEST
PrometheusClusterKubernetes - ClusterBYIMAGE_NUM
PrometheusClusterKubernetes - ClusterBYSTATUS_KPOD_NUM
PrometheusClusterKubernetes - ClusterCONTAINER_NUM
PrometheusClusterKubernetes - ClusterCONTROLLER_NUM
PrometheusClusterKubernetes - ClusterCPU_ALLOCATABLE
PrometheusClusterKubernetes - ClusterCPU_LIMIT
PrometheusClusterKubernetes - ClusterCPU_NUM
PrometheusClusterKubernetes - ClusterCPU_REQUEST
PrometheusClusterKubernetes - ClusterCPU_USED_NUM
PrometheusClusterKubernetes - ClusterCPU_UTIL
Prometheus ClusterKubernetes - ClusterJOB_NUM
PrometheusClusterKubernetes - ClusterKPOD_NUM
PrometheusClusterKubernetes - ClusterKPOD_NUM_MAX
PrometheusClusterKubernetes - ClusterKUBERNETES_VERSION
PrometheusClusterKubernetes - ClusterMEM_ACTIVE
PrometheusClusterKubernetes - ClusterMEM_KLIMIT
PrometheusClusterKubernetes - ClusterMEM_PAGE_MAJOR_FAULT_RATE
PrometheusClusterKubernetes - ClusterMEM_REQUEST
PrometheusClusterKubernetes - ClusterMEM_USED
PrometheusClusterKubernetes - ClusterMEM_UTIL
PrometheusClusterKubernetes - ClusterMEM_REAL_USED
PrometheusClusterKubernetes - ClusterMEMORY_ALLOCATABLE
PrometheusClusterKubernetes - ClusterSECRET_NUM
PrometheusClusterKubernetes - ClusterSERVICE_NUM
PrometheusClusterKubernetes - ClusterST_ALLOCATED
PrometheusClusterKubernetes - ClusterTOTAL_FS_FREE
PrometheusClusterKubernetes - ClusterTOTAL_FS_SIZE
PrometheusClusterKubernetes - ClusterTOTAL_FS_USED
PrometheusClusterKubernetes - ClusterTOTAL_FS_UTIL
PrometheusClusterKubernetes - ClusterTOTAL_REAL_MEM
PrometheusDaemonSet, ReplicaSet, StatefulSet, ReplicationControllerKubernetes - ControllerBYIMAGE_CPU_REQUEST
PrometheusDaemonSet, ReplicaSet, StatefulSet, ReplicationControllerKubernetes - ControllerBYIMAGE_MEM_REQUEST
PrometheusDaemonSet, ReplicaSet, StatefulSet, ReplicationControllerKubernetes - ControllerBYIMAGE_NUM
PrometheusDaemonSet, ReplicaSet, StatefulSet, ReplicationControllerKubernetes - ControllerBYSTATUS_KPOD_NUM
PrometheusDaemonSet, ReplicaSet, StatefulSet, ReplicationControllerKubernetes - ControllerCONTAINER_NUM
PrometheusDaemonSet, ReplicaSet, StatefulSet, ReplicationControllerKubernetes - ControllerCONTROLLER_TYPE
PrometheusDaemonSet, ReplicaSet, StatefulSet, ReplicationControllerKubernetes - ControllerCPU_LIMIT
PrometheusDaemonSet, ReplicaSet, StatefulSet, ReplicationControllerKubernetes - ControllerCPU_REQUEST
PrometheusDaemonSet, ReplicaSet, StatefulSet, ReplicationControllerKubernetes - ControllerCPU_USED_NUM
PrometheusDaemonSet, ReplicaSet, StatefulSet, ReplicationControllerKubernetes - ControllerCPU_UTIL
PrometheusDaemonSet, ReplicaSet, StatefulSet, ReplicationControllerKubernetes - ControllerCREATION_TIME
KUBERNETES API (if HPA configured)DaemonSet, ReplicaSet, StatefulSet, ReplicationControllerKubernetes - ControllerHPA_STATUS
PrometheusDaemonSet, ReplicaSet, StatefulSet, ReplicationControllerKubernetes - ControllerKPOD_NUM
PrometheusDaemonSet, ReplicaSet, StatefulSet, ReplicationControllerKubernetes - ControllerKPOD_REPLICA_UPTODATE_NUM
KUBERNETES API (if HPA configured)DaemonSet, ReplicaSet, StatefulSet, ReplicationControllerKubernetes - ControllerMAX_REPLICAS
PrometheusDaemonSet, ReplicaSet, StatefulSet, ReplicationControllerKubernetes - ControllerMEM_ACTIVE
PrometheusDaemonSet, ReplicaSet, StatefulSet, ReplicationControllerKubernetes - ControllerMEM_KLIMIT
PrometheusDaemonSet, ReplicaSet, StatefulSet, ReplicationControllerKubernetes - ControllerMEM_REQUEST
PrometheusDaemonSet, ReplicaSet, StatefulSet, ReplicationControllerKubernetes - ControllerMEM_USED
KUBERNETES API (if HPA configured)DaemonSet, ReplicaSet, StatefulSet, ReplicationControllerKubernetes - ControllerMIN_REPLICAS
PrometheusDaemonSet, ReplicaSet, StatefulSet, ReplicationControllerKubernetes - ControllerNET_BIT_RATE
PrometheusDaemonSet, ReplicaSet, StatefulSet, ReplicationControllerKubernetes - ControllerNET_IN_BIT_RATE
PrometheusDaemonSet, ReplicaSet, StatefulSet, ReplicationControllerKubernetes - ControllerNET_IN_BYTE_RATE
PrometheusDaemonSet, ReplicaSet, StatefulSet, ReplicationControllerKubernetes - ControllerNET_IN_ERROR_RATE
PrometheusDaemonSet, ReplicaSet, StatefulSet, ReplicationControllerKubernetes - ControllerNET_OUT_BIT_RATE
PrometheusDaemonSet, ReplicaSet, StatefulSet, ReplicationControllerKubernetes - ControllerNET_OUT_BYTE_RATE
PrometheusDaemonSet, ReplicaSet, StatefulSet, ReplicationControllerKubernetes - ControllerNET_OUT_ERROR_RATE
PrometheusNamespaceKubernetes - NamespaceBYIMAGE_CPU_REQUEST
PrometheusNamespaceKubernetes - NamespaceBYIMAGE_MEM_REQUEST
PrometheusNamespaceKubernetes - NamespaceBYIMAGE_NUM
PrometheusNamespaceKubernetes - NamespaceBYSTATUS_KPOD_NUM
PrometheusNamespaceKubernetes - NamespaceCONTAINER_NUM
PrometheusNamespaceKubernetes - NamespaceCPU_LIMIT
PrometheusNamespaceKubernetes - NamespaceCPU_LIMIT_MAX
PrometheusNamespaceKubernetes - NamespaceCPU_REQUEST
PrometheusNamespaceKubernetes - NamespaceCPU_REQUEST_MAX
PrometheusNamespaceKubernetes - NamespaceCPU_USED_NUM
PrometheusNamespaceKubernetes - NamespaceCPU_UTIL
PrometheusNamespaceKubernetes - NamespaceCREATION_TIME
PrometheusNamespaceKubernetes - NamespaceKPOD_NUM
PrometheusNamespaceKubernetes - NamespaceKPOD_NUM_MAX
PrometheusNamespaceKubernetes - NamespaceMEM_ACTIVE
PrometheusNamespaceKubernetes - NamespaceMEM_KLIMIT
PrometheusNamespaceKubernetes - NamespaceMEM_LIMIT_MAX
PrometheusNamespaceKubernetes - NamespaceMEM_REQUEST
PrometheusNamespaceKubernetes - NamespaceMEM_REQUEST_MAX
PrometheusNamespaceKubernetes - NamespaceMEM_USED
PrometheusNamespaceKubernetes - NamespaceNET_BIT_RATE
PrometheusNamespaceKubernetes - NamespaceNET_IN_BIT_RATE
PrometheusNamespaceKubernetes - NamespaceNET_IN_BYTE_RATE
PrometheusNamespaceKubernetes - NamespaceNET_IN_ERROR_RATE
PrometheusNamespaceKubernetes - NamespaceNET_OUT_BIT_RATE
PrometheusNamespaceKubernetes - NamespaceNET_OUT_BYTE_RATE
PrometheusNamespaceKubernetes - NamespaceNET_OUT_ERROR_RATE
PrometheusNodeKubernetes - NodeBYIMAGE_CPU_REQUEST
PrometheusNodeKubernetes - NodeBYIMAGE_MEM_REQUEST
PrometheusNodeKubernetes - NodeBYIMAGE_NUM
PrometheusNodeKubernetes - NodeBYSTATUS_KPOD_NUM
PrometheusNodeKubernetes - NodeCONTAINER_NUM
PrometheusNodeKubernetes - NodeCPU_ALLOCATABLE
PrometheusNodeKubernetes - NodeCPU_LIMIT
PrometheusNodeKubernetes - NodeCPU_NUM
PrometheusNodeKubernetes - NodeCPU_REQUEST
PrometheusNodeKubernetes - NodeCPU_USED_NUM
PrometheusNodeKubernetes - NodeCPU_UTIL
PrometheusNodeKubernetes - NodeCREATION_TIME
PrometheusNodeKubernetes - NodeKPOD_NUM
PrometheusNodeKubernetes - NodeKPOD_NUM_MAX
PrometheusNodeKubernetes - NodeKUBERNETES_VERSION
PrometheusNodeKubernetes - NodeMEM_ACTIVE
PrometheusNodeKubernetes - NodeMEM_KLIMIT
PrometheusNodeKubernetes - NodeMEM_PAGE_MAJOR_FAULT_RATE
PrometheusNodeKubernetes - NodeMEM_REAL_USED
PrometheusNodeKubernetes - NodeMEM_REQUEST
PrometheusNodeKubernetes - NodeMEM_USED
PrometheusNodeKubernetes - NodeMEM_UTIL
PrometheusNodeKubernetes - NodeMEMORY_ALLOCATABLE
PrometheusNodeKubernetes - NodeNET_BIT_RATE
PrometheusNodeKubernetes - NodeNET_IN_BIT_RATE
PrometheusNodeKubernetes - NodeNET_IN_BYTE_RATE
PrometheusNodeKubernetes - NodeNET_IN_ERROR_RATE
PrometheusNodeKubernetes - NodeNET_OUT_BIT_RATE
PrometheusNodeKubernetes - NodeNET_OUT_BYTE_RATE
PrometheusNodeKubernetes - NodeNET_OUT_ERROR_RATE
PrometheusNodeKubernetes - NodeOS_TYPE
PrometheusNodeKubernetes - NodeTOTAL_FS_FREE
PrometheusNodeKubernetes - NodeTOTAL_FS_SIZE
PrometheusNodeKubernetes - NodeTOTAL_FS_USED
PrometheusNodeKubernetes - NodeTOTAL_FS_UTIL
PrometheusNodeKubernetes - NodeTOTAL_REAL_MEM
PrometheusNodeKubernetes - NodeUPTIME
Kubernetes APIPersistent VolumeKubernetes - Persistent VolumeCREATION_TIME
PrometheusPersistent VolumeKubernetes - Persistent VolumeST_ALLOCATED
Kubernetes APIPersistent VolumeKubernetes - Persistent VolumeST_PATH
Kubernetes APIPersistent VolumeKubernetes - Persistent VolumeST_SIZE
Kubernetes APIPersistent VolumeKubernetes - Persistent VolumeST_TYPE
Prometheus Pod, Container, ImageKubernetes - Pod WorkloadBYCONT_CPU_LIMIT
Prometheus Pod, Container, ImageKubernetes - Pod WorkloadBYCONT_CPU_REQUEST
Prometheus Pod, Container, ImageKubernetes - Pod WorkloadBYCONT_CPU_USED_NUM
Prometheus Pod, Container, ImageKubernetes - Pod WorkloadBYCONT_CPU_USED_NUM_HM
Prometheus Pod, Container, ImageKubernetes - Pod WorkloadBYCONT_IMAGE_CPU_LIMIT
Prometheus Pod, Container, ImageKubernetes - Pod WorkloadBYCONT_IMAGE_CPU_REQUEST
Prometheus Pod, Container, ImageKubernetes - Pod WorkloadBYCONT_IMAGE_CPU_USED_NUM
Prometheus Pod, Container, ImageKubernetes - Pod WorkloadBYCONT_IMAGE_KPOD_NUM
Prometheus Pod, Container, ImageKubernetes - Pod WorkloadBYCONT_IMAGE_MEM_ACTIVE
Prometheus Pod, Container, ImageKubernetes - Pod WorkloadBYCONT_IMAGE_MEM_KLIMIT
Prometheus Pod, Container, ImageKubernetes - Pod WorkloadBYCONT_IMAGE_MEM_REQUEST
Prometheus Pod, Container, ImageKubernetes - Pod WorkloadBYCONT_IMAGE_MEM_USED
Prometheus Pod, Container, ImageKubernetes - Pod WorkloadBYCONT_IMAGE_RESTART_COUNT
Prometheus Pod, Container, ImageKubernetes - Pod WorkloadBYCONT_KPOD_NUM
Prometheus Pod, Container, ImageKubernetes - Pod WorkloadBYCONT_MEM_ACTIVE
Prometheus Pod, Container, ImageKubernetes - Pod WorkloadBYCONT_MEM_KLIMIT
Prometheus Pod, Container, ImageKubernetes - Pod WorkloadBYCONT_MEM_REQUEST
Prometheus Pod, Container, ImageKubernetes - Pod WorkloadBYCONT_MEM_USED
Prometheus Pod, Container, ImageKubernetes - Pod WorkloadBYCONT_MEM_USED_HM
Prometheus Pod, Container, ImageKubernetes - Pod WorkloadBYCONT_RESTART_COUNT
Prometheus Pod, Container, ImageKubernetes - Pod WorkloadCPU_LIMIT
Prometheus Pod, Container, ImageKubernetes - Pod WorkloadCPU_REQUEST
Prometheus Pod, Container, ImageKubernetes - Pod WorkloadCPU_USED_NUM
Prometheus Pod (if JMX-Exporter enabled)Kubernetes - Pod WorkloadHEAPMEM_MAX
Prometheus Pod, Container, ImageKubernetes - Pod WorkloadKPOD_NUM
Prometheus Pod, Container, ImageKubernetes - Pod WorkloadMEM_ACTIVE
Prometheus Pod, Container, ImageKubernetes - Pod WorkloadMEM_KLIMIT
Prometheus Pod, Container, ImageKubernetes - Pod WorkloadMEM_REQUEST
Prometheus Pod, Container, ImageKubernetes - Pod WorkloadMEM_USED
Prometheus Pod (if JMX-Exporter enabled)Kubernetes - Pod WorkloadNONHEAPMEM_MAX
Prometheus Pod, Container, ImageKubernetes - Pod WorkloadRESTART_COUNT

For a detailed mapping between Prometheus API queries and each metirc, check out here(exclude for Derived metircs nor Kubernetes API)


Derived Metrics
 derived metrics

Data Source Entity

Data Source Metric

BMC BMC Helix Capacity Optimization Metric

BMC BMC Helix Capacity Optimization Entity

Kubernetes - ClusterCPU_USED_NUM / CPU_LIMITCPU_UTIL_LIMITKubernetes - Cluster
Kubernetes - ClusterCPU_USED_NUM / CPU_REQUESTCPU_UTIL_REQUESTKubernetes - Cluster
Kubernetes - ClusterMEM_USED / MEM_KLIMITMEM_UTIL_LIMITKubernetes - Cluster
Kubernetes - ClusterMEM_USED / MEM_REQUESTMEM_UTIL_REQUESTKubernetes – Cluster
Kubernetes - ClusterST_SIZEST_SIZEKubernetes - Cluster
Kubernetes - ClusterMEM_ACTIVE / TOTAL_REAL_MEMMEM_REAL_UTILKubernetes - Cluster
Kubernetes - ControllerCPU_USED_NUM / CPU_LIMITCPU_UTIL_LIMITKubernetes - Controller
Kubernetes - ControllerCPU_USED_NUM / CPU_REQUESTCPU_UTIL_REQUESTKubernetes - Controller
Kubernetes - ControllerMEM_USED / MEM_KLIMITMEM_UTIL_LIMITKubernetes - Controller
Kubernetes - ControllerMEM_USED / MEM_REQUESTMEM_UTIL_REQUESTKubernetes - Controller
Kubernetes - NamespaceCPU_USED_NUM / CPU_LIMITCPU_UTIL_LIMITKubernetes - Namespace
Kubernetes - NamespaceCPU_USED_NUM / CPU_REQUESTCPU_UTIL_REQUESTKubernetes - Namespace
Kubernetes – NamespaceMEM_USED / MEM_KLIMITMEM_UTIL_LIMITKubernetes - Namespace
Kubernetes - NamespaceMEM_USED / MEM_REQUESTMEM_UTIL_REQUESTKubernetes - Namespace
Kubernetes - NodeCPU_USED_NUM / CPU_LIMITCPU_UTIL_LIMITKubernetes - Node
Kubernetes - NodeCPU_USED_NUM / CPU_REQUESTCPU_UTIL_REQUESTKubernetes - Node
Kubernetes – NodeMEM_USED / MEM_KLIMITMEM_UTIL_LIMITKubernetes - Node
Kubernetes - NodeMEM_USED / MEM_REQUESTMEM_UTIL_REQUESTKubernetes - Node
Kubernetes - NodeTAG "node_role_kubernetes_io"SERVER_TYPEKubernetes - Node
Kubernetes - NodeMEM_ACTIVE / TOTAL_REAL_MEMMEM_REAL_UTILKubernetes - Node
Kubernetes - Pod WorkloadBYCONT_IMAGE_CPU_USED_NUM / BYCONT_IMAGE_CPU_LIMITBYCONT_CPU_UTIL_LIMITKubernetes - Pod Workload
Kubernetes - Pod WorkloadBYCONT_IMAGE_CPU_USED_NUM / BYCONT_IMAGE_CPU_REQUESTBYCONT_CPU_UTIL_REQUESTKubernetes - Pod Workload
Kubernetes - Pod WorkloadBYCONT_IMAGE_CPU_USED_NUM / BYCONT_IMAGE_CPU_LIMITBYCONT_IMAGE_CPU_UTIL_LIMITKubernetes - Pod Workload
Kubernetes - Pod WorkloadBYCONT_IMAGE_CPU_USED_NUM / BYCONT_IMAGE_CPU_REQUESTBYCONT_IMAGE_CPU_UTIL_REQUESTKubernetes - Pod Workload
Kubernetes - Pod WorkloadBYCONT_IMAGE_MEM_USED / BYCONT_IMAGE_MEM_KLIMITBYCONT_IMAGE_MEM_UTIL_LIMITKubernetes - Pod Workload
Kubernetes - Pod WorkloadBYCONT_IMAGE_MEM_USED / BYCONT_IMAGE_MEM_REQUESTBYCONT_IMAGE_MEM_UTIL_REQUESTKubernetes - Pod Workload
Kubernetes - Pod WorkloadBYCONT_IMAGE_MEM_USED / BYCONT_IMAGE_MEM_KLIMITBYCONT_MEM_UTIL_LIMITKubernetes - Pod Workload
Kubernetes - Pod WorkloadBYCONT_IMAGE_MEM_USED / BYCONT_IMAGE_MEM_REQUESTBYCONT_MEM_UTIL_REQUESTKubernetes - Pod Workload
Kubernetes - Pod WorkloadCPU_USED_NUM / CPU_LIMITCPU_UTIL_LIMITKubernetes - Pod Workload
Kubernetes - Pod WorkloadCPU_USED_NUM / CPU_REQUESTCPU_UTIL_REQUESTKubernetes - Pod Workload
Kubernetes - Pod WorkloadMEM_USED / MEM_KLIMITMEM_UTIL_LIMITKubernetes - Pod Workload
Kubernetes - Pod WorkloadMEM_USED / MEM_REQUESTMEM_UTIL_REQUESTKubernetes - Pod Workload

Lookup Field Considerations
 Lookup Consideration

Entity Type

Strong Lookup Field

Others

Kubernetes - Cluster

KUBE_CLUSTER&&KUBE_TYPE


Kubernetes - Namespace

KUBE_CLUSTER&&KUBE_TYPE&&KUBE_NS_NAME


Kuberneteds - Node

KUBE_CLUSTER&&KUBE_TYPE&&HOSTNAME&&NAME

_COMPATIBILITY_

Kubernetes - Controller

KUBE_CLUSTER&&KUBE_TYPE&&KUBE_NS_NAME&&KUBE_DP_NAME


Kubernetes - Pod Workload

KUBE_CLUSTER&&KUBE_TYPE&&KUBE_NS_NAME&&KUBE_WL_NAME&&KUBE_DP_NAME&&KUBE_DP_TYPE


Kubernetes - Persistent Volume

KUBE_CLUSTER&&KUBE_TYPE&&KUBE_PV_NAME


 

Tag Mapping (Optional)

The “Moviri Integrator for BMC Helix Capacity Optimization – k8s Prometheus” supports Kubernetes Labels as Tags inBMC Helix Capacity Optimization.
The keys in labels are imported as Tag types, and the values in labels are imported as Tag values. The integrator will replace special word delimiter appears in Kubernetes label key as underscore (_) in Tag type.
In particular, the entities we import labels are:

 Show Tags

Data Source Entity

Data Source


BMC TrueSight Capacity Optimization Entity

Prometheus

DAEMONSET

Kubernetes - Controller

Prometheus

STATEFULSET

Kubernetes - Controller

Kubernetes API

REPLICATSET

Kubernetes - Controller

Kubernetes API

REPLICATIONCONTROLLER

Kubernetes - Controller

Prometheus

POD WORKLOAD

Kubernetes - Pod Workload

Prometheus

NODE

Kubernetes - Node

Prometheus

NAMESPACE

Kubernetes - Namespace

Prometheus

PERSISTENT VOLUME

Kubernetes - Persistent Volume

Here’s a snapshot for what tag looks like:



k8s Heapster to k8s Prometheus Migration

The “Moviri Integrator for BMC Helix Capacity Optimization – k8s Prometheus” supports a seamless transition from entities and metrics imported by the “Moviri Integrator forBMC Helix Capacity Optimization – k8s Heapster”. Please follow these steps to migrate between the two integrators:

 Kubernetes ETL Integeration
  1. Stop “Moviri Integrator for BMC Helix Capacity Optimization – k8s Heapster” ETL task.
  2. Install and configure the “Moviri Integrator for BMC Helix Capacity Optimization – k8s Prometheus”, ensuring that the lookup is shared with the “Moviri Integrator for BMC Helix Capacity Optimization – k8s Heapster” ETL task.
  3. Start “Moviri Integrator for BMC Helix Capacity Optimization – k8s Prometheus” ETL task.


Pod Optimization - Pod Workloads replace Pods


The “Moviri Integrator for BMC Helix Capacity Optimization – k8s Prometheus” introduce a new entity "pod workload" from v20.02.01. Pod Workload is an aggregated entity that aggregate a group of pods that are running on the same controller. pod workload is the direct child of the controller that the pods are running on. Pod workload will use the same name as the parent controller. Pods at the same time will be dropped. 


 Kubernetes ETL Integeration

If you used the  “Moviri Integrator for  BMC Helix Capacity Optimization– k8s Prometheus” and imported pods before, once you upgrade to this version, all the Kubernetes pods will be gone and left in All systems and business drivers ->

"Unassigned".

Pod Workload
EntityPod workload is an aggregated entity, that represents all the pods that are running on the same controller. Pod workload will use the same name as the controller that all the pods are running on.
HierarchyPod Workload is direct child of the controller that the aggregated pods are running on. For stand along pods, the pod workload is exactly the same as stand along pods themselves, and is children of the namespace.
Metrics Meaning

Global Metrics: provides multiple statistics values (avg, max, min, sum) for the last hour, representing max, min, sum and average value of the average of all pods running on that controller.

BYCONT metrics: provdes multiple statistics value (avg, max, min, sum) for the last hour, representing max, min, sum and average value of the average of per container per image running on that controller.

BYCONT Highmar Counters: BYCONT_CPU_USED_NUM_HM and BYCONT_MEM_USED_HM provide multiple statistics value on highmark counters based on 1 minute resolution. Max value is 95th percentile of the max container's the 1m resolution data; Avg value is the 90th percentile of the max container's the 1m resolution data; MIn value is the 75th percentile of the max container's the max container's 1m resolution data.; SUM value is the 95th percentile max 1 min resolution data summing all containers on that controller.

BYCONT_IMAGE metrics: provides multiple statistics value (avg, max, min, sum) for the last hour, representing max, min, sum and average of average value of per container per image running on that controller. Container name and image name as the subentity name, using "::" connecting (eg. controllername::imagename)

Here are some screenshots of the hierarchy and metric


Common Issues



Error Messages / BehaviorsCauseSolve
Query erros HTTP 422 Unprocessable Entity

You will see these errors shows sometimes. The number of this error messages can vary a lot for each run.

This is usually caused by Prometheus rebuilding or restarting. Right after the Prometheus's rebuilding or reloading, there are couple of days you will see this error showing. They usually goes away organically as the Prometheus running more stable.

They usually goes away organically as the Prometheus running more stable.
Prometheus is running fine but no data is pulledThis usually caused by the last counter is set too far from today's date. Prometheus has a data retention periods which has a default value 15 days, and it can be configured. If the ETL is set to extracting data passed the data retention period, there's not gonna be any data. Prometheus's status page will show the data retention value in "storage retention" field.Modify the default last counter to a more recent date.
504 Gateway Timeouts

These 504 Timeout query error (server didn't respondd in time) is related to the route timeout is being used on Openshift. This can be configured on a route-to-route basis. For example, the Prometheus route can be increased to the 2min timeout that is also configured on the Prometheus backend. Please follow this link to understand what is the configured timeout and how can it be increased
https://docs.openshift.com/container-platform/4.6/networking/routes/route-configuration.html

Increase timeout period from Openshift side


Was this page helpful? Yes No Submitting... Thank you

Comments