• Kubernetes versions 1.5 to 1.21
  • Kubernetes versions 1.16+ use TLS 1.3 by default and there is a known issue with the Java implementation. Please contact BMC support for help setting up connections to these versions.
• Rancher 1.6 to 2.3 (when managing a Kubernetes cluster version 1.5 to 1.21)
• Openshift 3.11 to 4.8
• Google Kubernetes Engine (GKE)
• Amazon Kubernetes Service (EKS)
• Azure Kubernetes Service (AKS)
• Pivotal Kubernetes Service (PKS)

• Prometheus 2.7 and onward

The integrator requires the Prometheus monitoring component to be correctly monitoring the various entities supported by the integration via the following services:

• kube-state-metrics
• node-exporter
• kubelet
• (if import jvm metrics) JMX exporter

The connector has also the option to access to the Kubernetes API. Access to Kubernetes API is not mandatory to configure the ETL, but it is strongly suggested. Without the access to Kubernetes API, the integrator will not be able to import the following information

• Persistent volumes capacity-relevant metrics

• Kubernetes Labels for replicaset and replicasetcontroller

When using Racher, please make sure not to have Rancher Monitoring enabled along side with kube-state-metrics. The Integrator does not support Rancher Monitoring.

Only one service monitoring Kubernetes components should be active at the same time.

OpenShift

OpenShift Container Platform Monitoring ships with a Prometheus instance for cluster monitoring and a central Alertmanager cluster. You can get the addresses for accessing Prometheus, Alertmanager, and Grafana web UIs by running:

$ oc -n openshift-monitoring get routes

NAME                HOST/PORT                                          ...
alertmanager-main   alertmanager-main-openshift-monitoring.apps.url.openshift.com ...

grafana             grafana-openshift-monitoring.apps.url.openshift.com           ...

prometheus-k8s      prometheus-k8s-openshift-monitoring.apps.url.openshift.com    ...

Or you can access from Openshift Interface under "Networking" → "Routes" tab or "Networking" → "Services" tab from the cluster console.

Make sure to append https:// to these addresses. You cannot access web UIs using unencrypted connection.

Authentication is performed against the OpenShift Container Platform identity and uses the same credentials or means of authentication as is used elsewhere in OpenShift Container Platform. You need to use a role that has read access to all namespaces, such as the cluster-monitoring-view cluster role. 

Rancher

When installing Prometheus from the Catalog Apps, the default configuration sets up a Layer 7 ingress using xip.io. From the Load Balancing tab, you can see the endpoint to access Prometheus.

Google Kubernetes Engine

When installing Prometheus from the Google Cloud using ingress, the Prometheus endpoint will be the type of load balancer or ingress. Access the endpoint on "Service & Ingress" type, and you can see the endpoint to access Prometheus.

Kubectl

Kubernetes commandline tool kubectl can also be used to access Prometheus API. using the following command, and replace the namespace with the namespace where Prometheus pod is running, and the external IP address will show the url of Prometheus

kubectl get service -n <namespace>

Other Distributions

Prometheus does not directly support authentication for connections to the Prometheus expression browser and HTTP API. If you would like to enforce basic authentication for those connections, Prometheus documentation recommends using Prometheus in conjunction with a reverse proxy and applying authentication at the proxy layer. Please refer to the official Prometheus documentation for configuring a NGINX reverse proxy with basic authentication.

To verify, execute the following command from one of the BMC Helix Continuous Optimization servers:

When authentication is not required

curl -k https://<prometheus_url>:<port>/api/v1/query?query=kube_node_info

When basic authentication is required

curl -k https://<prometheus_url>:<port>/api/v1/query?query=kube_node_info -u <username>

When bear token authentication is required

curl -k 'https://<prometheus_url>:<port>/api/v1/query?query=kube_node_info' -H 'Authorization: Bearer <token>'

When OAuth authentication is required, first you need a token from OAuth

curl -vk 'https://<oauth_url>:<port>/oauth/authorize?client_id=openshift-challenging-client&response_type=token -u <oauth_username> -H "X-CSRF-Token:xxx" -H "accept:text/plain"
*** OMITTED RESPONSE ***
< Location: https://<oauth_url>:<port>/oauth/token/implicit#access_token=BM0btgKP00TGPjLBBW-Y_Hb7A2wdSt99LAGtqh7QiJw&expires_in=86400&scope=user%3Afull&token_type=Bearer
*** OMITTED RESPONSE ***

We look at the Response Header "Location" for the access token generated. In the response example above, the token starts with BM0btg...

Then with the new access token you can follow the same steps as the bearer token and replace the "Token" with our new access token.

curl -k 'https://<prometheus_url>:<port>/api/v1/query?query=kube_node_info' -H 'Authorization: Bearer <OAuth Access Token>'

To access the Kubernetes API, the Kubernetes connector uses a Service Account. The authentication will be performed using the service account token. Additionally, in order to prevent accidental changes, the integrator service account will be granted read-only privileges and will be allowed to query a set of specific API endpoints. Here follows an example procedure to create the service account in a Kubernetes cluster using the kubectl CLI.

Create a Service Account

First of all, create the service account to be used by the Kubernetes connector:

$ kubectl create serviceaccount tsco

Then, describe the service account to discover which secret is associated to it:

$ kubectl describe serviceaccount tsco

Name: tsco
Namespace: default

Labels: <none>

Annotations: <none>

Image pull secrets: <none>

Mountable secrets: tsco-token-6x9vs

Tokens: tsco-token-6x9vs

Now, describe the secret to get the corresponding token:

kubectl describe secret tsco-token-6x9vs

Name: tsco-token-6x9vs

Namespace: default

Labels: <none>

Annotations: kubernetes.io/service-account.name=tsco

kubernetes.io/service-account.uid=07bca5e7-7c3e-11e7-87bc-42010a8e0002

Type: kubernetes.io/service-account-token

Data

====

ca.crt: 1025 bytes

namespace: 7 bytes

token: eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6InRzY28tdG9rZW4tNng5dnMiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoidHNjbyIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjA3YmNhNWU3LTdjM2UtMTFlNy04N2JjLTQyMDEwYThlMDAwMiIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0OnRzY28ifQ.tA6c9AsVJ0QKD0s-g9JoBdWDfhBvClJDiZGqDW6doS0rNZ5-dwXCJTss97PXfCxd5G8Q_nxg-elB8rV805K-j8ogf5Ykr-JLsAbl9OORRzcUCShYVF1r7O_-ikGg7abtIPh_mE5eAgHkJ1P6ODvaZG_0l1fak4BxZMTVfzzelvHpVlLpJZObd7eZOEtEEEkcAhZ2ajLQoxLucReG2A25_SrVJ-6c82BWBKQHcTBL9J2d0iHBHv-zjJzXHQ07F62vpc3Q6QI_rOvaJgaK2pMJYdQymFff8OfVMDQhp9LkOkxBPuJPmNHmHJxSvCcvpNtVMz-Hd495vruZFjtwYYygBQ

The token data ("eyJhb ... YygBQ") will be used by the Kubernetes integrator to authenticate against the API. Save the token as it will be required at the connector creation time.

Grant the Service Account Read-only Privileges

The following section outlines an example configuration on the Kubernetes cluster that is suggested in order to allow API access to the service account used by the integrator. We provide example configurations for the two most common authorization schemes used in Kubernetes clusters, namely RBAC (Role-Based Access Control) and ABAC (Attribute-Based Access Control). To identify which mode is configured in your Kubernetes cluster, please refer to the official project documentation: https://kubernetes.io/docs/admin/authorization/

RBAC Authorization

RBAC is the authorization mode enabled by default from Kubernetes 1.6 onward. To grant read-only privileges to the connector service account, a new cluster role is created. The new cluster role allows to grant specific read-only privileges to a set of API operations and entities to the connector.

kubectl create -f tsco-cluster-role.yml
kubectl create clusterrolebinding tsco-view-binding --clusterrole=tsco-cluster-role --serviceaccount=default:tsco

Here is an example policy file that can be used for this purpose:

$ cat tsco-cluster-role.yml

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1

metadata:

 namespace: default

 name: tsco-cluster-role

rules:

- apiGroups: [""]

 resources: ["pods", "nodes", "namespaces", "replicationcontrollers", "persistentvolumes", "resourcequotas", "limitranges", "persistentvolumeclaims"]

 verbs: ["get", "list"]

- apiGroups: ["apps"]

 resources: ["deployments", "replicasets", "statefulsets"]

 verbs: ["get", "list"]

- apiGroups: ["autoscaling"]

 resources: ["HorizontalPodAutoscalers"]

 verbs: ["get", "list"]

ABAC Authorization

ABAC authorization grants access rights to users and service accounts via policies that are configured in a policy file. Such file is then used by the Kubernetes API server via the startup parameter --authorization-policy-file.

In order to allow read-only access to the integrator service account, the following policy line need to be appended to the aforementioned policy file:

{"apiVersion": "abac.authorization.kubernetes.io/v1beta1", "kind": "Policy", "spec": {"user":"system:serviceaccount:default:tsco", "namespace": "*", "resource": "*", "apiGroup": "*", "readonly": true, "nonResourcePath": "*"}}

The apiserver will need to be restarted to pick up the new policy lines.

OAuth Authentication & Authorization (OpenShift Only)

When you implement OAuth Authentication, you cannot use a ServiceAccount type for authentication. You will have to set up a separate User Type and bind the ClusterRole to that User account similar to how we set up the ClusterRoleBinding for the TSCO ServiceAccount above in the RBAC Authorization section. See the tsco-cluster-role.yml file template above for an example.

kubectl create -f tsco-cluster-role.yml
kubectl create clusterrolebinding tsco-view-binding --clusterrole=tsco-cluster-role --user=tsco

Or if the tsco-view-binding exists already, then you will have to edit the tsco-view-binding ClusterRoleBinding:

kubectl edit clusterrolebinding tsco-view-binding

and add to the subjects section under the other kubernetes accounts:

- apiGroup: rbac.authorization.k8s.io
  kind: User
  name: tsco
  namespace: default

This User is from the Identity provider (LDAP/BasicAuth/etc...). If the user was newly created, you will need to log into the Openshift Console once beforehand with the new account for it to be completely registered with the system. In this example we created an user account called 'tsco'.

You can check what users are in the Kubernetes system with these commands:

kubectl get users
kubectl describe user <username>

$ curl -s https://<KUBERNETES_API_HOSTNAME>:<KUBERNETES_API_PORT>/api/v1/nodes -k -H "Authorization: Bearer <KUBERNETES_CONNECTOR_TOKEN>"

You should get a JSON document describing the nodes comprising the Kubernetes cluster.

{
 "kind": "NodeList",
 "apiVersion": "v1",
 "metadata": {
 "selfLink": "/api/v1/nodes",
 "resourceVersion": "37317619"
 },
 "items": [
 {
 "metadata": {

OAuth Authentication & Authorization (OpenShift Only)

When OAuth authentication is required, first you need a token from OAuth

curl -vk 'https://<oauth_url>:<port>/oauth/authorize?client_id=openshift-challenging-client&response_type=token -u <oauth_username> -H "X-CSRF-Token:xxx" -H "accept:text/plain"
*** OMITTED RESPONSE ***
> Location: https://<oauth_url>:<port>/oauth/token/implicit#access_token=BM0btgKP00TGPjLBBW-Y_Hb7A2wdSt99LAGtqh7QiJw&expires_in=86400&scope=user%3Afull&token_type=Bearer
*** OMITTED RESPONSE ***

We look at the Response Header "Location" for the access token generated. In the response example above, the token starts with BM0btg...

Then with the new access token you can follow the same steps above replacing the "Token" with our new access token.

$ curl -s https://<KUBERNETES_API_HOSTNAME>:<KUBERNETES_API_PORT>/api/v1/nodes -k -H "Authorization: Bearer <OAUTH_ACCESS_TOKEN>"

The TrustStore file was located in the <CPIT BASE>/jre/lib/security directory and called cacerts). These steps need to be completed by the CPIT user, or whatever user installed TSCO.

Step 1: Grab the cert files needed
    Download the OAuth or Prometheus Public Cert, run this command:
    echo -n | openssl s_client -connect <Openshift OAuth / Prometheus URL>:<Port> | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > /tmp/oauth.crt

  
   Download the Kubernetes API Public Cert, run this command:
    echo -n | openssl s_client -connect <Kubernetes API URL>:<KubeAPI Port> | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > /tmp/kubeapi.crt  

Step 2: Import the certs to the TSCO EE JRE cacerts TrustStore. Keytool is a bit wonky with relative paths, so I suggest to change directories to  <CPIT BASE>/jre/lib/security
    <CPIT BASE>/jre/bin/keytool -keystore cacerts -importcert -noprompt -trustcacerts -alias oauth -file /tmp/oauth.crt
  <CPIT BASE>/jre/bin/keytool -keystore cacerts -importcert -noprompt -trustcacerts -alias kubeapi -file /tmp/kubeapi.crt

    Default keytool TrustStore password is changeit

Optional: Remove the old certs
    keytool -keystore cacerts -delete -noprompt -alias oauth
  keytool -keystore cacerts -delete -noprompt -alias kubeapi

Optional: If the above alias' does not work you can list the certs in the TrustStore using this command:
    keytool -list -v -keystore cacerts

The connector is able to replicate relationships and logical dependencies among these entities as they are found configured within the Kubernetes cluster.

In particular, the following structure is applied:

• a Kubernetes Cluster is attached to the root of the hierarchy

• each Kubernetes Cluster contains its own Nodes, Namespaces and Persistent Volumes

• each Kubernetes Namespace contains its own Controllers and Stand along pod workloads

• each Kubernetes Namespace contains persistent volume via persistent volume claim relationship
• each Kubernetes Controller contains it pod workloads, which is an aggregated entitiy that contains a group of pods that are running on the same controller

The following table lists all the metrics that are imported by the "Moviri Integrations - Kubernetes (Prometheus)" integration. If the user selects not to import POD metrics, only metrics imported at POD level are not imported.

Here’s a snapshot for what tag looks like: