Collecting Kubernetes logs

Kubernetes is a widely-used solution for container management that helps you run container applications on a large scale. With Kubernetes, you can manage the lifecycle of containers that are deployed in pods and are highly distributed. You need to proactively monitor and debug such vast and distributed environments, where errors can occur at multiple levels—container, node, or cluster. Ensure optimum system performance by monitoring the logs generated at all levels.

To monitor logs, you need to collect logs from the Kubernetes cluster. You can use the Kubernetes connector to collect logs from all pods that are running on a node. The Kubernetes connector offers the following advantages:

  • Connector deployment by using a Helm package
    The supported Helm version is 3.2.1 and later.
  • Log collection based on the agent-and-aggregator architecture
    • An agent is a log collector that collects logs from different nodes.
    • An aggregator enriches the logs with filtering and parsing and sends the logs to .
  • Log filtering at the agent and the aggregator level
  • Log parsing at the aggregator level
  • Log collection for Kubelet and Kubernetes audit logs
  • No downtime for updating any attributes or filtering logic through the helm upgrade command

The following image shows how logs are collected from a Kubernetes cluster:

KubernetesLogCollection.png

Supported clusters

supports the following versions of Kubernetes and OpenShift clusters:

  • Kubernetes cluster
    • Client Version: v1.21 and v1.23
    • Server Version: v1.21
  • OpenShift cluster
    • Client version: v1.23
    • Server version: v1.25
       

The following image provides an overview of the steps that you need to perform to collect Kubernetes logs:

Kubernetes_Log_Collection_process_Overview.png

Before you begin

Ensure that you have the following permissions:

  • Permission to create, update, and delete the namespace, daemonset, configmap, serviceaccount, cluster role, StatefullSet, and custom resources
  • Cluster role binding and list permission for all configurations
  • Permission to run, install, update, and delete commands for Helm

To collect Kubernetes logs

  1. Click the Collection  menu and select Kubernetes.

  2. In the Connector Name field, enter a unique name for the connector.

    Important

    If you are migrating the log collection from BMC Helix Developer Tools, ensure that the connector name is not the same as the integration name in BMC Helix Developer Tools.

  3. In the Connector Type field, select Helm.
  4. In the Agent Configuration section, click Configure.
  5. In the Customize Logs Data dialog box, perform the following steps:
    1. In the Container Platform Format list, select a container platform format.
      Docker and CRIO are the supported formats
    2. (Optional) Click Multiline Parser and select one or more parsers for the agent.
    3. (Optional) Click Enable Kubelet Logs to collect the Kubelet logs.
    4. (Optional) Click Enable Audit Logs and provide the location where the Kubernetes audit logs are stored.
    5. Click Save.
  6. In the Agent Configuration section, select a filtering rule or create a new filtering rule.
    Select the collection type as Kubernetes and the configuration type as Agent from the list.
    It is optional to provide filtering at the agent level.
  7. In the Aggregator Configuration section, click Configure. 
  8. In the Customize Aggregator Logs Data dialog box, perform the following steps:
    1. (Optional) In the Tags field, enter the tags to identify logs from a cluster or node.

    2. In the PVC StorageClass field, enter a valid StorageClass name to automatically create the PersistentVolume (PV) and PersistentVolumeClaim (PVC) storage requests.
      Use the following command to obtain the storage class details:

      kubectl get sc

    3. In the Watched Namespaces field, enter the namespace for which you want to collect logs.

      Important

      • To add multiple namespaces, ensure that you enter comma-separated values. For Example: default,kube-system.
      • If you have enabled Kubelet logs, audit logs, or both, you must include the kube-system namespace in the watched namespace. 
    4. (Optional) If you want to limit the rate at which logs are collected from a namespace, click Rate Limiter and add the following information:
      • Namespace
      • The maximum number of events that you want to be ingested for the namespace

      • The duration for which you want to restrict the event ingestion for the namespace

        Important

        You can click Plus Icon.pngto add information for another namespace. However, the rate at which logs are collected from a namespace is configured for each Fluentd instance. That means, if you add another namespace, the system considers it as a separate Fluentd instance. 

    5. Click Save.
  9. In the Aggregator Configuration section, select a parsing rule or create a new parsing rule.
    It is optional to provide parsing at the aggregator level
  10. Select a filtering rule or create a new filtering rule.
    Select the configuration type as Kubernetes and the configuration type as Aggregator.
    It is optional to provide filtering at the aggregator level.
  11. (For Openshift only) If you are configuring the Kubernetes connector in the Openshift environment, perform the following steps:
    1. Add the following service accounts to the privileged Security Context Constraints (SCC):
      • oc adm policy add-scc-to-user privileged -z fluent-bit -n bmc-k8s-logs
      • oc adm policy add-scc-to-user privileged -z fluentd -n bmc-k8s-logs
      • oc adm policy add-scc-to-user privileged -z fluent-operator -n bmc-k8s-logs

    2. In the values.yaml file, go to the Fluentbit section, and add the following value under the securityContext entry:

      privileged: true

      Use the following images for reference:
      Before adding the securityContext entry:
      before_scc.png
      After adding the securityContext entry:
      after_scc.png

  12. In the Download and Configure section, click Download to download the Helm package.

  13. Create the bmc-k8s-logs namespace by using the following command:

    kubectl create ns bmc-k8s-logs

    Important

    Ensure that the namespace is not restricted and is privileged.

  14. Create the image pull secret by using the following command:

    kubectl create secret docker-registry <image-pull-secret-name> --docker-server=containers.bmc.com --docker-username=<username> --docker-password=<password> --docker-email=<email-    id> -n bmc-k8s-logs
  15. In the Image pull secrets for Docker Registry field, add the < image-pull-secret-name > secret.
  16. Create the integration and download the BMC Kubernetes Helm configuration file.
  17. (Optional) If you want to add an additional connector node, perform the following actions:
    1. In the values.yaml file, go to the fluentd: section.
    2. Change the value of the replicas: property.
      The default value is 2. Change it to the number of nodes that you want to add.
  18. Copy the downloaded Helm package and the configuration YAML file on the Kubernetes controller host of the cluster.
  19. Perform the following steps to deploy the Helm package:
    1. Navigate to the folder where the connector helm package is extracted and copy the downloaded configuration YAML file.

    2. Install the Helm package by using the following command:

      helm install fluent-operator . --create-namespace -n bmc-k8s-logs -f /values.yaml

    3. Verify that the pods are running in the bmc-k8s-logs namespace by using the following command:

      kubectl get po -n bmc-k8s-logs

       

    4. If the agents or aggregators are not running, run the following command:

      helm upgrade fluent-operator . -n bmc-k8s-logs

      If you cannot upgrade artifacts by using Helm upgrade, see Troubleshooting-log-collection-from-Kubernetes.
       

  20.   Verify that the connector is configured correctly by using the following commands:

    kubectl get po -n bmc-k8s-logs

You can view the connector that you created on the Collection > Connectors page.

To view the agent and aggregator status

  1. Navigate to the Collection > Connectors page.
  2. Click the Actions menu for the enhanced Kubernetes connector that you installed and configured.
    ActionsMenuK8Connector.png
  3. Click View Agents/Aggregators.
    Use the Connector Agents/Aggregators page to view the status and host names of all agents and aggregators.

To migrate to the enhanced Kubernetes connector

The enhanced Kubernetes connector with the agent-and-aggregator architecture is available from release 24.1. If you are using the old Kubernetes connector, we recommend that you migrate to the enhanced connector by performing the following steps:

  1. Install and configure the enhanced Kubernetes connector. 
  2. Delete the existing connector by performing the following steps:

    1. Stop the daemonset by running the following command:

      kubectl delete daemonset bmc-daemonset -n bmc-logging

    2. Delete all the configurations by running the command:

      kubectl delete -f bmc-logging-config-<connector_name>.yaml
      Command output
      namespace "bmc-logging" deleted
      serviceaccount "bmc-service-account" deleted
      clusterrole.rbac.authorization.k8s.io "bmc-cluster-role" deleted
      clusterrolebinding.rbac.authorization.k8s.io "bmc-cluster-role-binding" deleted
      configmap "bmc-config-map" deleted
      configmap "bmc-mek-config-map" deleted
      daemonset.apps "bmc-daemonset" deleted
    3. In , click Collection and select Connectors.
    4. Search for the existing connector name that you want to delete.
    5. Click the Actions menu and select ForceDelete..
  3. On the Collection > Connectors page, verify that the status of the connector that you installed and configured is Active.

To verify the log collection

  1. Navigate to the Collection > Connectors page and verify that the status of the connector that you installed and configured is Active.
  2. Navigate to Explorer > Discover.
  3. Search the logs by using the tags that you assigned to the logs in the collection configuration.
    The tags that you added to the Tags  field while configuring the connector are available in the logs in the bmc_tags field.

To stop the log collection

  1. Stop the daemonset by running the following command:

    kubectl delete daemonset bmc-daemonset -n bmc-logging

  2. Delete all the configurations by running the command:

    kubectl delete -f bmc-logging-config-<connector_name>.yaml
    Command output
    namespace "bmc-logging" deleted
    serviceaccount "bmc-service-account" deleted
    clusterrole.rbac.authorization.k8s.io "bmc-cluster-role" deleted
    clusterrolebinding.rbac.authorization.k8s.io "bmc-cluster-role-binding" deleted
    configmap "bmc-config-map" deleted
    configmap "bmc-mek-config-map" deleted
    daemonset.apps "bmc-daemonset" deleted
  3. In , click Collection and select Connectors.
  4. Search for the enhanced Kubernetes connector that you configured.
  5. Click the Actions menu and select ForceDelete.

 

Tip: For faster searching, add an asterisk to the end of your partial query. Example: cert*