Collecting Kubernetes logs

Kubernetes is a widely used solution for container management that helps you run container applications on a large scale. It manages the lifecycle of containers deployed in pods and is highly distributed. Such a vast and distributed environment requires you to proactively monitor it and debug errors. These errors can be at multiple levels—container, node, or cluster. By monitor the logs generated at these levels, you ensure the best performance of the containers that host multiple applications. 

Collect logs from a Kubernetes cluster by using BMC Logging Daemonset. You download the daemonset yml and they get associated with the nodes that you configure. At the node level, you get logs from all pods running on the node.

The following image shows how logs are collected from a Kubernetes cluster:

The process to collect Kubernetes logs differs from other log collections in BMC Helix Log Analytics. The following image provides an overview of the steps that you need to perform to collect Kubernetes logs:

Before you begin

Ensure that you have the create permission for namespace, daemonset, configmap, serviceaccount, cluster role, and cluster role binding and list permission for all configurations. 

To collect Kubernetes logs

1. Click the Collection menu and select Kubernetes.

2. Enter the connector name.
   Use this name to identify the connector on the Connectors page.

   Important

   If you are migrating log collection from BMC Helix Developer Tools, ensure that the connector name is not same as the integration name in BMC Helix Developer Tools.

3. In the Customize Entity Configuration field, click Configure and enter the following information:
   1. In the Tags field, enter the tags to identify logs from a cluster or node.
   2. Collect Kubernetes metadata that is present in the logs by selecting the Include Kubernetes metadata tags check box and then selecting the data that you want to collect with the logs.
   3. Filter namespaces for collection by selecting the Add a Namespace/Service Filter check box.
      By default, logs from all namespaces and services of a cluster are collected. In this case, ensure that your Kubernetes cluster has sufficient resources like memory, CPU, and so on.
   4. In the Namespace and Service Name fields, enter the namespace and services names for which logs are collected.
   5. From the Format list, select Json.

1. 1. In the Time Key and Time Format fields, the key or field in which the time value is present in the logs and the time format are displayed; change these values, if required.
   2. Click Save.

2. In the Download and Configure section, download the docker connector image by clicking Download.
   The name of the downloaded file is tdc-connector-base-d0ccf25-<buildno>.tar.gz.

   Important

   Repeat the steps only if you are collecting logs from other clusters. Also, repeat these steps after the connector image is updated by BMC.

3. Upload the downloaded connector image to your public or private docker repository by performing the following actions:
   1. Upload the docker connector image file to a docker-enabled virtual machine that has access to your docker repository (private or public).
   2. Ensure that you have read and write permissions on the connector image file.
   3. Create an image file from the tar.gz file, and run the following command:
      docker load <tdc-connector-base-d0ccf25-<buildno>.tar.gz
   4. Verify whether the image is created by running the following command:
      docker imagesThe command output displays the list of images that are present in the docker. Ensure that the uploaded image file is present in the list. 
   5. Tag the connector image by running the following command:
      docker tag <image_name or image_name> <docker registry path>
      For example: docker tag <tad-connector-base-d0ccf25>-<buildno.> <mydockerregistryhost>:<port>/images/custom/tdc-connector-base-d0ccf25-<buildno>
   6. Push the tagged connector image to the docker repository by running the following command:
      docker image push <mydockerregistryhost>:<port>/images/custom/tdc-connector-base-d0ccf25-<buildno>
   7. Copy the docker registry URL of the connector image path.
   8. Verify whether the URL is correct by running the following command:
      docker pull <docker registry URL>
      
   9. Open the configurations that are set up for the log collection and paste the URL in the Docker Registry Path field. 
4. Install the connector on all nodes in a cluster by performing the following actions:
   1. Download the .yaml file that installs the connector on all nodes in a cluster by clicking Create & Download.
      The collection configurations on this page are saved and the bmc-logging-config-<connector name>.yaml file is downloaded.
   2. If you have not provided a value in the Docker Registry Path field, in the containers : env : image path, replace the PARAMETERS.docker_container_registry_path string with the docker registry URL.

   3. Copy the .yaml file to your kubectl host and run the following command:
      kubectl apply -f apply bmc-logging-config-<connector name>.yaml
      After the command runs successfully, a new namespace (bmc-logging) and other required configurations are created.
      
      

      Command output

      namespace/bmc-logging created
      serviceaccount/bmc-service-account created
      clusterrole.rbac.authorization.k8s.io/bmc-cluster-role created
      clusterrolebinding.rbac.authorization.k8s.io/bmc-cluster-role-binding created
      configmap/bmc-config-map created
      configmap/bmc-mek-config-map created
      daemonset.apps/bmc-daemonset created

5. Verify that all the required configurations are created correctly by running the following command:
   kubectl get pods -n bmc-logging
   The saved collection and connector configurations are shown on the Connecotrs page.
   

To verify log collection

1. Select Explorer > Discover.
2. Search the logs by using the tags that you assigned to the logs in the collection configuration.
   The tags that you add to the Tags field are present in the logs in the bmc_tags field.

To stop collection

1. Stop the daemonset by running the following command:
   kubectl delete daemonset bmc-daemonset -n bmc-logging

2. Delete all the configurations by running the command:
   kubectl delete -f bmc-logging-config-<connector_name>.yaml
   

   Command output

   namespace "bmc-logging" deleted
   serviceaccount "bmc-service-account" deleted
   clusterrole.rbac.authorization.k8s.io "bmc-cluster-role" deleted
   clusterrolebinding.rbac.authorization.k8s.io "bmc-cluster-role-binding" deleted
   configmap "bmc-config-map" deleted
   configmap "bmc-mek-config-map" deleted
   daemonset.apps "bmc-daemonset" deleted

3. In BMC Helix Log Analytics, click Collection and select Connectors.
4. Search for the connector name that you had entered while configuring Kubernetes log collection.
5. Click the Actions menu and select ForceDelete. 

Learn more

Read the following blog to learn how you can enhance observability by using the Kubernetes logs that you collect Kubernetes observability with logs