Collecting logs from Kubernetes

Kubernetes is a widely used solution for container management that helps you to run container applications at a large scale. It manages the lifecycle of containers deployed in pods and is highly distributed. Such a vast and distributed environment requires you to proactively monitor it and debug errors. These errors can be at multiple levels—container, node, or cluster. Monitor the logs generated at these levels and use them to ensure best performance of the containers that host multiple applications. 

Collect logs from a Kubernetes cluster by using BMC Logging Daemonset. You can collect logs from all nodes within a cluster. At the node level, you get logs from all pods running on the node. 

The following image shows how logs are collected from a Kubernetes cluster:

The following image provides an overview of the steps that you need to perform to collect Kubernetes logs:

Before you begin

Ensure that you have the create permission for namespace, daemonset, configmap, serviceaccount, cluster role, and cluster rolebinding and list permission for all configurations. 

To collect logs from Kubernetes

1. Set up collection details by performing the following actions:
   1. Click the Collection menu and select Create Integration.
      The Available Configurations tab is displayed.
   2. On the Kubernetes tile, click Configure.
   3. On the Configure Integration page, in the Integration Name field, enter an integration name.
   4. In the Customize Entity Configuration field, click Configure.
   5. In the Tags field, enter the tags to identify logs from a cluster or node.
   6. Collect Kubernetes metadata present in the logs, select the Include Kubernetes metadata tags check box and select the data that you want to collect with the logs.
   7. Filter namespaces for collection, select the Add a Namespace/Service Filter check box.
      By default, logs from all namespaces and services of a cluster are collected. In this case, ensure that your Kubernetes cluster has sufficient resources like memory, CPU, and so on.
   8. In the Namespace and Service Name fields, enter the namespace and services names for which logs are collected.
   9. From the Format list, select Json.
   10. In the Time Key and Time Format fields, the key or field in which time value is present in the logs and the time format are displayed; change, if required.
   11. Click Save.
2. In the Download and Configure section, download the docker connector image by clicking Download.
   The name of the downloaded file is tdc-connector-base-d0ccf25-<buildno>.tar.gz.

3. Upload the downloaded connector image to your public or private docker repository by performing the following actions:
   

   Important

   Only repeat the steps if you are configuring integration to collect logs from other clusters. Also, repeat these steps when the connector image is updated by BMC.

   
   

   1. Upload the docker connector image file to a docker-enabled virtual machine that has access to your docker repository (private or public).
   2. Ensure that you have read and write permissions on the connector image file.
   3. Create an image file from the tar.gz file, run the following command:
      docker load < tdc-connector-base-d0ccf25-<buildno>.tar.gz
   4. Verify whether the image is created, run the following command:
      docker images
The command output displays the list of images that are present in the docker. Ensure that the uploaded image file is present in the list. 
   5. Tag the connector image, run the following command:
      docker tag <image_name or image_name> <docker registry path>
      For example, docker tag <tad-connector-base-d0ccf25>-<buildno.> <mydockerregistryhost>:<port>/images/custom/tdc-connector-base-d0ccf25-<buildno>
   6. Push the tagged connector image to the docker repository by running following command:
      docker image push <mydockerregistryhost>:<port>/images/custom/tdc-connector-base-d0ccf25-<buildno>
   7. Copy the docker registry URL of the connector image path.
   8. Verify that the URL is correct by running the following command:
      docker pull <docker registry URL>

   9. Open the configurations that are set up for the log collection and paste the URL in the Docker Registry Path field. 
4. Install the connector on all nodes in a cluster by performing the following actions:
   1. Download the .yaml file that installs the connector on all nodes in a cluster, click Create & Download.
      The integration configurations on this page are saved and the bmc-logging-config-<integration name>-k8s-connector.yaml file is downloaded.
   2. If you have not provided a value in the Docker Registry Path field, in the containers : env : image path, replace the PARAMETERS.docker_container_registry_path string with the docker registry URL.

   3. Copy the .yaml file to your kubectl host and run the following command:
      kubectl apply -f apply bmc-logging-config.yaml
      After the command runs successfully, a new namespace (bmc-logging) and other required configurations are created. 
      

       Command output

      namespace/bmc-logging created
serviceaccount/bmc-service-account created
clusterrole.rbac.authorization.k8s.io/bmc-cluster-role created
clusterrolebinding.rbac.authorization.k8s.io/bmc-cluster-role-binding created
configmap/bmc-config-map created
configmap/bmc-mek-config-map created
daemonset.apps/bmc-daemonset created

5. Verify that all the required configurations are created correctly, run the following command:
   kubectl get pods -n bmc-logging



To verify log collection

To verify whether log collection has started, select Explorer > Discover. Search the logs by using the tags that you assigned to the logs in the integration configuration. The tags that you add to the Tags field are present in the logs in the bmc_tags field.

The bmc_integration_name field in the collected logs contain the Integration Name field that you provide on the configurations page. You can use the field to narrow your search results.

To stop the collection

1. Stop the daemonset, run the following command:
   kubectl delete daemonset bmc-daemonset -n bmc-logging

2. Delete all the configurations by running the command:
   kubectl delete -f bmc-logging-config-<integration_name>-k8s-connector.yaml


    Command output

   namespace "bmc-logging" deleted
   serviceaccount "bmc-service-account" deleted
   clusterrole.rbac.authorization.k8s.io "bmc-cluster-role" deleted
   clusterrolebinding.rbac.authorization.k8s.io "bmc-cluster-role-binding" deleted
   configmap "bmc-config-map" deleted
   configmap "bmc-mek-config-map" deleted
   daemonset.apps "bmc-daemonset" deleted

3. On the Confiured Integrations tab, delete the integration.
   
   You can delete an integration when its status is either Configured or Disconnected. You can edit an integration when it is in Active status. The Enable and Disable options are not supported for the Kubernetes integration.

Related topic

Troubleshooting log collection from Kubernetes