Managing worker nodes

When making changes to worker nodes, we recommend shifting workloads from them and redistributing the workloads to other healthy nodes in the cluster. This practice prevents outages. The best way to do this is to use the Kubernetes-included operations to drain and cordon the worker nodes.

Draining removes the pods from the targeted node and makes them available to the Kubernetes scheduler for rescheduling. Cordon prevents the node from receiving new pod assignments from the Kubernetes scheduler. After a node is drained and cordoned, you can safely perform maintenance activities.

To drain and cordon worker nodes

1. To list all the nodes in your cluster, run the following command:

   kubectl get nodes

   Identify the name of the node you wish to drain. 

2. To cordon a node, run the following command:

   kubectl cordon <node name>

3. To drain the node, run the following command:

   kubectl drain <node name>

   Important

   If a DaemonSet manages the pods, specify --ignore-daemonsets with kubectl to successfully drain the node:

   kubectl drain --ignore-daemonsets <node name>

   You must issue the kubectl drain command only to a single node at a time. However, you can run multiple kubectl drain commands for different nodes in parallel, in different terminals, or in the background.

   Be sure that your cluster has enough resources for the application to compensate for the number of nodes you remove during the draining process.

4. Perform the maintenance activities. For example, perform vMotion or kernel upgrade.

5. To uncordon a node, run the following command:

   kubectl uncordon <node name>

   When you uncordon the node, it is ready to accept workloads.