Kubernetes cluster (initiated with kubeadm) not starting after a reboot

Written by - 0 comments

Published on - last updated on February 15th 2023 - Listed in Kubernetes Linux Containers


In a lab environment I was working on a Kubernetes cluster (v1.25.1) which was started (initiated) using kubeadm. The Kubernetes cluster consisted of a single control plane (cp) node and a worker node. However after a reboot of the lab environment the cluster didn't come up anymore.

This blog post is about issues with kubelet related to enabled swap. Another possible reason could be missing Kernel modules (such as br_netfilter). Check out the follow-up article to find out how to do automatic modprobe (enable Linux Kernel modules) at boot time.

The kubelet service should be responsible to start up the cluster automatically after a reboot yet kubectl was unable to connect to the Kubernetes API server.

On the worker node the same issue happened, the kubelet service failed to start. Checking the logs revealed something interesting:

root@cka2:~# systemctl status kubelet
- kubelet.service - kubelet: The Kubernetes Node Agent
     Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/kubelet.service.d
             |-10-kubeadm.conf
     Active: activating (auto-restart) (Result: exit-code) since Wed 2022-11-02 07:45:13 CET; 3s ago
       Docs: https://kubernetes.io/docs/home/
    Process: 3324 ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS (code=exited, status=1/FAILURE)
   Main PID: 3324 (code=exited, status=1/FAILURE)

root@cka2:~# journalctl -fu kubelet
-- Logs begin at Mon 2022-06-20 16:00:33 CEST. --
Nov 02 07:45:34 cka2 kubelet[3377]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.27. Image garbage collector will get sandbox image information from CRI.
Nov 02 07:45:34 cka2 kubelet[3377]: I1102 07:45:34.229767    3377 server.go:413] "Kubelet version" kubeletVersion="v1.25.1"
Nov 02 07:45:34 cka2 kubelet[3377]: I1102 07:45:34.229829    3377 server.go:415] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 02 07:45:34 cka2 kubelet[3377]: I1102 07:45:34.229998    3377 server.go:825] "Client rotation is on, will bootstrap in background"
Nov 02 07:45:34 cka2 kubelet[3377]: I1102 07:45:34.231249    3377 certificate_store.go:130] Loading cert/key pair from "/var/lib/kubelet/pki/kubelet-client-current.pem".
Nov 02 07:45:34 cka2 kubelet[3377]: I1102 07:45:34.232066    3377 dynamic_cafile_content.go:157] "Starting controller" name="client-ca-bundle::/etc/kubernetes/pki/ca.crt"
Nov 02 07:45:34 cka2 kubelet[3377]: I1102 07:45:34.236688    3377 server.go:660] "--cgroups-per-qos enabled, but --cgroup-root was not specified.  defaulting to /"
Nov 02 07:45:34 cka2 kubelet[3377]: E1102 07:45:34.236945    3377 run.go:74] "command failed" err="failed to run Kubelet: running with swap on is not supported, please disable swap! or set --fail-swap-on flag to false. /proc/swaps contained: [Filename\t\t\t\tType\t\tSize\tUsed\tPriority /swapfile                               file\t\t1190340\t0\t-2]"
Nov 02 07:45:34 cka2 systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE
Nov 02 07:45:34 cka2 systemd[1]: kubelet.service: Failed with result 'exit-code'.

The reason might sound stupid but it's so easy to fix: swap needs to be disabled!

To do this, first correctly stop the kubelet service:

root@cka2:~# systemctl stop kubelet

Disable swap on all Kubernetes nodes:

root@cka2:~# swapoff -a

And then start the kubelet service (first on the cp node) again:

 root@cka2:~# systemctl start kubelet

The kubelet service is now successfully started and also the Kubernetes cluster is up again:

root@cka2:~# systemctl status kubelet
- kubelet.service - kubelet: The Kubernetes Node Agent
     Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/kubelet.service.d
             |-10-kubeadm.conf
     Active: active (running) since Wed 2022-11-02 07:45:59 CET; 14s ago
       Docs: https://kubernetes.io/docs/home/
   Main PID: 3446 (kubelet)
      Tasks: 16 (limit: 9416)
[...]

ckadm@cka1:~$ kubectl get node
NAME   STATUS   ROLES           AGE     VERSION
cka1   Ready    control-plane   5d18h   v1.25.1
cka2   Ready    <none>          5d17h   v1.25.1



Add a comment

Show form to leave a comment

Comments (newest first)

No comments yet.

RSS feed

Blog Tags:

  AWS   Android   Ansible   Apache   Apple   Atlassian   BSD   Backup   Bash   Bluecoat   CMS   Chef   Cloud   Coding   Consul   Containers   CouchDB   DB   DNS   Database   Databases   Docker   ELK   Elasticsearch   Filebeat   FreeBSD   Galera   Git   GlusterFS   Grafana   Graphics   HAProxy   HTML   Hacks   Hardware   Icinga   Influx   Internet   Java   KVM   Kibana   Kodi   Kubernetes   LVM   LXC   Linux   Logstash   Mac   Macintosh   Mail   MariaDB   Minio   MongoDB   Monitoring   Multimedia   MySQL   NFS   Nagios   Network   Nginx   OSSEC   OTRS   Office   PGSQL   PHP   Perl   Personal   PostgreSQL   Postgres   PowerDNS   Proxmox   Proxy   Python   Rancher   Rant   Redis   Roundcube   SSL   Samba   Seafile   Security   Shell   SmartOS   Solaris   Surveillance   Systemd   TLS   Tomcat   Ubuntu   Unix   VMWare   VMware   Varnish   Virtualization   Windows   Wireless   Wordpress   Wyse   ZFS   Zoneminder