[Release-1.29] - K3s fails to start after running k3s certificate rotate-ca

[brandond]

Backport fix for K3s fails to start after running k3s certificate rotate-ca

• K3s fails to start after running k3s certificate rotate-ca #11014

[endawkins]

Validated on release-1.29 using commit 56a9685 | version v1.29

Environment Details:

Node(s) CPU architecture, OS, and Version:

Linux ip-172-31-4-254 5.14.21-150500.55.44-default #1 SMP PREEMPT_DYNAMIC Mon Jan 15 10:03:40 UTC 2024 (cc7d8b6) x86_64 x86_64 x86_64 GNU/Linux
NAME="SLES"
VERSION="15-SP5"
VERSION_ID="15.5"
PRETTY_NAME="SUSE Linux Enterprise Server 15 SP5"
ID="sles"
ID_LIKE="suse"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:suse:sles:15:sp5"
DOCUMENTATION_URL="https://documentation.suse.com/"

Cluster Configuration:

1 server (configuration does not matter)

Files:

• config.yaml

cluster-init: true
write-kubeconfig-mode: 644

• rotate-default-ca-certs.sh
  https://github.com/k3s-io/k3s/blob/release-1.28/contrib/util/rotate-default-ca-certs.sh

Steps:

1. Install K3s
2. Update Certificates using script
3. Rotate ca-certs k3s certificate rotate-ca
4. Restart k3s sudo systemctl restart k3s
5. Check status of k3s sudo systemctl status k3s

Reproduction of the Issue:
#11014 (comment)

Validation of the Issue:

- Observations:

$ k3s -v
k3s version v1.29.9+k3s-56a96850 (56a96850)
go version go1.22.6

$ ./rotate-default-ca-certs.sh
To update certificates, you may now run:
    k3s certificate rotate-ca --path=/var/lib/rancher/k3s/server/rotate-ca

$ k3s certificate rotate-ca --path=/var/lib/rancher/k3s/server/rotate-ca
certificates saved to datastore

$ sudo systemctl restart k3s
$ sudo systemctl status k3s
● k3s.service - Lightweight Kubernetes
     Loaded: loaded (/etc/systemd/system/k3s.service; enabled; vendor preset: disabled)
     Active: active (running) since Mon 2024-10-14 20:46:10 UTC; 1min 25s ago
       Docs: https://k3s.io
    Process: 22658 ExecStartPre=/bin/sh -xc ! /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service 2>/dev/null (code=exited, status=0/SUCCESS)
    Process: 22660 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)
    Process: 22661 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
   Main PID: 22662 (k3s-server)

$ kubectl get nodes,pods -A -o wide
NAME                                              STATUS   ROLES                       AGE    VERSION                INTERNAL-IP    EXTERNAL-IP    OS-IMAGE                              KERNEL-VERSION                 CONTAINER-RUNTIME
node/ip-172-31-4-254.us-east-2.compute.internal   Ready    control-plane,etcd,master   145m   v1.29.9+k3s-56a96850   172.31.4.254   [REDACTED]     SUSE Linux Enterprise Server 15 SP5   5.14.21-150500.55.44-default   containerd://1.7.22-k3s1

NAMESPACE           NAME                                              READY   STATUS      RESTARTS   AGE    IP           NODE                                         NOMINATED NODE   READINESS GATES
kube-system         pod/coredns-559656f558-n7f7d                      1/1     Running     0          145m   10.42.0.4    ip-172-31-4-254.us-east-2.compute.internal   <none>           <none>
kube-system         pod/helm-install-traefik-crd-h66qz                0/1     Completed   0          145m   <none>       ip-172-31-4-254.us-east-2.compute.internal   <none>           <none>
kube-system         pod/helm-install-traefik-dnqrv                    0/1     Completed   1          145m   <none>       ip-172-31-4-254.us-east-2.compute.internal   <none>           <none>
kube-system         pod/local-path-provisioner-7677785564-lz86n       1/1     Running     0          145m   10.42.0.5    ip-172-31-4-254.us-east-2.compute.internal   <none>           <none>
kube-system         pod/metrics-server-7cbbc464f4-9qpdv               1/1     Running     0          145m   10.42.0.6    ip-172-31-4-254.us-east-2.compute.internal   <none>           <none>
kube-system         pod/svclb-nginx-loadbalancer-svc-acac3cd7-czk9s   1/1     Running     0          140m   10.42.0.22   ip-172-31-4-254.us-east-2.compute.internal   <none>           <none>
kube-system         pod/svclb-traefik-71150afe-4glqf                  2/2     Running     0          145m   10.42.0.7    ip-172-31-4-254.us-east-2.compute.internal   <none>           <none>
kube-system         pod/traefik-6c7b69cd74-gx52m                      1/1     Running     0          145m   10.42.0.8    ip-172-31-4-254.us-east-2.compute.internal   <none>           <none>
test-ingressroute   pod/whoami-8c9864b56-8tvlc                        1/1     Running     0          140m   10.42.0.24   ip-172-31-4-254.us-east-2.compute.internal   <none>           <none>
test-ingressroute   pod/whoami-8c9864b56-wlm8c                        1/1     Running     0          140m   10.42.0.25   ip-172-31-4-254.us-east-2.compute.internal   <none>           <none>
test-loadbalancer   pod/test-loadbalancer-bcbd6588c-d6zhs             1/1     Running     0          140m   10.42.0.21   ip-172-31-4-254.us-east-2.compute.internal   <none>           <none>
test-loadbalancer   pod/test-loadbalancer-bcbd6588c-lmlqh             1/1     Running     0          140m   10.42.0.23   ip-172-31-4-254.us-east-2.compute.internal   <none>           <none>