Flannel with many nodes results in dropped routes

[caroline-suse-rancher]

Environmental Info:
RKE2 Version: 1.24.6

Describe the bug:

A bug has been identified in flannel that relates to environments on VXLAN that introduce many nodes. There appear to be missing routes. See the related flannel issue:

• Missing routes with many nodes on vxlan flannel-io/flannel#958 (comment)

Additional context / logs:

Related JIRA ticket:

• https://jira.suse.com/SURE-5710

[thomasferrandiz]

Jira link: https://jira.suse.com/browse/SURE-5710

Related PR: #3885

(Canal with updated version of flannel including retries for netlink operation)

[rancher-max]

Validated on release-1.24 branch commitid deb9e1f

Performed the same steps as mentioned in #3878 (comment). Note that I was not able to reproduce the issue, but validating the flannel version bump should be sufficient.

Results

• Canal chart using expected version:

$ helm ls -A | grep -i canal
rke2-canal  	kube-system	1       	2023-02-17 00:01:44.320978886 +0000 UTC	deployed	rke2-canal-v3.25.0-build2023020901	v3.25.0 

• Flannel version has the expected bump:

$ sudo /var/lib/rancher/rke2/bin/crictl --config /var/lib/rancher/rke2/agent/etc/crictl.yaml images | grep -E 'calico|flannel'
docker.io/rancher/hardened-calico               v3.25.0-build20230209           fc496b892f5dc       216MB
docker.io/rancher/hardened-flannel              v0.21.1-build20230208           b20f74859b4cf       80MB

• Canal using the flannel version as expected:

$ k describe -n kube-system pod/rke2-canal-w8m6s | grep -i image | grep -i flannel
    Image:         rancher/hardened-flannel:v0.21.1-build20230208
    Image ID:      docker.io/rancher/hardened-flannel@sha256:c3c05ac7fa7bc81b18bf77cdda33a7e300516743be9bcf93c0b0c0f722efd4de
  Normal  Pulling    63s   kubelet            Pulling image "rancher/hardened-flannel:v0.21.1-build20230208"
  Normal  Pulled     59s   kubelet            Successfully pulled image "rancher/hardened-flannel:v0.21.1-build20230208" in 3.915770408s

Also validated on release-1.23 branch commitid b902ac8

Performed the same steps as mentioned in #3878 (comment).

Results

• Canal chart using expected version:

$ helm ls -A | grep -i canal
rke2-canal         	kube-system	1       	2023-02-16 19:32:44.527552958 +0000 UTC	deployed	rke2-canal-v3.25.0-build2023020901          	v3.25.0 

• Flannel version has the expected bump:

$ sudo /var/lib/rancher/rke2/bin/crictl --config /var/lib/rancher/rke2/agent/etc/crictl.yaml images | grep -E 'calico|flannel'
docker.io/rancher/hardened-calico                               v3.25.0-build20230209           fc496b892f5dc       216MB
docker.io/rancher/hardened-flannel                              v0.21.1-build20230208           b20f74859b4cf       80MB

• Canal using the flannel version as expected:

$ k describe -n kube-system pod/rke2-canal-xdmmp | grep -i image | grep -i flannel
    Image:         rancher/hardened-flannel:v0.21.1-build20230208
    Image ID:      docker.io/rancher/hardened-flannel@sha256:c3c05ac7fa7bc81b18bf77cdda33a7e300516743be9bcf93c0b0c0f722efd4de