HAProxy Ingress continuously reloading: tcp-services (configmap)

[bootc]

Description of the problem

Pods come and go in my cluster quite a lot, as one namespace in particular is used to run CI jobs for some of our projects. HAProxy Ingress notices these Pods and their associated Secrets coming and going, but notices non-existent changes to the tcp-services ConfigMap, triggering an unnecessary reload of HAProxy.

Expected behavior

The Ingress controller should see that no changes are actually needed and not reload HAProxy.

Steps to reproduce the problem

1. Configure a TCP services ConfigMap using controller.tcp in the Helm chart, for example.
2. Delete or create some Pods in the cluster.
3. dynupdate.go:104] need to reload due to config changes: [tcp-services (configmap)]

Environment information

HAProxy Ingress version: v0.14.2

Command-line options:

--configmap=tcl-ingress/haproxy-ingress
--ingress-class=haproxy
--master-socket=/var/run/haproxy/master.sock
--sort-backends
--tcp-services-configmap=tcl-ingress/haproxy-ingress-tcp
--ignore-ingress-without-class=true
--publish-service=tcl-ingress/haproxy-ingress-nlb
--wait-before-shutdown=30

Global options:

healthz-port: "10253"
https-log-format: default
prometheus-port: "9101"
stats-port: "1936"
syslog-endpoint: stdout
syslog-format: raw
tcp-log-format: default
use-haproxy-user: "true"
use-proxy-protocol: "true"

Ingress objects:

Probably not relevant.

TCP Services ConfigMap keys:

"22": gitlab/gitlab-gitlab-shell:22:PROXY
"8140": puppet/puppetserver:443:PROXY

Log from one of the updates:

I0404 12:07:37.002014       7 controller.go:338] starting haproxy update id=366
I0404 12:07:37.002096       7 converters.go:67] applying 2 change notifications: [update/Secret:gitlab-runner/runner-2gxzz8rg-project-63-concurrent-0q2tqt add/Secret:gitlab-runner/runner-2gxzz8rg-project-63-concurrent-0q2tqt]
I0404 12:07:37.002307       7 ingress.go:232] syncing 0 host(s) and 0 backend(s)
I0404 12:07:37.004877       7 instance.go:441] updating 0 host(s): []
I0404 12:07:37.004888       7 instance.go:458] updating 0 backend(s): []
I0404 12:07:37.004922       7 dynupdate.go:104] need to reload due to config changes: [tcp-services (configmap)]
I0404 12:07:37.041428       7 instance.go:401] haproxy successfully reloaded (external)
I0404 12:07:37.041479       7 controller.go:347] finish haproxy update id=366: parse_ingress=0.292283ms parse_tcp_svc=2.541356ms write_maps=0.006889ms write_config=2.515851ms reload_haproxy=34.022806ms total=39.379185ms

[jcmoraisjr]

Couldn't reproduce the issue, but it seems doable in the case k8s endpoints are provided in a distinct order between reconciliations. I hope #1003 can address that. It'd be really helpful any info just in case you can see differences in the tcp listener configuration between restarts, e.g. a duplicated endpoint entry, endpoints our of order, etc.

[jcmoraisjr]

v0.14.3 was just released with that improvement, please let us know if you continue to reproduce this behavior.

[bootc]

Sorry I never got back to you about this. Unfortunately I'm still seeing the same behaviour with quay.io/jcmoraisjr/haproxy-ingress:v0.14.3; and I think you're right it's because of the ordering - but it's a bit different after your attempted fix...

This is a diff before/after an update like the above:

--- haproxy.cfg.orig	2023-06-29 10:42:58
+++ haproxy.cfg.new1	2023-06-29 10:44:48
@@ -75,8 +75,8 @@
     bind :8140 accept-proxy
     mode tcp
     option tcplog
-    server srv003 10.198.28.230:443 check port 443 inter 2s
-    server srv002 10.198.3.250:443 check port 443 inter 2s
+    server srv002 10.198.28.230:443 check port 443 inter 2s
+    server srv003 10.198.3.250:443 check port 443 inter 2s
     server srv001 10.198.37.86:443 check port 443 inter 2s
 
   # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #

Notice how the IP addresses are indeed in sorted order and consistent, but the srv002 and srv003 entries swap between them!

[jcmoraisjr]

Thanks for the update, really appreciated. I in fact patched the wrong place, #1011 should solve it. New tags later this week.

[bootc]

Thanks, I can confirm v0.14.4 and #1011 solve the issue.