Haproxy.cfg is not reloaded when pod IP address changes and standalone-backend annotation is true

[dosmanak]

We experience a buggy behavior when haproxy.org/standalone-backeng: "true".

It is the most probable cause of error.

When the pod that is under service is deleted and its replicaset or statefulset starts a new one, the haproxy is not reloaded to put new pod IP into backend server list.

We also use ingres with serveral path prefixes, but we use that on different project without issue.

$ /haproxy-ingress-controller --version
2025/02/27 09:55:39 maxprocs: Leaving GOMAXPROCS=16: CPU quota undefined
HAProxy Ingress Controller v3.0.5 54dded2f.dirty
Build from: github.com/haproxytech/kubernetes-ingress
Git commit date: 2024-12-18T20:53:52Z

$ haproxy -v
HAProxy version 3.0.7-ce35390 2024/12/12 - https://haproxy.org/
Status: long-term supported branch - will stop receiving fixes around Q2 2029.
Known bugs: http://www.haproxy.org/bugs/bugs-3.0.7.html
Running on: Linux 5.15.0-1078-azure #87-Ubuntu SMP Wed Dec 18 19:21:34 UTC 2024 x86_64

I am not able to reproduce it clearly on staging environment at the moment.

We decided for standalone backend so the backend-config snippet for each ingress is uniq even though they have the same service in backend. That is necesary to be able to use custom errorfiles on each ingress.

[dosmanak]

I was able to get into state, where haproxy.cfg does not correspond with its state.

/ $ grep -A15 ing_customer-test_ing-haproxy-xmldata_jnp /etc/haproxy/haproxy.cfg 
backend ing_customer-test_ing-haproxy-xmldata_jnp
  mode http
  balance roundrobin
  option forwardfor
  no option abortonclose
  default-server check maxconn 100
  server SRV_1 10.0.1.61:8080 enabled
  server SRV_2 127.0.0.1:8080 disabled
  server SRV_3 127.0.0.1:8080 disabled
  server SRV_4 127.0.0.1:8080 disabled
  server SRV_5 127.0.0.1:8080 disabled
  server SRV_6 127.0.0.1:8080 disabled
  server SRV_7 127.0.0.1:8080 disabled
  server SRV_8 127.0.0.1:8080 disabled

backend keycloak-dev_svc-keycloak_http
/ $ echo -en "show servers state ing_customer-test_ing-haproxy-xmldata_jnp\r\n"| socat stdio  /run/haproxy-runtime-api.sock 
1
# be_id be_name srv_id srv_name srv_addr srv_op_state srv_admin_state srv_uweight srv_iweight srv_time_since_last_change srv_check_status srv_check_result srv_check_health srv_check_state srv_agent_state bk_f_forced_id srv_f_forced_id srv_fqdn srv_port srvrecord srv_use_ssl srv_check_port srv_check_addr srv_agent_addr srv_agent_port
9 ing_customer-test_ing-haproxy-xmldata_jnp 1 SRV_1 127.0.0.1 0 1 1 1 2498 6 3 0 14 0 0 0 - 8080 - 0 0 - - 0
9 ing_customer-test_ing-haproxy-xmldata_jnp 2 SRV_2 127.0.0.1 0 5 1 1 2881 1 0 0 14 0 0 0 - 8080 - 0 0 - - 0
9 ing_customer-test_ing-haproxy-xmldata_jnp 3 SRV_3 127.0.0.1 0 5 1 1 2881 1 0 0 14 0 0 0 - 8080 - 0 0 - - 0
9 ing_customer-test_ing-haproxy-xmldata_jnp 4 SRV_4 127.0.0.1 0 5 1 1 2881 1 0 0 14 0 0 0 - 8080 - 0 0 - - 0
9 ing_customer-test_ing-haproxy-xmldata_jnp 5 SRV_5 127.0.0.1 0 5 1 1 2881 1 0 0 14 0 0 0 - 8080 - 0 0 - - 0
9 ing_customer-test_ing-haproxy-xmldata_jnp 6 SRV_6 127.0.0.1 0 5 1 1 2881 1 0 0 14 0 0 0 - 8080 - 0 0 - - 0
9 ing_customer-test_ing-haproxy-xmldata_jnp 7 SRV_7 127.0.0.1 0 5 1 1 2881 1 0 0 14 0 0 0 - 8080 - 0 0 - - 0
9 ing_customer-test_ing-haproxy-xmldata_jnp 8 SRV_8 127.0.0.1 0 5 1 1 2881 1 0 0 14 0 0 0 - 8080 - 0 0 - - 0

[dosmanak]

2025/02/27 11:29:17 INFO    service/service.go:169 [transactionID=736de81a-66b0-4993-8d0a-48587b1fe57f] reload required : Service 'customer-test/svc-jnppub': new backend 'ing_customer-test_ing-haproxy-xmldata_jnp'
2025/02/27 11:29:17 INFO    controller/controller.go:203 [transactionID=736de81a-66b0-4993-8d0a-48587b1fe57f] HAProxy reloaded

[dosmanak]

I tried to send HUP to haproxy without success.

2025-02-27T12:59:20.617191148Z [WARNING]  (30696) : SIGHUP: Server ing_customer-test_ing-haproxy-xmldata_jnp/SRV_1 is DOWN. Conn: 0 act, 0 pend, 0 tot.
2025-02-27T12:59:20.617195378Z [WARNING]  (30696) : SIGHUP: Server ing_customer-test_ing-haproxy-xmldata_jnp/SRV_2 is DOWN. Conn: 0 act, 0 pend, 0 tot.
2025-02-27T12:59:20.617198902Z [WARNING]  (30696) : SIGHUP: Server ing_customer-test_ing-haproxy-xmldata_jnp/SRV_3 is DOWN. Conn: 0 act, 0 pend, 0 tot.
2025-02-27T12:59:20.617202627Z [WARNING]  (30696) : SIGHUP: Server ing_customer-test_ing-haproxy-xmldata_jnp/SRV_4 is DOWN. Conn: 0 act, 0 pend, 0 tot.
2025-02-27T12:59:20.617206473Z [WARNING]  (30696) : SIGHUP: Server ing_customer-test_ing-haproxy-xmldata_jnp/SRV_5 is DOWN. Conn: 0 act, 0 pend, 0 tot.
2025-02-27T12:59:20.617258416Z [WARNING]  (30696) : SIGHUP: Server ing_customer-test_ing-haproxy-xmldata_jnp/SRV_6 is DOWN. Conn: 0 act, 0 pend, 0 tot.
2025-02-27T12:59:20.617288338Z SIGHUP received, dumping servers states for proxy ing_customer-test_ing-haproxy-xmldata_jnp.
2025-02-27T12:59:20.617300929Z SIGHUP: Server ing_customer-test_ing-haproxy-xmldata_jnp/SRV_1 is DOWN. Conn: 0 act, 0 pend, 0 tot.
2025-02-27T12:59:20.617304987Z SIGHUP: Server ing_customer-test_ing-haproxy-xmldata_jnp/SRV_2 is DOWN. Conn: 0 act, 0 pend, 0 tot.
2025-02-27T12:59:20.617308782Z SIGHUP: Server ing_customer-test_ing-haproxy-xmldata_jnp/SRV_3 is DOWN. Conn: 0 act, 0 pend, 0 tot.
2025-02-27T12:59:20.617312951Z SIGHUP: Server ing_customer-test_ing-haproxy-xmldata_jnp/SRV_4 is DOWN. Conn: 0 act, 0 pend, 0 tot.
2025-02-27T12:59:20.617316753Z SIGHUP: Server ing_customer-test_ing-haproxy-xmldata_jnp/SRV_5 is DOWN. Conn: 0 act, 0 pend, 0 tot.
2025-02-27T12:59:20.617320374Z SIGHUP: Server ing_customer-test_ing-haproxy-xmldata_jnp/SRV_6 is DOWN. Conn: 0 act, 0 pend, 0 tot.
2025-02-27T12:59:20.617324047Z SIGHUP: Server ing_customer-test_ing-haproxy-xmldata_jnp/SRV_7 is DOWN. Conn: 0 act, 0 pend, 0 tot.
2025-02-27T12:59:20.617327767Z SIGHUP: Server ing_customer-test_ing-haproxy-xmldata_jnp/SRV_8 is DOWN. Conn: 0 act, 0 pend, 0 tot.
2025-02-27T12:59:20.617331710Z SIGHUP: Proxy ing_customer-test_ing-haproxy-xmldata_jnp has no server available ! Conn: act(FE+BE): 0+0, 0 pend (0 unass), tot(FE+BE): 0+3.
2025-02-27T12:59:20.617274856Z [WARNING]  (30696) : SIGHUP: Server ing_customer-test_ing-haproxy-xmldata_jnp/SRV_7 is DOWN. Conn: 0 act, 0 pend, 0 tot.
2025-02-27T12:59:20.617389702Z [WARNING]  (30696) : SIGHUP: Server ing_customer-test_ing-haproxy-xmldata_jnp/SRV_8 is DOWN. Conn: 0 act, 0 pend, 0 tot.
2025-02-27T12:59:20.617394443Z [WARNING]  (30696) : SIGHUP: Proxy ing_customer-test_ing-haproxy-xmldata_jnp has no server available ! Conn: act(FE+BE): 0+0, 0 pend (0 unass), tot(FE+BE): 0+3.

[ivanmatmati]

@dosmanak , Thanks for the complementary input. I'll see that.

[dosmanak]

We investigated the code and perhaps the controller decides it is not necessary to reload Haproxy.
Here https://github.com/haproxytech/kubernetes-ingress/blob/master/pkg/controller/controller.go#L206

Maybe the isNeededReload checks only one haproxy backend that is connected to kubernetes service, but not the other haproxy backend that is created due the standalone-backend annotation.

[ivanmatmati]

Congratulations for helping us in the debugging, actually that's an other part that is concerned about this issue. It has to do with update of servers lists from endpoints events that considers only the regular backend but not the derived ones from standalone annotations. I'll create a fix for that.

[dosmanak]

Hello. Do you have estimate in what version, the fix will be available?
Thank you.

[ivanmatmati]

Hi @dosmanak, it should be available in the next version. This should be around mid june.

[dosmanak]

That is dissapointing. I will rewrite multiple ingresses into multiple service and I hope that will resove the issue for us.

[dosmanak]

I have rewritten it into multiple services and just a single ingress with many host rules. It seems working properly without standalone backend annotation. So no hurry with fix needed.