Repository navigation
Commit 6a0fcd6
authored
LB: per-rule keepalive and timeouts, plus three HAProxy config fixes (#14131)
* Use a keepalive option HAProxy still accepts
With keepalive enabled, an LB rule that also uses HTTP stickiness or SSL
offload emitted "no option forceclose". HAProxy has rejected that keyword
since 2.0, and the system VM has shipped 2.x for several releases:
[ALERT] config : parsing [haproxy.cfg:22]: option 'forceclose' is not
supported any more since HAProxy 2.0, please just remove it, or use
'option httpclose' if absolutely needed.
It is a fatal parse error, so haproxy keeps running the previous config
and the rule silently never takes effect.
Replaced with "option http-keep-alive", which says the same thing and is
valid on every version the system VM has shipped.
Signed-off-by: Brad House <bhouse@nexthop.ai>
* Stop one load balancer's config leaking into the next
The global and defaults sections are static arrays. Wrapping them in
Arrays.asList() and calling set() writes through to the array, so a value
from one LB rule stays there for every config generated afterwards:
- keepalive on any rule leaves "no option httpclose" in the defaults
for every later rule, in any network, in any account
- an idle timeout from one rule becomes the default for the next
- idleTimeout 0 blanks the timeouts permanently
Copy the arrays instead. Same output, no shared state.
Signed-off-by: Brad House <bhouse@nexthop.ai>
* Replace appsession, removed from HAProxy in 1.6
An LB rule with AppCookie stickiness emitted "appsession", which haproxy
has refused since 1.6:
[ALERT] config : 'appsession' is not supported anymore since HAProxy 1.6.
[ALERT] config : Fatal errors found in configuration.
The whole file is rejected, so every rule on that router keeps serving
its previous config, not just the one with the policy. The VR advertises
AppCookie as supported, so this is reachable from the API.
A stick table on the cookie is the documented replacement:
stick-table type string len <length> size 10k expire <holdtime>
stick store-response res.cook(<name>)
stick match req.cook(<name>)
stick store-request req.cook(<name>) # only with request-learn
The prefix and mode options have no equivalent and are now logged and
ignored. They were never applied - the directive carrying them was
rejected - so no working behaviour changes.
Signed-off-by: Brad House <bhouse@nexthop.ai>
* Make keepalive and timeouts settable per load balancer rule
#12586 made the idle timeout global. Keepalive is set on the network
offering, at create time only, with no UI. Both land in the haproxy
defaults section, so every rule on a router shares them.
Three optional parameters on createLoadBalancerRule and
updateLoadBalancerRule, each written to that rule's own listen section:
| parameter | haproxy directive | unset |
| keepalive | option http-keep-alive / httpclose | offering |
| idletimeout | timeout client, timeout server | global |
| keepalivetimeout | timeout http-keep-alive | idletimeout|
Held as firewall rule details, so no schema change.
A rule on port 80 can now have keepalive and X-Forwarded-For together,
which no combination of the existing settings can produce:
offering flag off mode http closes per response XFF works
offering flag on mode tcp reuses connections XFF lost
per rule mode http reuses connections XFF works
A rule that sets nothing behaves exactly as before, including the fall
back to tcp mode when the offering flag is on.
Negative timeouts are rejected at the API and dropped in the generator.
Haproxy treats one as a fatal parse error, which would strand every rule
on the router.
Applies to public LB rules. Application load balancers are created
through a different command and are not covered.
Signed-off-by: Brad House <bhouse@nexthop.ai>
* Check the per rule settings in the VR health check
haproxy_check.py compares the router's haproxy.cfg against what the
management server thinks it configured. It knew about maxconn and the
global idle timeout, so a rule's own keepalive and timeouts could drift
without anything noticing.
The health check data now carries the three per rule values, empty when
the rule inherits, and the check validates them in that rule's listen
section:
ruleKeepAlive option http-keep-alive, or option httpclose
ruleIdleTimeout timeout client, timeout server
ruleKeepAliveTimeout timeout http-keep-alive
The http mode test needed widening too. It read the offering flag alone,
so a rule that sets keepalive itself stopped being checked at all:
before port 80 and the offering has keepalive off
after port 80 and (the rule sets keepalive, or the offering has it off)
An older management server sends none of these keys, and a rule that sets
nothing sends them empty. Both read as nothing to check.
Signed-off-by: Brad House <bhouse@nexthop.ai>
* Pass the update command's rule id to the connection settings
lb was looked up by that same id and is non null by this point, so the
Long unboxes safely. The comment claiming otherwise was wrong, and the
id now matches the one applyLoadBalancerConfig gets a few lines later.
Signed-off-by: Brad House <bhouse@nexthop.ai>
* Let updateLoadBalancerRule put a rule's settings back to the default
Once a rule had its own keepalive or timeout there was no way back to
inheriting. Leaving a parameter out keeps the current value, and there
is no value that means unset: negative timeouts are refused, and
keepalive is a boolean.
updateLoadBalancerRule takes cleanupconnectionsettings, after the
cleanupdetails flag on updateVirtualMachine and updateTemplate. It drops
all three of the rule's own settings, then sets again any passed in the
same call, so one call can reset one and keep the rest:
rule holds call rule holds after
idletimeout=5000 cleanupconnectionsettings=true nothing
keepalive=true
idletimeout=5000 cleanupconnectionsettings=true idletimeout=5000
keepalive=true idletimeout=5000
idletimeout=5000 keepalivetimeout=2000 idletimeout=5000
keepalivetimeout=2000
A change is now worked out by comparing the settings before and after,
so a cleanup on a rule that holds none does not reapply the config.
The update parameter descriptions said an unset value inherits, which is
only true on create. They now say it keeps the current value.
Signed-off-by: Brad House <bhouse@nexthop.ai>
* Send cleanupconnectionsettings when a field is cleared in the UI
Clearing keepalive, idle timeout or keepalive timeout in the edit rule
dialog dropped the parameter, which the API reads as keep the current
value, so the old setting stayed. When a field the rule had set is now
blank the dialog sends cleanupconnectionsettings, along with the fields
still filled in.
Signed-off-by: Brad House <bhouse@nexthop.ai>
* Roll back a rule's connection settings when the router update fails
When applying an updated rule to the router fails, updateLoadBalancerRule
puts back the name, description, algorithm and cidrs, but not the
keepalive and timeouts. Those stayed in the database while the router
still ran the old values, and the next successful apply of anything on
that router would quietly push them.
The settings held before the update are now restored along with the
rest.
They were also written before the provider validated the rule, so a
rule it rejected kept the new settings. The other fields only reach the
database at _lbDao.update, after validation; the settings are now
written at the same point.
Signed-off-by: Brad House <bhouse@nexthop.ai>
* Mark the new load balancer parameters as since 24.0
The keepalive, idle timeout and keepalive timeout parameters, and the
cleanupconnectionsettings flag, were marked since 4.23.0. That release
shipped without them, and main is now 24.0.0-SNAPSHOT following the
vote to drop the "4." prefix starting with release 24.
Signed-off-by: Brad House <bhouse@nexthop.ai>
* Give the update load balancer test command a rule id
updateLoadBalancerRule now passes the command's id to the connection
settings, which take a long. UpdateLoadBalancerTest built its command
without an id, so the null unboxed and testValidateRuleBeforeUpdateLB
threw. The API marks id as required, so only the test could get there.
Set the id on the command and stub the DAO calls with it, and mock the
firewall rule details DAO the settings are read from. testRuleNotValidated
only passed before because the null id missed the lookup and raised the
same exception type. It now finds the rule and fails on validation, which
is what it means to test.
Signed-off-by: Brad House <bhouse@nexthop.ai>
* Do not fail a rule update when the rule has no protocol
updateLoadBalancerRule called lb.getLbProtocol().equals(...) after
applying the config. lb_protocol is null for a rule created through the
API without a protocol, so the update threw an NPE and reported failure
although the change had already been applied to the router. Stock hits
this on an algorithm or CIDR change; with the per-rule connection
settings every keepalive or timeout update on such a rule hits it.
Signed-off-by: Brad House <bhouse@nexthop.ai>
* Do not fail an update that only changes connection settings
updateLoadBalancerRule took its result from _lbDao.update(), which
returns false when no column of the rule changed. The keepalive and
timeout settings are kept in firewall_rule_details, so an update that
only changed them applied the new config to the router and then failed
with "Failed to update load balancer rule". Count a settings change as
an update. An update that changes nothing still fails, as before.
Signed-off-by: Brad House <bhouse@nexthop.ai>
* Check a new rule's timeouts before the rule is created
createLoadBalancerRule stored the per rule settings only after the
rule had been created, so a refused keepalivetimeout or idletimeout
left the rule behind in state Add. It held its port, and the next
apply on the network pushed it to the router.
Check the timeouts first, so a refused create leaves nothing. Also
refuse a timeout above 2147483647 ms, which HAProxy rejects as a timer
overflow; it used to reach the router and fail the whole apply.
Signed-off-by: Brad House <bhouse@nexthop.ai>
---------
Signed-off-by: Brad House <bhouse@nexthop.ai>1 parent 2a17946 commit 6a0fcd6
19 files changed
Lines changed: 1011 additions & 44 deletions
File tree
- api/src
- main/java
- com/cloud
- agent/api/to
- network
- lb
- rules
- org/apache/cloudstack/api
- command/user/loadbalancer
- response
- test/java/org/apache/cloudstack/api/command/user/loadbalancer
- core/src
- main/java/com/cloud/network
- test/java/com/cloud/network
- server/src
- main/java/com/cloud
- api
- network
- lb
- router
- test/java/com/cloud/network/lb
- systemvm/debian/root/health_checks
- ui
- public/locales
- src/views/network
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
54 | 54 | | |
55 | 55 | | |
56 | 56 | | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
57 | 60 | | |
58 | 61 | | |
59 | 62 | | |
| |||
249 | 252 | | |
250 | 253 | | |
251 | 254 | | |
| 255 | + | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
| 267 | + | |
| 268 | + | |
| 269 | + | |
| 270 | + | |
| 271 | + | |
| 272 | + | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
| 278 | + | |
252 | 279 | | |
253 | 280 | | |
254 | 281 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
70 | 70 | | |
71 | 71 | | |
72 | 72 | | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
73 | 87 | | |
74 | 88 | | |
75 | 89 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
21 | 21 | | |
22 | 22 | | |
23 | 23 | | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
24 | 29 | | |
25 | 30 | | |
26 | 31 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
377 | 377 | | |
378 | 378 | | |
379 | 379 | | |
| 380 | + | |
380 | 381 | | |
381 | 382 | | |
382 | 383 | | |
| |||
388 | 389 | | |
389 | 390 | | |
390 | 391 | | |
| 392 | + | |
| 393 | + | |
391 | 394 | | |
392 | 395 | | |
393 | 396 | | |
| |||
1253 | 1256 | | |
1254 | 1257 | | |
1255 | 1258 | | |
| 1259 | + | |
1256 | 1260 | | |
1257 | 1261 | | |
1258 | 1262 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
119 | 119 | | |
120 | 120 | | |
121 | 121 | | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
122 | 138 | | |
123 | 139 | | |
124 | 140 | | |
| |||
135 | 151 | | |
136 | 152 | | |
137 | 153 | | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
138 | 166 | | |
139 | 167 | | |
140 | 168 | | |
| |||
301 | 329 | | |
302 | 330 | | |
303 | 331 | | |
| 332 | + | |
| 333 | + | |
304 | 334 | | |
305 | 335 | | |
306 | 336 | | |
307 | 337 | | |
308 | 338 | | |
309 | 339 | | |
| 340 | + | |
310 | 341 | | |
311 | 342 | | |
312 | 343 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
68 | 68 | | |
69 | 69 | | |
70 | 70 | | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
71 | 96 | | |
72 | 97 | | |
73 | 98 | | |
74 | 99 | | |
75 | 100 | | |
76 | 101 | | |
77 | 102 | | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
78 | 119 | | |
79 | 120 | | |
80 | 121 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
63 | 63 | | |
64 | 64 | | |
65 | 65 | | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
66 | 78 | | |
67 | 79 | | |
68 | 80 | | |
| |||
143 | 155 | | |
144 | 156 | | |
145 | 157 | | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
146 | 170 | | |
147 | 171 | | |
148 | 172 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
0 commit comments