Skip to content

Commit 6a0fcd6

Browse files
LB: per-rule keepalive and timeouts, plus three HAProxy config fixes (#14131)
* Use a keepalive option HAProxy still accepts With keepalive enabled, an LB rule that also uses HTTP stickiness or SSL offload emitted "no option forceclose". HAProxy has rejected that keyword since 2.0, and the system VM has shipped 2.x for several releases: [ALERT] config : parsing [haproxy.cfg:22]: option 'forceclose' is not supported any more since HAProxy 2.0, please just remove it, or use 'option httpclose' if absolutely needed. It is a fatal parse error, so haproxy keeps running the previous config and the rule silently never takes effect. Replaced with "option http-keep-alive", which says the same thing and is valid on every version the system VM has shipped. Signed-off-by: Brad House <bhouse@nexthop.ai> * Stop one load balancer's config leaking into the next The global and defaults sections are static arrays. Wrapping them in Arrays.asList() and calling set() writes through to the array, so a value from one LB rule stays there for every config generated afterwards: - keepalive on any rule leaves "no option httpclose" in the defaults for every later rule, in any network, in any account - an idle timeout from one rule becomes the default for the next - idleTimeout 0 blanks the timeouts permanently Copy the arrays instead. Same output, no shared state. Signed-off-by: Brad House <bhouse@nexthop.ai> * Replace appsession, removed from HAProxy in 1.6 An LB rule with AppCookie stickiness emitted "appsession", which haproxy has refused since 1.6: [ALERT] config : 'appsession' is not supported anymore since HAProxy 1.6. [ALERT] config : Fatal errors found in configuration. The whole file is rejected, so every rule on that router keeps serving its previous config, not just the one with the policy. The VR advertises AppCookie as supported, so this is reachable from the API. A stick table on the cookie is the documented replacement: stick-table type string len <length> size 10k expire <holdtime> stick store-response res.cook(<name>) stick match req.cook(<name>) stick store-request req.cook(<name>) # only with request-learn The prefix and mode options have no equivalent and are now logged and ignored. They were never applied - the directive carrying them was rejected - so no working behaviour changes. Signed-off-by: Brad House <bhouse@nexthop.ai> * Make keepalive and timeouts settable per load balancer rule #12586 made the idle timeout global. Keepalive is set on the network offering, at create time only, with no UI. Both land in the haproxy defaults section, so every rule on a router shares them. Three optional parameters on createLoadBalancerRule and updateLoadBalancerRule, each written to that rule's own listen section: | parameter | haproxy directive | unset | | keepalive | option http-keep-alive / httpclose | offering | | idletimeout | timeout client, timeout server | global | | keepalivetimeout | timeout http-keep-alive | idletimeout| Held as firewall rule details, so no schema change. A rule on port 80 can now have keepalive and X-Forwarded-For together, which no combination of the existing settings can produce: offering flag off mode http closes per response XFF works offering flag on mode tcp reuses connections XFF lost per rule mode http reuses connections XFF works A rule that sets nothing behaves exactly as before, including the fall back to tcp mode when the offering flag is on. Negative timeouts are rejected at the API and dropped in the generator. Haproxy treats one as a fatal parse error, which would strand every rule on the router. Applies to public LB rules. Application load balancers are created through a different command and are not covered. Signed-off-by: Brad House <bhouse@nexthop.ai> * Check the per rule settings in the VR health check haproxy_check.py compares the router's haproxy.cfg against what the management server thinks it configured. It knew about maxconn and the global idle timeout, so a rule's own keepalive and timeouts could drift without anything noticing. The health check data now carries the three per rule values, empty when the rule inherits, and the check validates them in that rule's listen section: ruleKeepAlive option http-keep-alive, or option httpclose ruleIdleTimeout timeout client, timeout server ruleKeepAliveTimeout timeout http-keep-alive The http mode test needed widening too. It read the offering flag alone, so a rule that sets keepalive itself stopped being checked at all: before port 80 and the offering has keepalive off after port 80 and (the rule sets keepalive, or the offering has it off) An older management server sends none of these keys, and a rule that sets nothing sends them empty. Both read as nothing to check. Signed-off-by: Brad House <bhouse@nexthop.ai> * Pass the update command's rule id to the connection settings lb was looked up by that same id and is non null by this point, so the Long unboxes safely. The comment claiming otherwise was wrong, and the id now matches the one applyLoadBalancerConfig gets a few lines later. Signed-off-by: Brad House <bhouse@nexthop.ai> * Let updateLoadBalancerRule put a rule's settings back to the default Once a rule had its own keepalive or timeout there was no way back to inheriting. Leaving a parameter out keeps the current value, and there is no value that means unset: negative timeouts are refused, and keepalive is a boolean. updateLoadBalancerRule takes cleanupconnectionsettings, after the cleanupdetails flag on updateVirtualMachine and updateTemplate. It drops all three of the rule's own settings, then sets again any passed in the same call, so one call can reset one and keep the rest: rule holds call rule holds after idletimeout=5000 cleanupconnectionsettings=true nothing keepalive=true idletimeout=5000 cleanupconnectionsettings=true idletimeout=5000 keepalive=true idletimeout=5000 idletimeout=5000 keepalivetimeout=2000 idletimeout=5000 keepalivetimeout=2000 A change is now worked out by comparing the settings before and after, so a cleanup on a rule that holds none does not reapply the config. The update parameter descriptions said an unset value inherits, which is only true on create. They now say it keeps the current value. Signed-off-by: Brad House <bhouse@nexthop.ai> * Send cleanupconnectionsettings when a field is cleared in the UI Clearing keepalive, idle timeout or keepalive timeout in the edit rule dialog dropped the parameter, which the API reads as keep the current value, so the old setting stayed. When a field the rule had set is now blank the dialog sends cleanupconnectionsettings, along with the fields still filled in. Signed-off-by: Brad House <bhouse@nexthop.ai> * Roll back a rule's connection settings when the router update fails When applying an updated rule to the router fails, updateLoadBalancerRule puts back the name, description, algorithm and cidrs, but not the keepalive and timeouts. Those stayed in the database while the router still ran the old values, and the next successful apply of anything on that router would quietly push them. The settings held before the update are now restored along with the rest. They were also written before the provider validated the rule, so a rule it rejected kept the new settings. The other fields only reach the database at _lbDao.update, after validation; the settings are now written at the same point. Signed-off-by: Brad House <bhouse@nexthop.ai> * Mark the new load balancer parameters as since 24.0 The keepalive, idle timeout and keepalive timeout parameters, and the cleanupconnectionsettings flag, were marked since 4.23.0. That release shipped without them, and main is now 24.0.0-SNAPSHOT following the vote to drop the "4." prefix starting with release 24. Signed-off-by: Brad House <bhouse@nexthop.ai> * Give the update load balancer test command a rule id updateLoadBalancerRule now passes the command's id to the connection settings, which take a long. UpdateLoadBalancerTest built its command without an id, so the null unboxed and testValidateRuleBeforeUpdateLB threw. The API marks id as required, so only the test could get there. Set the id on the command and stub the DAO calls with it, and mock the firewall rule details DAO the settings are read from. testRuleNotValidated only passed before because the null id missed the lookup and raised the same exception type. It now finds the rule and fails on validation, which is what it means to test. Signed-off-by: Brad House <bhouse@nexthop.ai> * Do not fail a rule update when the rule has no protocol updateLoadBalancerRule called lb.getLbProtocol().equals(...) after applying the config. lb_protocol is null for a rule created through the API without a protocol, so the update threw an NPE and reported failure although the change had already been applied to the router. Stock hits this on an algorithm or CIDR change; with the per-rule connection settings every keepalive or timeout update on such a rule hits it. Signed-off-by: Brad House <bhouse@nexthop.ai> * Do not fail an update that only changes connection settings updateLoadBalancerRule took its result from _lbDao.update(), which returns false when no column of the rule changed. The keepalive and timeout settings are kept in firewall_rule_details, so an update that only changed them applied the new config to the router and then failed with "Failed to update load balancer rule". Count a settings change as an update. An update that changes nothing still fails, as before. Signed-off-by: Brad House <bhouse@nexthop.ai> * Check a new rule's timeouts before the rule is created createLoadBalancerRule stored the per rule settings only after the rule had been created, so a refused keepalivetimeout or idletimeout left the rule behind in state Add. It held its port, and the next apply on the network pushed it to the router. Check the timeouts first, so a refused create leaves nothing. Also refuse a timeout above 2147483647 ms, which HAProxy rejects as a timer overflow; it used to reach the router and fail the whole apply. Signed-off-by: Brad House <bhouse@nexthop.ai> --------- Signed-off-by: Brad House <bhouse@nexthop.ai>
1 parent 2a17946 commit 6a0fcd6

19 files changed

Lines changed: 1011 additions & 44 deletions

File tree

‎api/src/main/java/com/cloud/agent/api/to/LoadBalancerTO.java‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -54,6 +54,9 @@ public class LoadBalancerTO {
5454
final static int MAX_HEALTHCHECK_POLICIES = 1;
5555

5656
private String cidrList;
57+
private Boolean keepAlive;
58+
private Long idleTimeout;
59+
private Long keepAliveTimeout;
5760

5861
public LoadBalancerTO(String uuid, String srcIp, int srcPort, String protocol, String algorithm, boolean revoked, boolean alreadyAdded, boolean inline,
5962
List<LbDestination> destinations) {
@@ -249,6 +252,30 @@ public String getCidrList() {
249252
return cidrList;
250253
}
251254

255+
public void setKeepAlive(Boolean keepAlive) {
256+
this.keepAlive = keepAlive;
257+
}
258+
259+
public Boolean getKeepAlive() {
260+
return keepAlive;
261+
}
262+
263+
public void setIdleTimeout(Long idleTimeout) {
264+
this.idleTimeout = idleTimeout;
265+
}
266+
267+
public Long getIdleTimeout() {
268+
return idleTimeout;
269+
}
270+
271+
public void setKeepAliveTimeout(Long keepAliveTimeout) {
272+
this.keepAliveTimeout = keepAliveTimeout;
273+
}
274+
275+
public Long getKeepAliveTimeout() {
276+
return keepAliveTimeout;
277+
}
278+
252279
public static class StickinessPolicyTO {
253280
private String methodName;
254281
private List<Pair<String, String>> params;

‎api/src/main/java/com/cloud/network/lb/LoadBalancingRulesService.java‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -70,6 +70,20 @@ LoadBalancer createPublicLoadBalancerRule(String xId, String name, String descri
7070

7171
LoadBalancer updateLoadBalancerRule(UpdateLoadBalancerRuleCmd cmd);
7272

73+
/**
74+
* Stores the optional per rule haproxy settings. A null leaves the current value alone.
75+
*
76+
* @param cleanup remove all three first, so the rule inherits again, then store any non null value given
77+
* @return true if any value changed
78+
*/
79+
boolean updateLoadBalancerConnectionSettings(long lbRuleId, Boolean keepAlive, Long idleTimeout, Long keepAliveTimeout, boolean cleanup);
80+
81+
/**
82+
* Checks the per rule timeouts without storing them, so a create can refuse a bad value
83+
* before the rule exists. A null is not checked.
84+
*/
85+
void validateLoadBalancerConnectionSettings(Long idleTimeout, Long keepAliveTimeout);
86+
7387
boolean deleteLoadBalancerRule(long lbRuleId, boolean apply);
7488

7589
/**

‎api/src/main/java/com/cloud/network/rules/LoadBalancer.java‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,11 @@
2121
*/
2222
public interface LoadBalancer extends FirewallRule, LoadBalancerContainer {
2323

24+
// Optional per rule haproxy settings, held as firewall rule details. Absent means inherit.
25+
String KEEPALIVE = "keepalive";
26+
String IDLE_TIMEOUT = "idletimeout";
27+
String KEEPALIVE_TIMEOUT = "keepalivetimeout";
28+
2429
int getDefaultPortStart();
2530

2631
int getDefaultPortEnd();

‎api/src/main/java/org/apache/cloudstack/api/ApiConstants.java‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -377,6 +377,7 @@ public class ApiConstants {
377377
public static final String IS_ISO = "isiso";
378378
public static final String IS_PORTABLE = "isportable";
379379
public static final String IS_PUBLIC = "ispublic";
380+
public static final String IDLE_TIMEOUT = "idletimeout";
380381
public static final String IS_PERSISTENT = "ispersistent";
381382
public static final String EGRESS_DEFAULT_POLICY = "egressdefaultpolicy";
382383
public static final String IS_READY = "isready";
@@ -388,6 +389,8 @@ public class ApiConstants {
388389
public static final String JAVA_VERSION = "javaversion";
389390
public static final String JOB_ID = "jobid";
390391
public static final String JOB_STATUS = "jobstatus";
392+
public static final String KEEPALIVE = "keepalive";
393+
public static final String KEEPALIVE_TIMEOUT = "keepalivetimeout";
391394
public static final String KEEPALIVE_ENABLED = "keepaliveenabled";
392395
public static final String KERNEL_VERSION = "kernelversion";
393396
public static final String KEYPAIR_ID = "keypairid";
@@ -1253,6 +1256,7 @@ public class ApiConstants {
12531256
public static final String OVM3_POOL = "ovm3pool";
12541257
public static final String OVM3_CLUSTER = "ovm3cluster";
12551258
public static final String OVM3_VIP = "ovm3vip";
1259+
public static final String CLEAN_UP_CONNECTION_SETTINGS = "cleanupconnectionsettings";
12561260
public static final String CLEAN_UP_DETAILS = "cleanupdetails";
12571261
public static final String CLEAN_UP_EXTERNAL_DETAILS = "cleanupexternaldetails";
12581262
public static final String CLEAN_UP_EXTRA_CONFIG = "cleanupextraconfig";

‎api/src/main/java/org/apache/cloudstack/api/command/user/loadbalancer/CreateLoadBalancerRuleCmd.java‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -119,6 +119,22 @@ public class CreateLoadBalancerRuleCmd extends BaseAsyncCreateCmd /*implements L
119119
@Parameter(name = ApiConstants.FOR_DISPLAY, type = CommandType.BOOLEAN, description = "An optional field, whether to the display the rule to the end user or not", since = "4.4", authorized = {RoleType.Admin})
120120
private Boolean display;
121121

122+
@Parameter(name = ApiConstants.KEEPALIVE, type = CommandType.BOOLEAN, since = "24.0",
123+
description = "Whether the load balancer keeps client connections open between requests. "
124+
+ "Only applies to rules the router serves in HTTP mode. If not set, the network offering's setting is used.")
125+
private Boolean keepAlive;
126+
127+
@Parameter(name = ApiConstants.IDLE_TIMEOUT, type = CommandType.LONG, since = "24.0",
128+
description = "How long an idle connection is held open, in milliseconds. Use 0 for infinite. "
129+
+ "If not set, the global setting network.loadbalancer.haproxy.idle.timeout is used.")
130+
private Long idleTimeout;
131+
132+
@Parameter(name = ApiConstants.KEEPALIVE_TIMEOUT, type = CommandType.LONG, since = "24.0",
133+
description = "How long an idle keepalive connection is held open waiting for the next request, "
134+
+ "in milliseconds. Only applies to rules the router serves in HTTP mode. "
135+
+ "If not set, idletimeout applies.")
136+
private Long keepAliveTimeout;
137+
122138
/////////////////////////////////////////////////////
123139
/////////////////// Accessors ///////////////////////
124140
/////////////////////////////////////////////////////
@@ -135,6 +151,18 @@ public boolean isDisplay() {
135151
public String getAlgorithm() {
136152
return algorithm;
137153
}
154+
public Boolean getKeepAlive() {
155+
return keepAlive;
156+
}
157+
158+
public Long getIdleTimeout() {
159+
return idleTimeout;
160+
}
161+
public Long getKeepAliveTimeout() {
162+
return keepAliveTimeout;
163+
}
164+
165+
138166

139167
public String getDescription() {
140168
return description;
@@ -301,12 +329,15 @@ public void execute() throws ResourceAllocationException, ResourceUnavailableExc
301329
@Override
302330
public void create() {
303331
try {
332+
// Check the settings before the rule exists: a refused create must leave no rule behind
333+
_lbService.validateLoadBalancerConnectionSettings(getIdleTimeout(), getKeepAliveTimeout());
304334
LoadBalancer result =
305335
_lbService.createPublicLoadBalancerRule(getXid(), getName(), getDescription(), getSourcePortStart(), getSourcePortEnd(), getDefaultPortStart(),
306336
getDefaultPortEnd(), getSourceIpAddressId(), getProtocol(), getAlgorithm(), getNetworkId(), getEntityOwnerId(), getOpenFirewall(), getLbProtocol(), isDisplay(),
307337
getCidrList());
308338
this.setEntityId(result.getId());
309339
this.setEntityUuid(result.getUuid());
340+
_lbService.updateLoadBalancerConnectionSettings(result.getId(), getKeepAlive(), getIdleTimeout(), getKeepAliveTimeout(), false);
310341
} catch (NetworkRuleConflictException e) {
311342
logger.warn("Exception: ", e);
312343
throw new ServerApiException(ApiErrorCode.NETWORK_RULE_CONFLICT_ERROR, e.getMessage());

‎api/src/main/java/org/apache/cloudstack/api/command/user/loadbalancer/UpdateLoadBalancerRuleCmd.java‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -68,13 +68,54 @@ public class UpdateLoadBalancerRuleCmd extends BaseAsyncCustomIdCmd {
6868
@Parameter(name = ApiConstants.CIDR_LIST, type = CommandType.LIST, collectionType = CommandType.STRING, description = "the cidr list to forward traffic from", since = "4.22")
6969
private List<String> cidrList;
7070

71+
@Parameter(name = ApiConstants.KEEPALIVE, type = CommandType.BOOLEAN, since = "24.0",
72+
description = "Whether the load balancer keeps client connections open between requests. "
73+
+ "Only applies to rules the router serves in HTTP mode. If not set, the rule's current value is kept; "
74+
+ "use cleanupconnectionsettings to go back to the network offering's setting.")
75+
private Boolean keepAlive;
76+
77+
@Parameter(name = ApiConstants.IDLE_TIMEOUT, type = CommandType.LONG, since = "24.0",
78+
description = "How long an idle connection is held open, in milliseconds. Use 0 for infinite. "
79+
+ "If not set, the rule's current value is kept; use cleanupconnectionsettings to go back to "
80+
+ "the global setting network.loadbalancer.haproxy.idle.timeout.")
81+
private Long idleTimeout;
82+
83+
@Parameter(name = ApiConstants.KEEPALIVE_TIMEOUT, type = CommandType.LONG, since = "24.0",
84+
description = "How long an idle keepalive connection is held open waiting for the next request, "
85+
+ "in milliseconds. Only applies to rules the router serves in HTTP mode. "
86+
+ "If not set, the rule's current value is kept; use cleanupconnectionsettings to go back to "
87+
+ "following idletimeout.")
88+
private Long keepAliveTimeout;
89+
90+
@Parameter(name = ApiConstants.CLEAN_UP_CONNECTION_SETTINGS, type = CommandType.BOOLEAN, since = "24.0",
91+
description = "If true, the rule's own keepalive, idletimeout and keepalivetimeout are removed, so the rule goes back "
92+
+ "to the network offering and global settings. Any of the three passed in the same call are then set again, "
93+
+ "so a single call can reset one and keep the others. If false or not set, no action.")
94+
private Boolean cleanupConnectionSettings;
95+
7196
/////////////////////////////////////////////////////
7297
/////////////////// Accessors ///////////////////////
7398
/////////////////////////////////////////////////////
7499

75100
public String getAlgorithm() {
76101
return algorithm;
77102
}
103+
public Boolean getKeepAlive() {
104+
return keepAlive;
105+
}
106+
107+
public Long getIdleTimeout() {
108+
return idleTimeout;
109+
}
110+
public Long getKeepAliveTimeout() {
111+
return keepAliveTimeout;
112+
}
113+
114+
public boolean isCleanupConnectionSettings() {
115+
return Boolean.TRUE.equals(cleanupConnectionSettings);
116+
}
117+
118+
78119

79120
public String getDescription() {
80121
return description;

‎api/src/main/java/org/apache/cloudstack/api/response/LoadBalancerResponse.java‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -63,6 +63,18 @@ public class LoadBalancerResponse extends BaseResponse implements ControlledEnti
6363
@Param(description = "The ID of the guest Network the LB rule belongs to")
6464
private String networkId;
6565

66+
@SerializedName(ApiConstants.KEEPALIVE)
67+
@Param(description = "whether the load balancer keeps client connections open between requests, unset means the network offering's setting is used", since = "24.0")
68+
private Boolean keepAlive;
69+
70+
@SerializedName(ApiConstants.IDLE_TIMEOUT)
71+
@Param(description = "how long an idle connection is held open, in milliseconds, unset means the global setting is used", since = "24.0")
72+
private Long idleTimeout;
73+
74+
@SerializedName(ApiConstants.KEEPALIVE_TIMEOUT)
75+
@Param(description = "how long an idle keepalive connection is held open waiting for the next request, in milliseconds", since = "24.0")
76+
private Long keepAliveTimeout;
77+
6678
@SerializedName(ApiConstants.CIDR_LIST)
6779
@Param(description = "The CIDR list to allow traffic, all other CIDRs will be blocked. Multiple entries must be separated by a single comma character (,).")
6880
private String cidrList;
@@ -143,6 +155,18 @@ public void setCidrList(String cidrs) {
143155
this.cidrList = cidrs;
144156
}
145157

158+
public void setKeepAlive(Boolean keepAlive) {
159+
this.keepAlive = keepAlive;
160+
}
161+
162+
public void setIdleTimeout(Long idleTimeout) {
163+
this.idleTimeout = idleTimeout;
164+
}
165+
166+
public void setKeepAliveTimeout(Long keepAliveTimeout) {
167+
this.keepAliveTimeout = keepAliveTimeout;
168+
}
169+
146170
public void setAlgorithm(String algorithm) {
147171
this.algorithm = algorithm;
148172
}
Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
// Licensed to the Apache Software Foundation (ASF) under one
2+
// or more contributor license agreements. See the NOTICE file
3+
// distributed with this work for additional information
4+
// regarding copyright ownership. The ASF licenses this file
5+
// to you under the Apache License, Version 2.0 (the
6+
// "License"); you may not use this file except in compliance
7+
// with the License. You may obtain a copy of the License at
8+
//
9+
// http://www.apache.org/licenses/LICENSE-2.0
10+
//
11+
// Unless required by applicable law or agreed to in writing,
12+
// software distributed under the License is distributed on an
13+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
// KIND, either express or implied. See the License for the
15+
// specific language governing permissions and limitations
16+
// under the License.
17+
package org.apache.cloudstack.api.command.user.loadbalancer;
18+
19+
import org.apache.cloudstack.api.ApiErrorCode;
20+
import org.apache.cloudstack.api.ServerApiException;
21+
import org.junit.Assert;
22+
import org.junit.Test;
23+
import org.mockito.Mockito;
24+
import org.springframework.test.util.ReflectionTestUtils;
25+
26+
import com.cloud.exception.InvalidParameterValueException;
27+
import com.cloud.network.lb.LoadBalancingRulesService;
28+
29+
public class CreateLoadBalancerRuleCmdTest {
30+
31+
@Test
32+
public void testCreateRefusesBadTimeoutBeforeTheRuleExists() throws Exception {
33+
LoadBalancingRulesService lbService = Mockito.mock(LoadBalancingRulesService.class);
34+
Mockito.doThrow(new InvalidParameterValueException("keepalivetimeout must be 0 or greater"))
35+
.when(lbService).validateLoadBalancerConnectionSettings(null, -1L);
36+
37+
CreateLoadBalancerRuleCmd cmd = new CreateLoadBalancerRuleCmd();
38+
cmd._lbService = lbService;
39+
ReflectionTestUtils.setField(cmd, "keepAliveTimeout", -1L);
40+
41+
try {
42+
cmd.create();
43+
Assert.fail("A negative keepalivetimeout should be refused");
44+
} catch (ServerApiException e) {
45+
Assert.assertEquals(ApiErrorCode.PARAM_ERROR, e.getErrorCode());
46+
}
47+
Mockito.verify(lbService).validateLoadBalancerConnectionSettings(null, -1L);
48+
Mockito.verifyNoMoreInteractions(lbService);
49+
}
50+
}

0 commit comments

Comments
 (0)