@@ -22,13 +22,15 @@ Circuit breaking is an important pattern for creating resilient microservice app
22
22
Please refer to [ quickstart] ( https://kmesh.net/en/docs/setup/quick-start/ ) and change into ads mode
23
23
- Ensure proper RBAC permissions are configured for KMesh and Istio CNI
24
24
- Make sure the following RBAC permissions are set up for Istio CNI:
25
+
25
26
``` bash
26
27
kubectl create clusterrolebinding istio-cni-cluster-admin --clusterrole=cluster-admin --serviceaccount=istio-system:istio-cni
27
28
```
28
29
29
30
## Deploy the test applications
30
31
31
32
1 . Create the test service:
33
+
32
34
``` yaml
33
35
# sample-app.yaml
34
36
apiVersion : apps/v1
71
73
` ` `
72
74
73
75
2. Deploy Fortio load tester:
76
+
74
77
` ` ` yaml
75
78
# fortio.yaml
76
79
apiVersion : apps/v1
@@ -108,23 +111,27 @@ spec:
108
111
` ` `
109
112
110
113
3. Enable Istio sidecar injection in the default namespace:
114
+
111
115
` ` ` bash
112
116
kubectl label namespace default istio-injection=enabled --overwrite
113
117
```
114
118
115
119
4 . Apply the test service and Fortio deployment:
120
+
116
121
``` bash
117
122
kubectl apply -f sample-app.yaml
118
123
kubectl apply -f fortio.yaml
119
124
```
120
125
121
126
5 . Verify the pods are running with Istio sidecars (should show 2/2 READY):
127
+
122
128
``` bash
123
129
kubectl get pods
124
130
```
125
131
126
132
You should see output similar to:
127
- ```
133
+
134
+ ``` text
128
135
NAME READY STATUS RESTARTS AGE
129
136
fortio-demo-57459944b-kk8c9 2/2 Running 0 72s
130
137
httpbin-demo-54bcd57df4-hxqhn 2/2 Running 0 6s
@@ -133,6 +140,7 @@ httpbin-demo-54bcd57df4-hxqhn 2/2 Running 0 6s
133
140
## Configure circuit breaker
134
141
135
142
Apply the circuit breaker configuration:
143
+
136
144
``` yaml
137
145
# circuit-breaker.yaml
138
146
apiVersion : networking.istio.io/v1alpha3
@@ -155,16 +163,19 @@ spec:
155
163
` ` `
156
164
157
165
Apply the configuration:
166
+
158
167
` ` ` bash
159
168
kubectl apply -f circuit-breaker.yaml
160
169
```
161
170
162
171
Verify the destination rule was created correctly:
172
+
163
173
``` bash
164
174
kubectl get destinationrule httpbin-demo-cb -o yaml
165
175
```
166
176
167
177
You should see output similar to:
178
+
168
179
``` yaml
169
180
apiVersion : networking.istio.io/v1
170
181
kind : DestinationRule
@@ -204,7 +215,8 @@ kubectl exec -it $(kubectl get pod -l app=fortio-demo -o jsonpath={.items[0].met
204
215
```
205
216
206
217
You should see output similar to:
207
- ```
218
+
219
+ ``` text
208
220
Fortio 1.69.5 running at 0 queries per second, 12->12 procs, for 50 calls: http://httpbin-demo:8000/get
209
221
Starting at max qps with 20 thread(s) [gomax 12] for exactly 50 calls (2 per thread + 10)
210
222
Ended after 410.534881ms : 50 calls. qps=121.79
@@ -240,7 +252,8 @@ kubectl exec -it $(kubectl get pod -l app=fortio-demo -o jsonpath={.items[0].met
240
252
```
241
253
242
254
You should see output similar to:
243
- ```
255
+
256
+ ``` text
244
257
Fortio 1.69.5 running at 0 queries per second, 12->12 procs, for 50 calls: http://httpbin-demo:8000/get
245
258
Starting at max qps with 20 thread(s) [gomax 12] for exactly 50 calls (2 per thread + 10)
246
259
05:36:14.160 r109 [WRN] http_client.go:1151> Non ok http code, code=503, status="HTTP/1.1 503", thread=19, run=0
@@ -266,6 +279,7 @@ All done 50 calls (plus 0 warmup) 9.869 ms avg, 383.4 qps
266
279
```
267
280
268
281
Now you can see the circuit breaker in action:
282
+
269
283
- Only 24% of requests succeeded with HTTP 200 responses
270
284
- 76% of requests failed with HTTP 503 responses
271
285
- The circuit breaker effectively rejected excess connections
@@ -280,7 +294,7 @@ kubectl exec $(kubectl get pod -l app=fortio-demo -o jsonpath={.items[0].metadat
280
294
281
295
You should see output that includes metrics related to the httpbin service. Look for entries with response codes and UO (Upstream Overflow) flags:
282
296
283
- ```
297
+ ``` text
284
298
istiocustom.istio_requests_total.reporter.source.source_workload.fortio-demo.source_canonical_service.fortio-demo.source_canonical_revision.latest.source_workload_namespace.default.source_principal.spiffe://cluster.local/ns/default/sa/default.source_app.fortio-demo.source_version.source_cluster.Kubernetes.destination_workload.httpbin-demo.destination_workload_namespace.default.destination_principal.spiffe://cluster.local/ns/default/sa/default.destination_app.httpbin-demo.destination_version.latest.destination_service.httpbin-demo.default.svc.cluster.local.destination_canonical_service.httpbin-demo.destination_canonical_revision.latest.destination_service_name.httpbin-demo.destination_service_namespace.default.destination_cluster.Kubernetes.request_protocol.http.response_code.200.grpc_response_status.response_flags.-.connection_security_policy.unknown: 12
285
299
istiocustom.istio_requests_total.reporter.source.source_workload.fortio-demo.source_canonical_service.fortio-demo.source_canonical_revision.latest.source_workload_namespace.default.source_principal.unknown.source_app.fortio-demo.source_version.source_cluster.Kubernetes.destination_workload.unknown.destination_workload_namespace.unknown.destination_principal.unknown.destination_app.unknown.destination_version.unknown.destination_service.httpbin-demo.default.svc.cluster.local.destination_canonical_service.unknown.destination_canonical_revision.latest.destination_service_name.httpbin-demo.destination_service_namespace.default.destination_cluster.unknown.request_protocol.http.response_code.503.grpc_response_status.response_flags.UO.connection_security_policy.unknown: 38
286
300
```
@@ -306,20 +320,23 @@ kubectl get pods -w
306
320
```
307
321
308
322
Output:
309
- ```
323
+
324
+ ``` text
310
325
NAME READY STATUS RESTARTS AGE
311
326
fortio-demo-57459944b-kk8c9 2/2 Running 0 72s
312
327
httpbin-demo-54bcd57df4-hxqhn 2/2 Running 0 6s
313
328
```
314
329
315
330
Now test the recovery:
331
+
316
332
``` bash
317
333
kubectl exec -it $( kubectl get pod -l app=fortio-demo -o jsonpath={.items[0].metadata.name}) \
318
334
-c fortio -- fortio load -c 5 -qps 10 -n 50 -loglevel Warning http://httpbin-demo:8000/get
319
335
```
320
336
321
337
Output:
322
- ```
338
+
339
+ ``` text
323
340
Fortio 1.69.5 running at 10 queries per second, 12->12 procs, for 50 calls: http://httpbin-demo:8000/get
324
341
Starting at 10 qps with 5 thread(s) [gomax 12] : exactly 50, 10 calls each (total 50 + 0)
325
342
05:37:00.310 r73 [WRN] http_client.go:1151> Non ok http code, code=503, status="HTTP/1.1 503", thread=2, run=0
@@ -352,7 +369,8 @@ kubectl logs $(kubectl get pod -l app=fortio-demo -o jsonpath={.items[0].metadat
352
369
```
353
370
354
371
Example output:
355
- ```
372
+
373
+ ``` text
356
374
{"ts":1749274539.796037,"level":"info","r":1,"file":"updater.go","line":50,"msg":"Configmap flag value watching on /etc/fortio"}
357
375
{"ts":1749274539.796072,"level":"crit","r":1,"file":"scli.go","line":83,"msg":"Unable to watch config/flag changes in /etc/fortio: dflag: error initializing fsnotify watcher"}
358
376
{"ts":1749274539.796122,"level":"info","r":1,"file":"scli.go","line":122,"msg":"Starting","command":"Φορτίο","version":"1.69.5 h1:h+42fJ1HF61Jj+WgPmC+C2wPtM5Ct8JLHSLDyEgGID4= go1.23.9 amd64 linux","go-max-procs":12}
@@ -386,26 +404,30 @@ kubectl get destinationrule httpbin-demo-cb -o yaml
386
404
## Understanding what happened
387
405
388
406
The circuit breaker configuration:
407
+
389
408
- Limits concurrent HTTP connections to 1 (` maxConnections: 1 ` )
390
409
- Limits pending HTTP requests to 1 (` http1MaxPendingRequests: 1 ` )
391
410
- Limits requests per connection to 1 (` maxRequestsPerConnection: 1 ` )
392
411
- Configures outlier detection to eject hosts after 1 consecutive error
393
412
394
413
When the service is overloaded:
414
+
395
415
1 . Circuit breaker trips when connection thresholds are exceeded
396
416
2 . Subsequent requests are rejected with HTTP 503 errors
397
417
3 . The service continues to handle requests within its configured capacity
398
418
4 . Only a small percentage of requests succeed (20-28% in our tests)
399
419
5 . After the ` baseEjectionTime ` (3s), the circuit will attempt to close and allow traffic again
400
420
401
421
Our test results clearly show the circuit breaker in action:
422
+
402
423
- Without circuit breaker: 100% success rate (all HTTP 200)
403
424
- With circuit breaker: ~ 24% success rate (24% HTTP 200, 76% HTTP 503)
404
425
- After recovery test: ~ 28% success rate (28% HTTP 200, 72% HTTP 503)
405
426
406
427
## Clean up
407
428
408
429
Remove test components:
430
+
409
431
``` bash
410
432
kubectl delete -f circuit-breaker.yaml
411
433
kubectl delete -f sample-app.yaml
@@ -417,35 +439,41 @@ kubectl delete -f fortio.yaml
417
439
If you encounter issues:
418
440
419
441
1 . Check pod status:
442
+
420
443
``` bash
421
444
kubectl get pods
422
445
kubectl describe pod < pod-name>
423
446
```
424
447
425
448
2 . Verify Istio sidecar injection:
449
+
426
450
``` bash
427
451
# Pods should show 2/2 READY if sidecar is properly injected
428
452
kubectl get pods
429
453
```
430
454
431
455
3 . Check circuit breaker configuration:
456
+
432
457
``` bash
433
458
kubectl get destinationrule
434
459
kubectl describe destinationrule httpbin-demo-cb
435
460
```
436
461
437
462
4 . Check for CNI issues:
463
+
438
464
``` bash
439
465
kubectl get pods -n istio-system
440
466
kubectl logs -n istio-system $( kubectl get pod -n istio-system -l app=istio-cni-node -o jsonpath=' {.items[0].metadata.name}' )
441
467
```
442
468
443
469
5 . Ensure proper RBAC permissions:
470
+
444
471
``` bash
445
472
kubectl create clusterrolebinding istio-cni-cluster-admin --clusterrole=cluster-admin --serviceaccount=istio-system:istio-cni
446
473
```
447
474
448
475
6 . Check file descriptor limits in Istio CNI pods:
476
+
449
477
``` bash
450
478
kubectl apply -f - << EOF
451
479
apiVersion: v1
469
497
```
470
498
471
499
7 . View application logs:
500
+
472
501
``` bash
473
502
kubectl logs deploy/httpbin-demo
474
503
kubectl logs $( kubectl get pod -l app=fortio-demo -o jsonpath={.items[0].metadata.name})
475
- ```
504
+ ```
0 commit comments