Skip to content

Commit 92b1bb2

Browse files
author
Anya
committed
Fix markdown linting issues in circuit breaker documentation
Signed-off-by: Anya <[email protected]>
1 parent 2a1fa01 commit 92b1bb2

File tree

1 file changed

+37
-8
lines changed

1 file changed

+37
-8
lines changed

docs/transpot-layer/circuit-breaker.md

Lines changed: 37 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -22,13 +22,15 @@ Circuit breaking is an important pattern for creating resilient microservice app
2222
Please refer to [quickstart](https://kmesh.net/en/docs/setup/quick-start/) and change into ads mode
2323
- Ensure proper RBAC permissions are configured for KMesh and Istio CNI
2424
- Make sure the following RBAC permissions are set up for Istio CNI:
25+
2526
```bash
2627
kubectl create clusterrolebinding istio-cni-cluster-admin --clusterrole=cluster-admin --serviceaccount=istio-system:istio-cni
2728
```
2829

2930
## Deploy the test applications
3031

3132
1. Create the test service:
33+
3234
```yaml
3335
# sample-app.yaml
3436
apiVersion: apps/v1
@@ -71,6 +73,7 @@ spec:
7173
```
7274
7375
2. Deploy Fortio load tester:
76+
7477
```yaml
7578
# fortio.yaml
7679
apiVersion: apps/v1
@@ -108,23 +111,27 @@ spec:
108111
```
109112
110113
3. Enable Istio sidecar injection in the default namespace:
114+
111115
```bash
112116
kubectl label namespace default istio-injection=enabled --overwrite
113117
```
114118

115119
4. Apply the test service and Fortio deployment:
120+
116121
```bash
117122
kubectl apply -f sample-app.yaml
118123
kubectl apply -f fortio.yaml
119124
```
120125

121126
5. Verify the pods are running with Istio sidecars (should show 2/2 READY):
127+
122128
```bash
123129
kubectl get pods
124130
```
125131

126132
You should see output similar to:
127-
```
133+
134+
```text
128135
NAME READY STATUS RESTARTS AGE
129136
fortio-demo-57459944b-kk8c9 2/2 Running 0 72s
130137
httpbin-demo-54bcd57df4-hxqhn 2/2 Running 0 6s
@@ -133,6 +140,7 @@ httpbin-demo-54bcd57df4-hxqhn 2/2 Running 0 6s
133140
## Configure circuit breaker
134141

135142
Apply the circuit breaker configuration:
143+
136144
```yaml
137145
# circuit-breaker.yaml
138146
apiVersion: networking.istio.io/v1alpha3
@@ -155,16 +163,19 @@ spec:
155163
```
156164
157165
Apply the configuration:
166+
158167
```bash
159168
kubectl apply -f circuit-breaker.yaml
160169
```
161170

162171
Verify the destination rule was created correctly:
172+
163173
```bash
164174
kubectl get destinationrule httpbin-demo-cb -o yaml
165175
```
166176

167177
You should see output similar to:
178+
168179
```yaml
169180
apiVersion: networking.istio.io/v1
170181
kind: DestinationRule
@@ -204,7 +215,8 @@ kubectl exec -it $(kubectl get pod -l app=fortio-demo -o jsonpath={.items[0].met
204215
```
205216

206217
You should see output similar to:
207-
```
218+
219+
```text
208220
Fortio 1.69.5 running at 0 queries per second, 12->12 procs, for 50 calls: http://httpbin-demo:8000/get
209221
Starting at max qps with 20 thread(s) [gomax 12] for exactly 50 calls (2 per thread + 10)
210222
Ended after 410.534881ms : 50 calls. qps=121.79
@@ -240,7 +252,8 @@ kubectl exec -it $(kubectl get pod -l app=fortio-demo -o jsonpath={.items[0].met
240252
```
241253

242254
You should see output similar to:
243-
```
255+
256+
```text
244257
Fortio 1.69.5 running at 0 queries per second, 12->12 procs, for 50 calls: http://httpbin-demo:8000/get
245258
Starting at max qps with 20 thread(s) [gomax 12] for exactly 50 calls (2 per thread + 10)
246259
05:36:14.160 r109 [WRN] http_client.go:1151> Non ok http code, code=503, status="HTTP/1.1 503", thread=19, run=0
@@ -266,6 +279,7 @@ All done 50 calls (plus 0 warmup) 9.869 ms avg, 383.4 qps
266279
```
267280

268281
Now you can see the circuit breaker in action:
282+
269283
- Only 24% of requests succeeded with HTTP 200 responses
270284
- 76% of requests failed with HTTP 503 responses
271285
- The circuit breaker effectively rejected excess connections
@@ -280,7 +294,7 @@ kubectl exec $(kubectl get pod -l app=fortio-demo -o jsonpath={.items[0].metadat
280294

281295
You should see output that includes metrics related to the httpbin service. Look for entries with response codes and UO (Upstream Overflow) flags:
282296

283-
```
297+
```text
284298
istiocustom.istio_requests_total.reporter.source.source_workload.fortio-demo.source_canonical_service.fortio-demo.source_canonical_revision.latest.source_workload_namespace.default.source_principal.spiffe://cluster.local/ns/default/sa/default.source_app.fortio-demo.source_version.source_cluster.Kubernetes.destination_workload.httpbin-demo.destination_workload_namespace.default.destination_principal.spiffe://cluster.local/ns/default/sa/default.destination_app.httpbin-demo.destination_version.latest.destination_service.httpbin-demo.default.svc.cluster.local.destination_canonical_service.httpbin-demo.destination_canonical_revision.latest.destination_service_name.httpbin-demo.destination_service_namespace.default.destination_cluster.Kubernetes.request_protocol.http.response_code.200.grpc_response_status.response_flags.-.connection_security_policy.unknown: 12
285299
istiocustom.istio_requests_total.reporter.source.source_workload.fortio-demo.source_canonical_service.fortio-demo.source_canonical_revision.latest.source_workload_namespace.default.source_principal.unknown.source_app.fortio-demo.source_version.source_cluster.Kubernetes.destination_workload.unknown.destination_workload_namespace.unknown.destination_principal.unknown.destination_app.unknown.destination_version.unknown.destination_service.httpbin-demo.default.svc.cluster.local.destination_canonical_service.unknown.destination_canonical_revision.latest.destination_service_name.httpbin-demo.destination_service_namespace.default.destination_cluster.unknown.request_protocol.http.response_code.503.grpc_response_status.response_flags.UO.connection_security_policy.unknown: 38
286300
```
@@ -306,20 +320,23 @@ kubectl get pods -w
306320
```
307321

308322
Output:
309-
```
323+
324+
```text
310325
NAME READY STATUS RESTARTS AGE
311326
fortio-demo-57459944b-kk8c9 2/2 Running 0 72s
312327
httpbin-demo-54bcd57df4-hxqhn 2/2 Running 0 6s
313328
```
314329

315330
Now test the recovery:
331+
316332
```bash
317333
kubectl exec -it $(kubectl get pod -l app=fortio-demo -o jsonpath={.items[0].metadata.name}) \
318334
-c fortio -- fortio load -c 5 -qps 10 -n 50 -loglevel Warning http://httpbin-demo:8000/get
319335
```
320336

321337
Output:
322-
```
338+
339+
```text
323340
Fortio 1.69.5 running at 10 queries per second, 12->12 procs, for 50 calls: http://httpbin-demo:8000/get
324341
Starting at 10 qps with 5 thread(s) [gomax 12] : exactly 50, 10 calls each (total 50 + 0)
325342
05:37:00.310 r73 [WRN] http_client.go:1151> Non ok http code, code=503, status="HTTP/1.1 503", thread=2, run=0
@@ -352,7 +369,8 @@ kubectl logs $(kubectl get pod -l app=fortio-demo -o jsonpath={.items[0].metadat
352369
```
353370

354371
Example output:
355-
```
372+
373+
```text
356374
{"ts":1749274539.796037,"level":"info","r":1,"file":"updater.go","line":50,"msg":"Configmap flag value watching on /etc/fortio"}
357375
{"ts":1749274539.796072,"level":"crit","r":1,"file":"scli.go","line":83,"msg":"Unable to watch config/flag changes in /etc/fortio: dflag: error initializing fsnotify watcher"}
358376
{"ts":1749274539.796122,"level":"info","r":1,"file":"scli.go","line":122,"msg":"Starting","command":"Φορτίο","version":"1.69.5 h1:h+42fJ1HF61Jj+WgPmC+C2wPtM5Ct8JLHSLDyEgGID4= go1.23.9 amd64 linux","go-max-procs":12}
@@ -386,26 +404,30 @@ kubectl get destinationrule httpbin-demo-cb -o yaml
386404
## Understanding what happened
387405

388406
The circuit breaker configuration:
407+
389408
- Limits concurrent HTTP connections to 1 (`maxConnections: 1`)
390409
- Limits pending HTTP requests to 1 (`http1MaxPendingRequests: 1`)
391410
- Limits requests per connection to 1 (`maxRequestsPerConnection: 1`)
392411
- Configures outlier detection to eject hosts after 1 consecutive error
393412

394413
When the service is overloaded:
414+
395415
1. Circuit breaker trips when connection thresholds are exceeded
396416
2. Subsequent requests are rejected with HTTP 503 errors
397417
3. The service continues to handle requests within its configured capacity
398418
4. Only a small percentage of requests succeed (20-28% in our tests)
399419
5. After the `baseEjectionTime` (3s), the circuit will attempt to close and allow traffic again
400420

401421
Our test results clearly show the circuit breaker in action:
422+
402423
- Without circuit breaker: 100% success rate (all HTTP 200)
403424
- With circuit breaker: ~24% success rate (24% HTTP 200, 76% HTTP 503)
404425
- After recovery test: ~28% success rate (28% HTTP 200, 72% HTTP 503)
405426

406427
## Clean up
407428

408429
Remove test components:
430+
409431
```bash
410432
kubectl delete -f circuit-breaker.yaml
411433
kubectl delete -f sample-app.yaml
@@ -417,35 +439,41 @@ kubectl delete -f fortio.yaml
417439
If you encounter issues:
418440

419441
1. Check pod status:
442+
420443
```bash
421444
kubectl get pods
422445
kubectl describe pod <pod-name>
423446
```
424447

425448
2. Verify Istio sidecar injection:
449+
426450
```bash
427451
# Pods should show 2/2 READY if sidecar is properly injected
428452
kubectl get pods
429453
```
430454

431455
3. Check circuit breaker configuration:
456+
432457
```bash
433458
kubectl get destinationrule
434459
kubectl describe destinationrule httpbin-demo-cb
435460
```
436461

437462
4. Check for CNI issues:
463+
438464
```bash
439465
kubectl get pods -n istio-system
440466
kubectl logs -n istio-system $(kubectl get pod -n istio-system -l app=istio-cni-node -o jsonpath='{.items[0].metadata.name}')
441467
```
442468

443469
5. Ensure proper RBAC permissions:
470+
444471
```bash
445472
kubectl create clusterrolebinding istio-cni-cluster-admin --clusterrole=cluster-admin --serviceaccount=istio-system:istio-cni
446473
```
447474

448475
6. Check file descriptor limits in Istio CNI pods:
476+
449477
```bash
450478
kubectl apply -f - <<EOF
451479
apiVersion: v1
@@ -469,7 +497,8 @@ EOF
469497
```
470498

471499
7. View application logs:
500+
472501
```bash
473502
kubectl logs deploy/httpbin-demo
474503
kubectl logs $(kubectl get pod -l app=fortio-demo -o jsonpath={.items[0].metadata.name})
475-
```
504+
```

0 commit comments

Comments
 (0)