Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
152 changes: 152 additions & 0 deletions docs/application-layer/circuit-breaker.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,152 @@
---
title: Circuit Breaker
sidebar_position: 10
---

This document provides a step-by-step guide on how to test the circuit breaking functionality of kmesh. It covers deploying the necessary components, configuring traffic rules, and observing the circuit breaking behavior.

## Step 1. Deploy kmesh

Please read [Quick Start](https://kmesh.net/docs/setup/quick-start) to complete the deployment of kmesh.

## Step 2. Deploy fortio and httpbin

``` sh
kubectl apply -f -<<EOF
apiVersion: v1
kind: Service
metadata:
name: fortio
labels:
app: fortio
service: fortio
spec:
ports:
- port: 8080
name: http
selector:
app: fortio
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: fortio-deploy
spec:
replicas: 1
selector:
matchLabels:
app: fortio
template:
metadata:
annotations:
# This annotation causes Envoy to serve cluster.outbound statistics via 15000/stats
# in addition to the stats normally served by Istio. The Circuit Breaking example task
# gives an example of inspecting Envoy stats via proxy config.
proxy.istio.io/config: |-
proxyStatsMatcher:
inclusionPrefixes:
- "cluster.outbound"
- "cluster_manager"
- "listener_manager"
- "server"
- "cluster.xds-grpc"
labels:
app: fortio
spec:
containers:
- name: fortio
image: fortio/fortio:latest_release
imagePullPolicy: Always
ports:
- containerPort: 8080
name: http-fortio
- containerPort: 8079
name: grpc-ping
EOF
```

```sh
kubectl apply -f samples/httpbin/httpbin.yaml
```

## Step 3. Configure waypoint for httpbin

First, if you haven't installed the Kubernetes Gateway API CRDs, run the following command to install.

``` sh
kubectl get crd gateways.gateway.networking.k8s.io &> /dev/null || \
{ kubectl kustomize "github.com/kubernetes-sigs/gateway-api/config/crd/experimental?ref=444631bfe06f3bcca5d0eadf1857eac1d369421d" | kubectl apply -f -; }
```

Next, create a dedicated Waypoint proxy for the `httpbin` service and label the service to direct its traffic through this Waypoint.

```sh
kmeshctl waypoint apply -n default --name httpbin-waypoint --image ghcr.io/kmesh-net/waypoint:latest
kubectl label service httpbin istio.io/use-waypoint=httpbin-waypoint
```

## Step 4. Configure DestinationRule

```sh
kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
name: httpbin
spec:
host: httpbin
trafficPolicy:
connectionPool:
tcp:
# Maximum number of TCP connections to the target service
maxConnections: 1
http:
# Maximum number of pending HTTP requests
http1MaxPendingRequests: 1
# Maximum number of requests allowed per connection.
maxRequestsPerConnection: 1
outlierDetection:
# Circuit breaker settings
consecutive5xxErrors: 1
interval: 1s
baseEjectionTime: 3m
maxEjectionPercent: 100
EOF
```

## Step 5. View the cds configuration in waypoint through istioctl

First, get the name of the Waypoint pod.

```sh
export WAYPOINT_POD=$(kubectl get pod -l gateway.networking.k8s.io/gateway-name=httpbin-waypoint -o jsonpath='{.items[0].metadata.name}')

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

For clarity and robustness, it's good practice to explicitly specify the namespace when getting Kubernetes resources. Since the waypoint proxy is created in the default namespace, please add -n default to the command.

Suggested change
export WAYPOINT_POD=$(kubectl get pod -l gateway.networking.k8s.io/gateway-name=httpbin-waypoint -o jsonpath='{.items[0].metadata.name}')
export WAYPOINT_POD=$(kubectl get pod -n default -l gateway.networking.k8s.io/gateway-name=httpbin-waypoint -o jsonpath='{.items[0].metadata.name}')

```

Then, view the configuration.

```sh
istioctl proxy-config all $WAYPOINT_POD
```

## Step 6. Access through fortio to see the actual phenomenon

Now, let's use `fortio` to send a burst of traffic to `httpbin`.

```sh
export FORTIO_POD=$(kubectl get pods -l app=fortio -o 'jsonpath={.items[0].metadata.name}')

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

For clarity and robustness, it's good practice to explicitly specify the namespace. The fortio pod is deployed in the default namespace, so please add -n default to this command.

Suggested change
export FORTIO_POD=$(kubectl get pods -l app=fortio -o 'jsonpath={.items[0].metadata.name}')
export FORTIO_POD=$(kubectl get pods -n default -l app=fortio -o 'jsonpath={.items[0].metadata.name}')


kubectl exec "$FORTIO_POD" -c fortio -- /usr/bin/fortio load -c 5 -qps 0 -n 50 -loglevel Warning http://httpbin:8000/get
```

You should see some requests failing with 503 errors, indicating that the circuit breaker is functioning as expected.

```sh
...
IP addresses distribution:
10.96.56.163:8000: 33
Code 200 : 19 (38.0 %)
Code 503 : 31 (62.0 %)
Response Header Sizes : count 50 avg 114.38 +/- 146.1 min 0 max 301 sum 5719
Response Body/Total Sizes : count 50 avg 382.48 +/- 46.6 min 346 max 442 sum 19124
All done 50 calls (plus 0 warmup) 3.162 ms avg, 1247.0 qps
```
225 changes: 225 additions & 0 deletions docs/application-layer/local-rate-limiting.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,225 @@
---
title: Local Rate Limiting
sidebar_position: 11
---

This document provides a step-by-step guide on how to test the local rate limiting functionality of kmesh. It covers deploying the necessary components, configuring traffic rules, and observing the rate limiting behavior.

## Step 1. Deploy Kmesh and istiod (>=1.24)

Please read [Quick Start](https://kmesh.net/docs/setup/quick-start) to complete the deployment of kmesh.

## Step 2. Deploy sleep and httpbin

We will deploy `httpbin` as the backend service for receiving requests and `sleep` as the client for sending requests.

``` sh
kubectl apply -f samples/sleep/sleep.yaml
kubectl apply -f samples/httpbin/httpbin.yaml
```

## Step 3. Deploy waypoint for httpbin

First, if you haven't installed the Kubernetes Gateway API CRDs, run the following command to install.

``` sh
kubectl get crd gateways.gateway.networking.k8s.io &> /dev/null || \
{ kubectl kustomize "github.com/kubernetes-sigs/gateway-api/config/crd/experimental?ref=444631bfe06f3bcca5d0eadf1857eac1d369421d" | kubectl apply -f -; }
```

Next, create a dedicated Waypoint proxy for the `httpbin` service and label the service to direct its traffic through this Waypoint.

```sh
kmeshctl waypoint apply -n default --name httpbin-waypoint --image ghcr.io/kmesh-net/waypoint:latest

kubectl label service httpbin istio.io/use-waypoint=httpbin-waypoint
```

## Step 4. Deploy envoyFilter

This `EnvoyFilter` resource injects a local rate-limiting filter into the `httpbin` service's Waypoint proxy. The filter is configured with the following rules:

- A request with the header `quota: low` will be limited to **1 request per 300 seconds**.
- A request with the header `quota: medium` will be limited to **3 requests per 300 seconds**.
- Other requests will be subject to a default limit of **10 requests per 300 seconds**.

The `workloadSelector` ensures that this filter is applied only to the `httpbin-waypoint` proxy.

```sh
kubectl apply -f -<<EOF
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
name: httpbin.ratelimit
namespace: default
spec:
configPatches:
- applyTo: HTTP_FILTER
match:
context: SIDECAR_INBOUND
listener:
filterChain:
filter:
name: envoy.filters.network.http_connection_manager
subFilter:
name: envoy.filters.http.router
proxy:
proxyVersion: ^1.*
patch:
operation: INSERT_BEFORE
value:
name: envoy.filters.http.local_ratelimit
typed_config:
'@type': type.googleapis.com/udpa.type.v1.TypedStruct
type_url: type.googleapis.com/envoy.extensions.filters.http.local_ratelimit.v3.LocalRateLimit
value:
customResponseBody: local_rate_limited
statPrefix: http_local_rate_limiter
- applyTo: HTTP_ROUTE
match:
proxy:
proxyVersion: ^1.*
routeConfiguration:
vhost:
name: inbound|http|8000
route:
name: default
patch:
operation: MERGE
value:
typed_per_filter_config:
envoy.filters.http.local_ratelimit:
'@type': type.googleapis.com/udpa.type.v1.TypedStruct
type_url: type.googleapis.com/envoy.extensions.filters.http.local_ratelimit.v3.LocalRateLimit
value:
customResponseBody: local_rate_limited
descriptors:
- entries:
- key: header_match
value: Service[httpbin.default]-User[none]-Id[3100861967]
tokenBucket:
fillInterval: 300s
maxTokens: 1
tokensPerFill: 1
- entries:
- key: header_match
value: Service[httpbin.default]-User[none]-Id[4123289408]
tokenBucket:
fillInterval: 300s
maxTokens: 3
tokensPerFill: 3
filterEnabled:
defaultValue:
numerator: 100
runtimeKey: local_rate_limit_enabled
filterEnforced:
defaultValue:
numerator: 100
runtimeKey: local_rate_limit_enforced
rateLimits:
- actions:
- headerValueMatch:
descriptorValue: Service[httpbin.default]-User[none]-Id[3100861967]
headers:
- exactMatch: low
name: quota
- actions:
- headerValueMatch:
descriptorValue: Service[httpbin.default]-User[none]-Id[4123289408]
headers:
- exactMatch: medium
name: quota
responseHeadersToAdd:
- append: false
header:
key: x-local-rate-limit
value: "true"
statPrefix: http_local_rate_limiter
tokenBucket:
fillInterval: 300s
maxTokens: 10
tokensPerFill: 10
workloadSelector:
labels:
gateway.networking.k8s.io/gateway-name: httpbin-waypoint
EOF
```

## Step 5. View the envoy filter configuration in waypoint through istioctl

To verify the configuration, first get the name of the Waypoint pod, then use `istioctl` to inspect its configuration.

```sh
export WAYPOINT_POD=$(kubectl get pod -l gateway.networking.k8s.io/gateway-name=httpbin-waypoint -o jsonpath='{.items[0].metadata.name}')

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

For clarity and robustness, it's good practice to explicitly specify the namespace when getting Kubernetes resources. Since the waypoint proxy is created in the default namespace, please add -n default to the command.

Suggested change
export WAYPOINT_POD=$(kubectl get pod -l gateway.networking.k8s.io/gateway-name=httpbin-waypoint -o jsonpath='{.items[0].metadata.name}')
export WAYPOINT_POD=$(kubectl get pod -n default -l gateway.networking.k8s.io/gateway-name=httpbin-waypoint -o jsonpath='{.items[0].metadata.name}')

istioctl proxy-config all $WAYPOINT_POD -ojson | grep ratelimit -A 20
```

## Step 6. Find the following results, which means the configuration has been sent to waypoint

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

This heading is a bit long and reads like an instruction rather than a section title. Consider rephrasing it to be more concise and descriptive, for example: ## Step 6. Verify Envoy Filter Configuration.

Suggested change
## Step 6. Find the following results, which means the configuration has been sent to waypoint
## Step 6. Verify Envoy Filter Configuration


```sh
"envoy.filters.http.local_ratelimit": {
"@type": "type.googleapis.com/udpa.type.v1.TypedStruct",
"type_url": "type.googleapis.com/envoy.extensions.filters.http.local_ratelimit.v3.LocalRateLimit",
"value": {
"stat_prefix": "http_local_rate_limiter",
"token_bucket": {
"max_tokens": 10,
"tokens_per_fill": 10,
"fill_interval": "300s"
},
"filter_enabled": {
"default_value": {
"numerator": 100
},
"runtime_key": "local_rate_limit_enabled"
},
"filter_enforced": {
"default_value": {
"numerator": 100
},
"runtime_key": "local_rate_limit_enforced"
},
"response_headers_to_add": [
```

## Step 7. Access httpbin through sleep to see if the rate limit is working

Now, let's send requests from the `sleep` pod to the `httpbin` service to test the rate limit rules.

First, get the name of the `sleep` pod:

```sh
export SLEEP_POD=$(kubectl get pod -l app=sleep -o jsonpath='{.items[0].metadata.name}')

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

For clarity and robustness, it's good practice to explicitly specify the namespace. The sleep pod is deployed in the default namespace, so please add -n default to this command.

Suggested change
export SLEEP_POD=$(kubectl get pod -l app=sleep -o jsonpath='{.items[0].metadata.name}')
export SLEEP_POD=$(kubectl get pod -n default -l app=sleep -o jsonpath='{.items[0].metadata.name}')

```

### Test Case 1: "medium" quota

The rule for `quota: medium` allows 3 requests. The fourth request should be rate-limited.

```sh
kubectl exec -it $SLEEP_POD -- curl -H 'quota:medium' http://httpbin:8000/headers
kubectl exec -it $SLEEP_POD -- curl -H 'quota:medium' http://httpbin:8000/headers
kubectl exec -it $SLEEP_POD -- curl -H 'quota:medium' http://httpbin:8000/headers
kubectl exec -it $SLEEP_POD -- curl -H 'quota:medium' http://httpbin:8000/headers
Comment on lines +200 to +203

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The -it flags are used to run a command in an interactive terminal session. Since these curl commands are non-interactive, the flags are unnecessary. Removing them would also make the commands consistent with others in the documentation (e.g., in circuit-breaker.md). This applies to all kubectl exec commands in this section.

Suggested change
kubectl exec -it $SLEEP_POD -- curl -H 'quota:medium' http://httpbin:8000/headers
kubectl exec -it $SLEEP_POD -- curl -H 'quota:medium' http://httpbin:8000/headers
kubectl exec -it $SLEEP_POD -- curl -H 'quota:medium' http://httpbin:8000/headers
kubectl exec -it $SLEEP_POD -- curl -H 'quota:medium' http://httpbin:8000/headers
kubectl exec $SLEEP_POD -- curl -H 'quota:medium' http://httpbin:8000/headers
kubectl exec $SLEEP_POD -- curl -H 'quota:medium' http://httpbin:8000/headers
kubectl exec $SLEEP_POD -- curl -H 'quota:medium' http://httpbin:8000/headers
kubectl exec $SLEEP_POD -- curl -H 'quota:medium' http://httpbin:8000/headers

```

Expected output for the fourth command:

``` sh
local_rate_limited
```

### Test Case 2: "low" quota

The rule for `quota: low` allows only 1 request. The second request should be rate-limited.

```sh
kubectl exec -it $SLEEP_POD -- curl -H 'quota:low' http://httpbin:8000/headers
kubectl exec -it $SLEEP_POD -- curl -H 'quota:low' http://httpbin:8000/headers
Comment on lines +217 to +218

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

As mentioned previously, the -it flags are not needed for these non-interactive curl commands.

Suggested change
kubectl exec -it $SLEEP_POD -- curl -H 'quota:low' http://httpbin:8000/headers
kubectl exec -it $SLEEP_POD -- curl -H 'quota:low' http://httpbin:8000/headers
kubectl exec $SLEEP_POD -- curl -H 'quota:low' http://httpbin:8000/headers
kubectl exec $SLEEP_POD -- curl -H 'quota:low' http://httpbin:8000/headers

```

Expected output for the second command:

``` sh
local_rate_limited
```