Skip to content

Commit

Permalink
No commit message
Browse files Browse the repository at this point in the history
  • Loading branch information
alice-yyds committed Oct 23, 2023
1 parent 596aa8f commit 4ec691e
Showing 1 changed file with 52 additions and 58 deletions.
110 changes: 52 additions & 58 deletions content/en/docs/kitex/Tutorials/service-governance/circuitbreaker.md
Original file line number Diff line number Diff line change
@@ -1,12 +1,8 @@
---
title: "Circuit Breaker"
date: 2021-08-31
weight: 5
keywords: ["Kitex", "Circuit Breaker"]
description: "This doc covers Kitex Circuit Breaker use guide and principle introduction."
---
# Kitex - Circuit Breaker | circuitbreaker

## Introduction

Kitex provides a default implementation of Circuit Breaker, but it's disabled by default.
Kitex provides a default implementation of Circuit Breaker, but its disabled by default.

The following document will introduce that how to enable circuit breaker and configure the policy.

Expand All @@ -16,41 +12,43 @@ The following document will introduce that how to enable circuit breaker and con

```go
import (
...
"github.com/cloudwego/kitex/client"
"github.com/cloudwego/kitex/pkg/circuitbreak"
"github.com/cloudwego/kitex/pkg/rpcinfo"
...
"github.com/cloudwego/kitex/client"
"github.com/cloudwego/kitex/pkg/circuitbreak"
"github.com/cloudwego/kitex/pkg/rpcinfo"
)

// GenServiceCBKeyFunc returns a key which determines the granularity of the CBSuite
func GenServiceCBKeyFunc(ri rpcinfo.RPCInfo) string {
// circuitbreak.RPCInfo2Key returns "$fromServiceName/$toServiceName/$method"
return circuitbreak.RPCInfo2Key(ri)
// circuitbreak.RPCInfo2Key returns "$fromServiceName/$toServiceName/$method"
return circuitbreak.RPCInfo2Key(ri)
}

func main() {
// build a new CBSuite with
cbs := circuitbreak.NewCBSuite(GenServiceCBKeyFunc)

var opts []client.Option
// add to the client options
opts = append(opts, client.WithCircuitBreaker(cbs))
// init client
cli, err := echoservice.NewClient(targetService, opts...)
// update circuit breaker config for a certain key (should be consistent with GenServiceCBKeyFunc)
// this can be called at any time, and will take effect for following requests
cbs.UpdateServiceCBConfig("fromServiceName/toServiceName/method", circuitbreak.CBConfig{
Enable: true,
ErrRate: 0.3, // requests will be blocked if error rate >= 30%
MinSample: 200, // this config takes effect if sampled requests are more than `MinSample`
})

// send requests with the client above
...
// build a new CBSuite with
cbs := circuitbreak.NewCBSuite(GenServiceCBKeyFunc)

var opts []client.Option
// add to the client options
opts = append(opts, client.WithCircuitBreaker(cbs))
// init client
cli, err := echoservice.NewClient(targetService, opts...)
// update circuit breaker config for a certain key (should be consistent with GenServiceCBKeyFunc)
// this can be called at any time, and will take effect for following requests
cbs.UpdateServiceCBConfig("fromServiceName/toServiceName/method", circuitbreak.CBConfig{
Enable: true,
ErrRate: 0.3, // requests will be blocked if error rate >= 30%
MinSample: 200, // this config takes effect if sampled requests are more than `MinSample`
})

// send requests with the client above
...
}


```

### Introduction
Expand All @@ -59,29 +57,24 @@ Kitex provides a set of CBSuite that encapsulates both service-level breaker and

- Service-Level Breaker

Statistics by service granularity, enabled via WithMiddleware.

The specific division of service granularity depends on the Circuit Breaker Key, which is the key for breaker statistics. When initializing the CBSuite, you need to pass it in **GenServiceCBKeyFunc**. The default key is `circuitbreak.RPCInfo2Key`, and the format of RPCInfo2Key is ` fromServiceName/toServiceName/method`.

- Statistics by service granularity, enabled via WithMiddleware.
- The specific service granularity depends on the Circuit Breaker Key, which is the key for breaker statistics. When initializing the CBSuite, you need to pass it in **GenServiceCBKeyFunc**. The default key is `circuitbreak.RPCInfo2Key`, and the format of RPCInfo2Key is `fromServiceName/toServiceName/method`.
- Instance-Level Breaker

Statistics by instance granularity, enabled via WithInstanceMW.

Instance-Level Breaker is used to solve the single-instance exception problem. If it's triggered, the framework will automatically retry the request.

Note that the premise of retry is that you need to enable breaker with **WithInstanceMW**, which will be executed after load balancing.

- Statistics by instance granularity, enabled via WithInstanceMW.
- Instance-Level Breaker is used to solve the single-instance exception problem. If it’s triggered, the framework will automatically retry the request.
- Note that the premise of retry is that you need to enable breaker with **WithInstanceMW**, which will be executed after load balancing.
- Threshold and **Threshold Change**

The default breaker threshold is `ErrRate: 0.5, MinSample: 200`, which means it's triggered by an error rate of 50% and requires the count of requests > 200.
The default breaker threshold is `ErrRate: 0.5, MinSample: 200`, which means its triggered by an error rate of 50% and requires the amount of requests > 200.

If you want to change the threshold, you can modify the `UpdateServiceCBConfig` and `UpdateInstanceCBConfig` in CBSuite.

## The Role of Circuit Breaker

When making RPC calls, errors are inevitable for downstream services.

When a downstream has a problem, if the upstream continues to make calls to it, it both prevents the downstream from recovering and wastes the upstream's resources.
When a downstream has a problem, if the upstream continues to make calls to it, it both prevents the downstream from recovering and wastes the upstreams resources.

To solve this problem, you can set up some dynamic switches that manually shut down calls to the downstream when it goes wrong.

Expand Down Expand Up @@ -115,16 +108,15 @@ In general, the transition of the three states is roughly as follows:
The timeout for cooling
| v
+-- detect succeed --<-[HALFOPEN]-->--+
```

### Trigger Strategies

Kitex provides three basic fuse triggering strategies by default:

- Number of consecutive errors reaches threshold (ConsecutiveTripFunc)

- Error count reaches threshold (ThresholdTripFunc)

- Error rate reaches the threshold (RateTripFunc)

Of course, you can write your own triggering strategy by implementing the TripFunc function.
Expand All @@ -141,13 +133,13 @@ Entering HALFOPEN when cooling is complete.

### Half-Open Strategy

During HALFOPEN, the breaker will let a request go every "interval", and after a "number" of consecutive successful requests, the breaker will become CLOSED; If any of them fail, it will become OPEN.
During HALFOPEN, the breaker will let a request go every interval, and after a number of consecutive successful requests, the breaker will become CLOSED; If any of them fail, it will become OPEN.

The process is a gradual-trial process.

Both the "interval" (DetectTimeout) and the "number" (DEFAULT_HALFOPEN_SUCCESSES) are configurable.
Both the interval (DetectTimeout) and the number (DEFAULT_HALFOPEN_SUCCESSES) are configurable.

## Statistics
## Statistics Algothrithm

### Default Config

Expand All @@ -171,18 +163,20 @@ As time moves, the oldest bucket in the window expires. The jitter occurs when t

As an example:

- You divide 10 seconds into 10 buckets, bucket 0 corresponds to a time of [0S, 1S), bucket 1 corresponds to [1S, 2S), ... , and bucket 9 corresponds to [9S, 10S).

- You divide 10 seconds into 10 buckets, bucket 0 corresponds to a time of [0S, 1S), bucket 1 corresponds to [1S, 2S), … , and bucket 9 corresponds to [9S, 10S).
- At 10.1S, a Succ is executed, and the following operations occur within the circuitbreaker.

- (1) detects that bucket 0 has expired and discards it;
- (2) creates a new bucket 10, corresponding to [10S, 11S);
- (3) puts that Succ into bucket 10.

- (1) detects that bucket 0 has expired and discards it;
- (2) creates a new bucket 10, corresponding to [10S, 11S);
- (3) puts that Succ into bucket 10.
- At 10.2S, you execute Successes() to query the number of successes in the window, then you get the actual statistics for [1S, 10.2S), not [0.2S, 10.2S).

Such jitter cannot be avoided if you use time-window-bucket statistics. A compromise approach is to increase the number of buckets, which can reduce the impact of jitter.

If 2000 buckets are divided, the impact of jitter on the overall data is at most 1/2000. In this package, the default number of buckets is 2000, the bucket time is 5ms, and the time window is 10S.

There are various technical solutions to avoid this problem, but they all introduce other problems, so if you have good ideas, please create a issue or PR.

---

# Internal Supplementary information

0 comments on commit 4ec691e

Please sign in to comment.