You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: follower-read.md
+57-30Lines changed: 57 additions & 30 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -5,69 +5,96 @@ summary: This document describes the use and implementation of Follower Read.
5
5
6
6
# Follower Read
7
7
8
-
When a read hotspot appears in a Region, the Region leader can become a read bottleneck for the entire system. In this situation, enabling the Follower Read feature can significantly reduce the load of the leader, and improve the throughput of the whole system by balancing the load among multiple followers. This document introduces the use and implementation mechanism of Follower Read.
8
+
In TiDB, to ensure high availability and data security, TiKV stores multiple replicas for each Region, one of which is the leader and the others are followers. By default, all read and write requests are processed by the leader. The Follower Read feature allows reading data from the follower replicas of a Region while maintaining strong consistency, thereby distributing the read pressure on the leader and improving the overall read throughput of the cluster.
9
9
10
-
## Overview
10
+
When performing Follower Read, TiDB selects the appropriate replica based on the topology information. Specifically, TiDB uses the `zone` label to determine whether a replica is a local replica: when the `zone` label of TiDB is the same as that of the target TiKV, TiDB considers the replica as a local replica. For more information, see [Schedule replicas by topology labels](schedule-replicas-by-topology-labels.md).
11
11
12
-
The Follower Read feature refers to using any follower replica of a Region to serve a read request under the premise of strongly consistent reads. This feature improves the throughput of the TiDB cluster and reduces the load of the leader. It contains a series of load balancing mechanisms that offload TiKV read loads from the leader replica to the follower replica in a Region. TiKV's Follower Read implementation provides users with strongly consistent reads.
12
+
By allowing followers to participate in data reading, Follower Read can achieve the following goals:
13
+
14
+
- Distribute read hotspots and reduce leader load.
15
+
- In multi-AZ or multi-datacenter deployments, prioritize reading local replicas to reduce cross-region traffic.
16
+
17
+
## Applicable scenarios
18
+
19
+
Follower Read is suitable for the following scenarios:
20
+
21
+
- Businesses with large read request volume and obvious read hotspots.
22
+
- Scenarios where it is desirable to prioritize reading local replicas in multi-AZ deployments to save bandwidth.
23
+
- In a read-write separation architecture, it is desirable to further improve the overall read performance of the cluster.
13
24
14
25
> **Note:**
15
26
>
16
-
> To achieve strongly consistent reads, the follower node currently needs to request the current execution progress from the leader node (that is`ReadIndex`), which causes an additional network request overhead. Therefore, the main benefits of Follower Read are to isolate read requests from write requests in the cluster and to increase overall read throughput.
27
+
> To ensure the strong consistency of the read results, Follower Read needs to communicate with the leader to confirm the current commit progress (that is, execute the Raft `ReadIndex` operation) before reading, which introduces an additional network interaction. Therefore, Follower Read is most effective when there are a large number of read requests or read-write isolation is required; however, the performance improvement may not be significant for low-latency single queries.
17
28
18
29
## Usage
19
30
20
-
To enable TiDB's Follower Read feature, modify the value of the `tidb_replica_read` variable as follows:
31
+
To enable TiDB's Follower Read feature, set the value of the `tidb_replica_read` variable to the desired value:
21
32
22
33
{{< copyable "sql" >}}
23
34
24
35
```sql
25
-
set [session | global] tidb_replica_read ='<target value>';
36
+
set [session | global] tidb_replica_read ='<目标值>';
26
37
```
27
38
28
39
Scope: SESSION | GLOBAL
29
40
30
41
Default: leader
31
42
32
-
This variable is used to set the expected data read mode.
43
+
This variable is used to set the expected data read mode. From v8.5.4, this variable only takes effect on read-only SQL statements.
33
44
34
-
- When you set the value of `tidb_replica_read` to `leader` or an empty string, TiDB maintains its default behavior and sends all read operations to the leader replica to perform.
35
-
- When you set the value of `tidb_replica_read` to `follower`, TiDB selects a follower replica of the Region to perform read operations. If the Region has learner replicas, TiDB also considers them for reads with the same priority. If no available follower or learner replicas exist for the current Region, TiDB reads from the leader replica.
36
-
- When the value of `tidb_replica_read` is set to `leader-and-follower`, TiDB can select any replicas to perform read operations. In this mode, read requests are load balanced between the leader and follower.
37
-
- When the value of `tidb_replica_read` is set to `prefer-leader`, TiDB prefers to select the leader replica to perform read operations. If the leader replica is obviously slow in processing read operations (such as caused by disk or network performance jitter), TiDB will select other available follower replicas to perform read operations.
38
-
- When the value of `tidb_replica_read` is set to `closest-replicas`, TiDB prefers to select a replica in the same availability zone to perform read operations, which can be a leader or a follower. If there is no replica in the same availability zone, TiDB reads from the leader replica.
39
-
- When the value of `tidb_replica_read` is set to `closest-adaptive`:
45
+
In scenarios where you need to save cross-region traffic by reading local replicas, the following configurations are recommended:
40
46
41
-
- If the estimated result of a read request is greater than or equal to the value of [`tidb_adaptive_closest_read_threshold`](/system-variables.md#tidb_adaptive_closest_read_threshold-new-in-v630), TiDB prefers to select a replica in the same availability zone for read operations. To avoid unbalanced distribution of read traffic across availability zones, TiDB dynamically detects the distribution of availability zones for all online TiDB and TiKV nodes. In each availability zone, the number of TiDB nodes whose `closest-adaptive` configuration takes effect is limited, which is always the same as the number of TiDB nodes in the availability zone with the fewest TiDB nodes, and the other TiDB nodes automatically read from the leader replica. For example, if TiDB nodes are distributed across 3 availability zones (A, B, and C), where A and B each contains 3 TiDB nodes and C contains only 2 TiDB nodes, the number of TiDB nodes whose `closest-adaptive` configuration takes effect in each availability zone is 2, and the other TiDB node in each of the A and B availability zones automatically selects the leader replica for read operations.
42
-
- If the estimated result of a read request is less than the value of [`tidb_adaptive_closest_read_threshold`](/system-variables.md#tidb_adaptive_closest_read_threshold-new-in-v630), TiDB can only select the leader replica for read operations.
47
+
- The default value `leader` provides the best performance.
48
+
-`closest-adaptive` saves traffic as much as possible with minimal performance loss.
49
+
-`closest-replicas` can save network traffic to the greatest extent.
43
50
44
-
- When you set the value of `tidb_replica_read` to `learner`, TiDB reads data from the learner replica. If no learner replica is available for the current Region, TiDB reads from an available leader or follower replica.
51
+
If you are currently using other configurations, refer to the following table to modify them to the recommended configurations:
45
52
46
-
<CustomContentplatform="tidb">
53
+
| Configuration being used | Recommended configuration to modify to |
54
+
| ------------- | ------------- |
55
+
|`follower`|`closest-replicas`|
56
+
|`leader-and-follower`|`closest-replicas`|
57
+
|`prefer-leader`|`closest-adaptive`|
58
+
|`learner`|`closest-replicas`|
47
59
48
-
> **Note:**
49
-
>
50
-
> When you set `tidb_replica_read` to `closest-replicas` or `closest-adaptive`, to ensure that replicas are distributed across availability zones according to the specified configuration, you need to configure `location-labels` for PD and set the correct `labels` for TiDB and TiKV according to [Schedule replicas by topology labels](/schedule-replicas-by-topology-labels.md). TiDB depends on the `zone` label to match TiKV nodes in the same availability zone, so you need to make sure that the `zone` label is included in the `location-labels` of PD and `zone` is included in the configuration of each TiDB and TiKV node. If your cluster is deployed using TiDB Operator, refer to [High availability of data](https://docs.pingcap.com/tidb-in-kubernetes/stable/configure-a-tidb-cluster#high-availability-of-data).
51
-
>
52
-
> For TiDB v7.5.0 and earlier versions:
53
-
>
54
-
> - If you set `tidb_replica_read` to `follower` and no follower or learner replicas are available, TiDB returns an error.
55
-
> - If you set `tidb_replica_read` to `learner` and no learner replicas are available, TiDB returns an error.
60
+
If you want to use a more precise read replica selection policy, refer to the complete list of optional configurations:
61
+
62
+
## Basic monitoring
56
63
57
-
</CustomContent>
64
+
By observing the [**TiDB** > **KV Request** > **Read Req Traffic** panel (new in v8.5.4)](/grafana-tidb-dashboard.md#kv-request), you can determine whether you need to use Follower Read and view the effect of saving traffic after enabling Follower Read.
58
65
59
66
## Implementation mechanism
60
67
61
68
Before the Follower Read feature was introduced, TiDB applied the strong leader principle and submitted all read and write requests to the leader node of a Region to handle. Although TiKV can distribute Regions evenly on multiple physical nodes, for each Region, only the leader can provide external services. The other followers can do nothing to handle read requests but receive the data replicated from the leader at all times and prepare for voting to elect a leader in case of a failover.
62
69
63
-
To allow data reading in the follower node without violating linearizability or affecting Snapshot Isolation in TiDB, the follower node needs to use `ReadIndex` of the Raft protocol to ensure that the read request can read the latest data that has been committed on the leader. At the TiDB level, the Follower Read feature simply needs to send the read request of a Region to a follower replica based on the load balancing policy.
70
+
Follower Read includes a series of load balancing mechanisms that offload TiKV read load from Region leader replicas to follower replicas. To allow data reading in the follower node without violating linearizability or affecting Snapshot Isolation in TiDB, the follower node needs to use `ReadIndex` of the Raft protocol to ensure that the read request can read the latest data that has been committed on the leader node. At the TiDB level, the Follower Read feature simply needs to send the read request of a Region to a follower replica based on the load balancing policy.
64
71
65
72
### Strongly consistent reads
66
73
67
-
When the follower node processes a read request, it first uses `ReadIndex` of the Raft protocol to interact with the leader of the Region, to obtain the latest commit index of the current Raft group. After the latest commit index of the leader is applied locally to the follower, the processing of a read request starts.
74
+
When the follower node processes a read request, it first uses `ReadIndex` of the Raft protocol to interact with the leader node of the Region to obtain the latest commit index (read index) of the current Raft group. After the latest commit index of the leader node is applied locally to the follower, the processing of a read request starts.
Because the Follower Read feature does not affect TiDB's Snapshot Isolation transaction isolation level, TiDB adopts the round-robin strategy to select the follower replica. Currently, for the coprocessor requests, the granularity of the Follower Read load balancing policy is at the connection level. For a TiDB client connected to a specific Region, the selected follower is fixed, and is switched only when it fails or the scheduling policy is adjusted.
80
+
The Follower Read feature does not affect TiDB's Snapshot Isolation transaction isolation level. TiDB selects a replica for the first time based on the configuration of `tidb_replica_read`. From the second retry, TiDB will prioritize ensuring successful reading. Therefore, when the selected follower node has an inaccessible fault or other errors, it will switch to the leader for service.
81
+
82
+
#### `leader`
83
+
84
+
- Select the leader replica for reading, regardless of replica location.
85
+
86
+
#### `closest-replicas`
87
+
88
+
- When the replica in the same AZ as TiDB is the leader node, Follower Read is not used.
89
+
- When the replica in the same AZ as TiDB is not the leader node, Follower Read is used.
90
+
91
+
#### `closest-adaptive`
92
+
93
+
- If the estimated return result is not large enough, use the `leader` policy and do not perform Follower Read.
94
+
- If the estimated return result is large enough, use the `closest-replicas` policy.
95
+
96
+
### Follower Read performance overhead
97
+
98
+
To ensure strong data consistency, Follower Read needs to perform a `ReadIndex` regardless of how much data is read, which inevitably consumes more TiKV CPU resources. Therefore, in small query (such as point query) scenarios, the performance loss of Follower Read is relatively more obvious. At the same time, because the traffic saved by local reading for small queries is limited, it is more recommended to use Follower Read in large query or batch reading scenarios.
72
99
73
-
However, for the non-coprocessor requests, such as a point query, the granularity of the Follower Read load balancing policy is at the transaction level. For a TiDB transaction on a specific Region, the selected follower is fixed, and is switched only when it fails or the scheduling policy is adjusted. If a transaction contains both point queries and coprocessor requests, the two types of requests are scheduled for reading separately according to the preceding scheduling policy. In this case, even if a coprocessor request and a point query are for the same Region, TiDB processes them as independent events.
100
+
When `tidb_replica_read` is `closest-adaptive`, TiDB does not use Follower Read for small queries, so the extra overhead of TiKV CPU compared to the `leader`policy is generally within +10% in various workloads.
Copy file name to clipboardExpand all lines: system-variables.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -5375,7 +5375,7 @@ SHOW WARNINGS;
5375
5375
- Type: Enumeration
5376
5376
- Default value: `leader`
5377
5377
- Possible values: `leader`, `follower`, `leader-and-follower`, `prefer-leader`, `closest-replicas`, `closest-adaptive`, and `learner`. The `learner` value is introduced in v6.6.0.
5378
-
- This variable is used to control where TiDB reads data.
5378
+
- This variable is used to control where TiDB reads data. From v8.5.4, this variable only takes effect on read-only SQL statements.
5379
5379
- For more details about usage and implementation, see [Follower read](/follower-read.md).
5380
5380
5381
5381
### tidb_restricted_read_only <span class="version-mark">New in v5.2.0</span>
0 commit comments