forked from ionos-enterprise/ibnbd
-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathREADME
408 lines (303 loc) · 15.6 KB
/
README
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
****************************
RDMA Transport (RTRS)
****************************
RTRS (RDMA Transport) is a reliable high speed transport library
which provides support to establish optimal number of connections
between client and server machines using RDMA (InfiniBand, RoCE, iWarp)
transport. It is optimized to transfer (read/write) IO blocks.
In its core interface it follows the BIO semantics of providing the
possibility to either write data from an sg list to the remote side
or to request ("read") data transfer from the remote side into a given
sg list.
RTRS provides I/O fail-over and load-balancing capabilities by using
multipath I/O (see "add_path" and "mp_policy" configuration entries).
RTRS is used by the RNBD (RDMA Network Block Device) modules.
======================
Client Sysfs Interface
======================
This chapter describes only the most important files of sysfs interface
on client side.
Entries under /sys/devices/virtual/rtrs-client/
================================================
When a user of RTRS API creates a new session, a directory entry with
the name of that session is created.
Entries under /sys/devices/virtual/rtrs-client/<session-name>/
===============================================================
add_path (RW)
-------------
Adds a new path (connection) to an existing session. Expected format is the
following:
<[source addr,]destination addr>
*addr ::= [ ip:<ipv4|ipv6> | gid:<gid> ]
max_reconnect_attempts (RW)
---------------------------
Maximum number reconnect attempts the client should make before giving up
after connection breaks unexpectedly.
mp_policy (RW)
--------------
Multipath policy specifies which path should be selected on each IO:
round-robin (0):
select path in per CPU round-robin manner.
min-inflight (1):
select path with minimum inflights.
Entries under /sys/devices/virtual/rtrs-client/<session-name>/paths/
=====================================================================
Each path belonging to a given session is listed here by its source and
destination address. When a new path is added to a session by writing to
the "add_path" entry, a directory <src@dst> is created.
Entries under /sys/devices/virtual/rtrs-client/<session-name>/paths/<src@dst>/
===============================================================================
state (R)
---------
Contains "connected" if the session is connected to the peer and fully
functional. Otherwise the file contains "disconnected"
reconnect (RW)
--------------
Write "1" to the file in order to reconnect the path.
Operation is blocking and returns 0 if reconnect was successful.
disconnect (RW)
---------------
Write "1" to the file in order to disconnect the path.
Operation blocks until RTRS path is disconnected.
remove_path (RW)
----------------
Write "1" to the file in order to disconnected and remove the path
from the session. Operation blocks until the path is disconnected
and removed from the session.
hca_name (R)
------------
Contains the the name of HCA the connection established on.
hca_port (R)
------------
Contains the port number of active port traffic is going through.
src_addr (R)
------------
Contains the source address of the path
dst_addr (R)
------------
Contains the destination address of the path
Entries under /sys/devices/virtual/rtrs-client/<session-name>/paths/<src@dst>/stats/
=====================================================================================
Write "0" to any file in that directory to reset corresponding statistics.
reset_all (RW)
--------------
Read will return usage help, write 0 will clear all the statistics.
cpu_migration (RW)
------------------
RTRS expects that each HCA IRQ is pinned to a separate CPU. If it's
not the case, the processing of an I/O response could be processed on a
different CPU than where it was originally submitted. This file shows
how many interrupts where generated on a non expected CPU.
"from:" is the CPU on which the IRQ was expected, but not generated.
"to:" is the CPU on which the IRQ was generated, but not expected.
reconnects (RW)
---------------
Contains 2 unsigned int values, the first one records number of successful
reconnects in the path lifetime, the second one records number of failed
reconnects in the path lifetime.
rdma (RW)
---------
Contains statistics regarding rdma operations and inflight operations.
The output consists of 6 values:
<read-count> <read-total-size> <write-count> <write-total-size> \
<inflights> <failovered>
======================
Server Sysfs Interface
======================
Entries under /sys/devices/virtual/rtrs-server/
================================================
When a user of RTRS API creates a new session on a client side, a
directory entry with the name of that session is created in here.
Entries under /sys/devices/virtual/rtrs-server/<session-name>/paths/
=====================================================================
When new path is created by writing to "add_path" entry on client side,
a directory entry named as <source address>@<destination address> is created
on server.
Entries under /sys/devices/virtual/rtrs-server/<session-name>/paths/<src@dst>/
===============================================================================
disconnect (RW)
---------------
When "1" is written to the file, the RTRS session is being disconnected.
Operations is non-blocking and returns control immediately to the caller.
hca_name (R)
------------
Contains the the name of HCA the connection established on.
hca_port (R)
------------
Contains the port number of active port traffic is going through.
src_addr (R)
------------
Contains the source address of the path
dst_addr (R)
------------
Contains the destination address of the path
Entries under /sys/devices/virtual/rtrs-server/<session-name>/paths/<src@dst>/stats/
=====================================================================================
When "0" is written to a file in this directory, the corresponding counters
will be reset.
rdma (RW)
---------
Contains statistics regarding rdma operations and inflight operations.
The output consists of 5 values:
<read-count> <read-total-size> <write-count> <write-total-size> <inflights>
==================
Transport protocol
==================
Overview
--------
An established connection between a client and a server is called rtrs
session. A session is associated with a set of memory chunks reserved on the
server side for a given client for rdma transfer. A session
consists of multiple paths, each representing a separate physical link
between client and server. Those are used for load balancing and failover.
Each path consists of as many connections (QPs) as there are cpus on
the client.
When processing an incoming write or read request, rtrs client uses memory
chunks reserved for him on the server side. Their number, size and addresses
need to be exchanged between client and server during the connection
establishment phase. Apart from the memory related information client needs to
inform the server about the session name and identify each path and connection
individually.
On an established session client sends to server write or read messages.
Server uses immediate field to tell the client which request is being
acknowledged and for errno. Client uses immediate field to tell the server
which of the memory chunks has been accessed and at which offset the message
can be found.
Module parameter always_invalidate is introduced for the security problem
discussed in LPC RDMA MC 2019. When always_invalidate=Y, on the server side we
invalidate each rdma buffer before we hand it over to RNBD server and
then pass it to the block layer. A new rkey is generated and registered for the
buffer after it returns back from the block layer and RNBD server.
The new rkey is sent back to the client along with the IO result.
The procedure is the default behaviour of the driver. This invalidation and
registration on each IO causes performance drop of up to 20%. A user of the
driver may choose to load the modules with this mechanism switched off
(always_invalidate=N), if he understands and can take the risk of a malicious
client being able to corrupt memory of a server it is connected to. This might
be a reasonable option in a scenario where all the clients and all the servers
are located within a secure datacenter.
Connection establishment
------------------------
1. Client starts establishing connections belonging to a path of a session one
by one via attaching RTRS_MSG_CON_REQ messages to the rdma_connect requests.
Those include uuid of the session and uuid of the path to be
established. They are used by the server to find a persisting session/path or
to create a new one when necessary. The message also contains the protocol
version and magic for compatibility, total number of connections per session
(as many as cpus on the client), the id of the current connection and
the reconnect counter, which is used to resolve the situations where
client is trying to reconnect a path, while server is still destroying the old
one.
2. Server accepts the connection requests one by one and attaches
RTRS_MSG_CONN_RSP messages to the rdma_accept. Apart from magic and
protocol version, the messages include error code, queue depth supported by
the server (number of memory chunks which are going to be allocated for that
session) and the maximum size of one io, RTRS_MSG_NEW_RKEY_F flags is set
when always_invalidate=Y.
3. After all connections of a path are established client sends to server the
RTRS_MSG_INFO_REQ message, containing the name of the session. This message
requests the address information from the server.
4. Server replies to the session info request message with RTRS_MSG_INFO_RSP,
which contains the addresses and keys of the RDMA buffers allocated for that
session.
5. Session becomes connected after all paths to be established are connected
(i.e. steps 1-4 finished for all paths requested for a session)
6. Server and client exchange periodically heartbeat messages (empty rdma
messages with an immediate field) which are used to detect a crash on remote
side or network outage in an absence of IO.
7. On any RDMA related error or in the case of a heartbeat timeout, the
corresponding path is disconnected, all the inflight IO are failed over to a
healthy path, if any, and the reconnect mechanism is triggered.
CLT SRV
*for each connection belonging to a path and for each path:
RTRS_MSG_CON_REQ ------------------->
<------------------- RTRS_MSG_CON_RSP
...
*after all connections are established:
RTRS_MSG_INFO_REQ ------------------->
<------------------- RTRS_MSG_INFO_RSP
*heartbeat is started from both sides:
-------------------> [RTRS_HB_MSG_IMM]
[RTRS_HB_MSG_ACK] <-------------------
[RTRS_HB_MSG_IMM] <-------------------
-------------------> [RTRS_HB_MSG_ACK]
IO path
-------
* Write (always_invalidate=N) *
1. When processing a write request client selects one of the memory chunks
on the server side and rdma writes there the user data, user header and the
RTRS_MSG_RDMA_WRITE message. Apart from the type (write), the message only
contains size of the user header. The client tells the server which chunk has
been accessed and at what offset the RTRS_MSG_RDMA_WRITE can be found by
using the IMM field.
2. When confirming a write request server sends an "empty" rdma message with
an immediate field. The 32 bit field is used to specify the outstanding
inflight IO and for the error code.
CLT SRV
usr_data + usr_hdr + rtrs_msg_rdma_write -----------------> [RTRS_IO_REQ_IMM]
[RTRS_IO_RSP_IMM] <----------------- (id + errno)
* Write (always_invalidate=Y) *
1. When processing a write request client selects one of the memory chunks
on the server side and rdma writes there the user data, user header and the
RTRS_MSG_RDMA_WRITE message. Apart from the type (write), the message only
contains size of the user header. The client tells the server which chunk has
been accessed and at what offset the RTRS_MSG_RDMA_WRITE can be found by
using the IMM field, Server invalidate rkey associated to the memory chunks
first, when it finishes, pass the IO to RNBD server module.
2. When confirming a write request server sends an "empty" rdma message with
an immediate field. The 32 bit field is used to specify the outstanding
inflight IO and for the error code. The new rkey is sent back using
SEND_WITH_IMM WR, client When it recived new rkey message, it validates
the message and finished IO after update rkey for the rbuffer, then post
back the recv buffer for later use.
CLT SRV
usr_data + usr_hdr + rtrs_msg_rdma_write -----------------> [RTRS_IO_REQ_IMM]
[RTRS_MSG_RKEY_RSP] <----------------- (RTRS_MSG_RKEY_RSP)
[RTRS_IO_RSP_IMM] <----------------- (id + errno)
* Read (always_invalidate=N)*
1. When processing a read request client selects one of the memory chunks
on the server side and rdma writes there the user header and the
RTRS_MSG_RDMA_READ message. This message contains the type (read), size of
the user header, flags (specifying if memory invalidation is necessary) and the
list of addresses along with keys for the data to be read into.
2. When confirming a read request server transfers the requested data first,
attaches an invalidation message if requested and finally an "empty" rdma
message with an immediate field. The 32 bit field is used to specify the
outstanding inflight IO and the error code.
CLT SRV
usr_hdr + rtrs_msg_rdma_read --------------> [RTRS_IO_REQ_IMM]
[RTRS_IO_RSP_IMM] <-------------- usr_data + (id + errno)
or in case client requested invalidation:
[RTRS_IO_RSP_IMM_W_INV] <-------------- usr_data + (INV) + (id + errno)
* Read (always_invalidate=Y)*
1. When processing a read request client selects one of the memory chunks
on the server side and rdma writes there the user header and the
RTRS_MSG_RDMA_READ message. This message contains the type (read), size of
the user header, flags (specifying if memory invalidation is necessary) and the
list of addresses along with keys for the data to be read into.
Server invalidate rkey associated to the memory chunks first, when it finishes,
passes the IO to RNBD server module.
2. When confirming a read request server transfers the requested data first,
attaches an invalidation message if requested and finally an "empty" rdma
message with an immediate field. The 32 bit field is used to specify the
outstanding inflight IO and the error code. The new rkey is sent back using
SEND_WITH_IMM WR, client When it recived new rkey message, it validates
the message and finished IO after update rkey for the rbuffer, then post
back the recv buffer for later use.
CLT SRV
usr_hdr + rtrs_msg_rdma_read --------------> [RTRS_IO_REQ_IMM]
[RTRS_IO_RSP_IMM] <-------------- usr_data + (id + errno)
[RTRS_MSG_RKEY_RSP] <----------------- (RTRS_MSG_RKEY_RSP)
or in case client requested invalidation:
[RTRS_IO_RSP_IMM_W_INV] <-------------- usr_data + (INV) + (id + errno)
=========================================
Contributors List(in alphabetical order)
=========================================
Danil Kipnis <[email protected]>
Fabian Holler <[email protected]>
Guoqing Jiang <[email protected]>
Jack Wang <[email protected]>
Kleber Souza <[email protected]>
Lutz Pogrell <[email protected]>
Milind Dumbare <[email protected]>
Roman Penyaev <[email protected]>