Provide support for direct_reads with async_io #10197

akankshamahajan15 · 2022-06-17T21:17:47Z

Summary: Provide support for use_direct_reads with async_io.

TestPlan:

Updated unit tests
db_bench: Results in Provide support for direct_reads with async_io #10197 (comment)
db_stress

export CRASH_TEST_EXT_ARGS=" --async_io=1 --use_direct_reads=1"
make crash_test -j

Ran db_bench on previous RocksDB version before any async_io implementation (as there have many changes in different PRs in this area) Provide support for direct_reads with async_io #10197 (comment).

akankshamahajan15 · 2022-06-17T21:29:55Z

Fill DB:

./db_bench -db=/tmp/prefix_scan_prefetch_main1 -benchmarks="fillseq" -key_size=32 -value_size=512 -num=50000000 -use_direct_io_for_flush_and_compaction=true -target_file_size_base=16777216

With async_io disabled:

sudo sh -c "/usr/bin/echo 1 > /proc/sys/vm/drop_caches";

./db_bench -use_existing_db=true -db=/tmp/prefix_scan_prefetch_main1 -benchmarks="seekrandom" -key_size=32 -value_size=512 -num=50000000 -use_direct_reads=true -seek_nexts=327680 -duration=30 -ops_between_duration_checks=1 -async_io=0
Set seed to 1655500928347361 because --seed was 0
Initializing RocksDB Options from the specified file
Initializing RocksDB Options from command-line flags
RocksDB:    version 7.4
Date:       Fri Jun 17 14:22:08 2022
CPU:        32 * Intel Xeon Processor (Skylake)
CPUCache:   16384 KB
Keys:       32 bytes each (+ 0 bytes user-defined timestamp)
Values:     512 bytes each (256 bytes after compression)
Entries:    50000000
Prefix:    0 bytes
Keys per prefix:    0
RawSize:    25939.9 MB (estimated)
FileSize:   13732.9 MB (estimated)
Write rate: 0 bytes/second
Read rate: 0 ops/second
Compression: Snappy
Compression sampling rate: 0
Memtablerep: SkipListFactory
Perf Level: 1
------------------------------------------------
DB path: [/tmp/prefix_scan_prefetch_main1]
seekrandom   :  490593.048 micros/op 2 ops/sec 30.417 seconds 62 operations;  346.5 MB/s (62 of 62 found)

With async_io enabled:

./db_bench -use_existing_db=true -db=/tmp/prefix_scan_prefetch_main1 -benchmarks="seekrandom" -key_size=32 -value_size=512 -num=50000000 -use_direct_reads=true -seek_nexts=327680 -duration=30 -ops_between_duration_checks=1 -async_io=1
Set seed to 1655501060424336 because --seed was 0

Initializing RocksDB Options from the specified file
Initializing RocksDB Options from command-line flags
RocksDB:    version 7.4
Date:       Fri Jun 17 14:24:20 2022
CPU:        32 * Intel Xeon Processor (Skylake)
CPUCache:   16384 KB
Keys:       32 bytes each (+ 0 bytes user-defined timestamp)
Values:     512 bytes each (256 bytes after compression)
Entries:    50000000
Prefix:    0 bytes
Keys per prefix:    0
RawSize:    25939.9 MB (estimated)
FileSize:   13732.9 MB (estimated)
Write rate: 0 bytes/second
Read rate: 0 ops/second
Compression: Snappy
Compression sampling rate: 0
Memtablerep: SkipListFactory
Perf Level: 1
------------------------------------------------
DB path: [/tmp/prefix_scan_prefetch_main1]
seekrandom   :  352502.988 micros/op 2 ops/sec 30.315 seconds 86 operations;  477.5 MB/s (86 of 86 found)

file/random_access_file_reader.cc

facebook-github-bot · 2022-06-17T22:31:08Z

@akankshamahajan15 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot · 2022-06-18T03:09:05Z

@akankshamahajan15 has updated the pull request. You must reimport the pull request before landing.

facebook-github-bot · 2022-06-18T03:26:49Z

@akankshamahajan15 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot · 2022-06-20T18:17:32Z

@akankshamahajan15 has updated the pull request. You must reimport the pull request before landing.

facebook-github-bot · 2022-06-20T18:19:38Z

@akankshamahajan15 has updated the pull request. You must reimport the pull request before landing.

akankshamahajan15 · 2022-06-20T18:22:40Z

file/random_access_file_reader.cc

facebook-github-bot · 2022-06-20T18:28:08Z

@akankshamahajan15 has updated the pull request. You must reimport the pull request before landing.

facebook-github-bot · 2022-06-20T18:37:47Z

@akankshamahajan15 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

akankshamahajan15 · 2022-06-20T19:32:37Z

db_bench results for any regression for async_io = false:

Fill db

./db_bench -db=/tmp/prefix_scan_prefetch_main1 -benchmarks="fillseq" -key_size=32 -value_size=512 -num=50000000 -use_direct_io_for_flush_and_compaction=true -target_file_size_base=16777216

On 7.0.fb (with no async_io support)
- use_direct_reads = false

sudo sh -c "/usr/bin/echo 1 > /proc/sys/vm/drop_caches";
./db_bench -use_existing_db=true -db=/tmp/prefix_scan_prefetch_main1 -benchmarks="seekrandom" -key_size=32 -value_size=512 -num=50000000 -use_direct_reads=false -seek_nexts=327680 -duration=30 -ops_between_duration_checks=1
Initializing RocksDB Options from the specified file
Initializing RocksDB Options from command-line flags
RocksDB: version 7.0
Date: Mon Jun 20 12:09:21 2022
CPU: 32 * Intel Xeon Processor (Skylake)
CPUCache: 16384 KB
Keys: 32 bytes each (+ 0 bytes user-defined timestamp)
Values: 512 bytes each (256 bytes after compression)
Entries: 50000000
Prefix: 0 bytes
Keys per prefix: 0
RawSize: 25939.9 MB (estimated)
FileSize: 13732.9 MB (estimated)
Write rate: 0 bytes/second
Read rate: 0 ops/second
Compression: Snappy
Compression sampling rate: 0
Memtablerep: SkipListFactory
Perf Level: 1

DB path: [/tmp/prefix_scan_prefetch_main1]
seekrandom : 380021.139 micros/op 2 ops/sec; 447.3 MB/s (79 of 79 found)

use_direct_reads = true

./db_bench -use_existing_db=true -db=/tmp/prefix_scan_prefetch_main1 -benchmarks="seekrandom" -key_size=32 -value_size=512 -num=50000000 -use_direct_reads=true -seek_nexts=327680 -duration=30 -ops_between_duration_checks=1
Initializing RocksDB Options from the specified file
Initializing RocksDB Options from command-line flags
RocksDB:    version 7.0
Date:       Mon Jun 20 12:12:37 2022
CPU:        32 * Intel Xeon Processor (Skylake)
CPUCache:   16384 KB
Keys:       32 bytes each (+ 0 bytes user-defined timestamp)
Values:     512 bytes each (256 bytes after compression)
Entries:    50000000
Prefix:    0 bytes
Keys per prefix:    0
RawSize:    25939.9 MB (estimated)
FileSize:   13732.9 MB (estimated)
Write rate: 0 bytes/second
Read rate: 0 ops/second
Compression: Snappy
Compression sampling rate: 0
Memtablerep: SkipListFactory
Perf Level: 1
------------------------------------------------
DB path: [/tmp/prefix_scan_prefetch_main1]
seekrandom   :  490148.839 micros/op 2 ops/sec;  346.8 MB/s (62 of 62 found)

Async support - this PR rebased on latest changes with async_io = false
- use_direct_reads = false

sudo sh -c "/usr/bin/echo 1 > /proc/sys/vm/drop_caches";
./db_bench -use_existing_db=true -db=/tmp/prefix_scan_prefetch_main1 -benchmarks="seekrandom" -key_size=32 -value_size=512 -num=50000000 -use_direct_reads=false -seek_nexts=327680 -duration=30 -ops_between_duration_checks=1
Set seed to 1655753135824170 because --seed was 0
Initializing RocksDB Options from the specified file
Initializing RocksDB Options from command-line flags
RocksDB:    version 7.5
Date:       Mon Jun 20 12:25:35 2022
CPU:        32 * Intel Xeon Processor (Skylake)
CPUCache:   16384 KB
Keys:       32 bytes each (+ 0 bytes user-defined timestamp)
Values:     512 bytes each (256 bytes after compression)
Entries:    50000000
Prefix:    0 bytes
Keys per prefix:    0
RawSize:    25939.9 MB (estimated)
FileSize:   13732.9 MB (estimated)
Write rate: 0 bytes/second
Read rate: 0 ops/second
Compression: Snappy
Compression sampling rate: 0
Memtablerep: SkipListFactory
Perf Level: 1
------------------------------------------------
DB path: [/tmp/prefix_scan_prefetch_main1]
seekrandom   :  375536.575 micros/op 2 ops/sec 30.043 seconds 80 operations;  450.8 MB/s (80 of 80 found)

use_direct_reads = true

sudo sh -c "/usr/bin/echo 1 > /proc/sys/vm/drop_caches";
./db_bench -use_existing_db=true -db=/tmp/prefix_scan_prefetch_main1 -benchmarks="seekrandom" -key_size=32 -value_size=512 -num=50000000 -use_direct_reads=true -seek_nexts=327680 -duration=30 -ops_between_duration_checks=1
Set seed to 1655753262363653 because --seed was 0
Initializing RocksDB Options from the specified file
Initializing RocksDB Options from command-line flags
RocksDB:    version 7.5
Date:       Mon Jun 20 12:27:42 2022
CPU:        32 * Intel Xeon Processor (Skylake)
CPUCache:   16384 KB
Keys:       32 bytes each (+ 0 bytes user-defined timestamp)
Values:     512 bytes each (256 bytes after compression)
Entries:    50000000
Prefix:    0 bytes
Keys per prefix:    0
RawSize:    25939.9 MB (estimated)
FileSize:   13732.9 MB (estimated)
Write rate: 0 bytes/second
Read rate: 0 ops/second
Compression: Snappy
Compression sampling rate: 0
Memtablerep: SkipListFactory
Perf Level: 1
------------------------------------------------
DB path: [/tmp/prefix_scan_prefetch_main1]
seekrandom   :  496248.410 micros/op 2 ops/sec 30.271 seconds 61 operations;  342.6 MB/s (61 of 61 found)

facebook-github-bot · 2022-06-20T22:33:36Z

@akankshamahajan15 has updated the pull request. You must reimport the pull request before landing.

facebook-github-bot · 2022-06-23T19:02:54Z

@akankshamahajan15 has updated the pull request. You must reimport the pull request before landing.

anand1976 · 2022-06-24T19:13:52Z

file/file_prefetch_buffer.cc

I think we need to keep rate_limiter_priority. And, perhaps in another PR, ReadAsync should use the rate_limiter_priority and also pass it in IOOptions to the file system.

I am planning to create another PR supporting rate_limiter. I will pass the argument there.

file/random_access_file_reader.cc

facebook-github-bot · 2022-06-28T18:16:50Z

@akankshamahajan15 has updated the pull request. You must reimport the pull request before landing.

facebook-github-bot · 2022-06-28T18:43:52Z

@akankshamahajan15 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot · 2022-06-28T18:52:09Z

@akankshamahajan15 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot · 2022-06-28T20:32:17Z

@akankshamahajan15 has updated the pull request. You must reimport the pull request before landing.

facebook-github-bot · 2022-06-28T20:32:36Z

@akankshamahajan15 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot · 2022-06-28T20:51:58Z

@akankshamahajan15 has updated the pull request. You must reimport the pull request before landing.

facebook-github-bot · 2022-06-28T20:54:18Z

@akankshamahajan15 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

anand1976

LGTM. Can we add a test (maybe in random_access_file_reader_test.cc) to verify ReadAsync called with an unaligned buffer?

file/random_access_file_reader.cc

Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags:

…supported. Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags:

Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags:

… clear. Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags:

alignment and new scratch and use the scratch passed down by caller Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags:

Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags:

facebook-github-bot · 2022-07-04T19:48:33Z

@akankshamahajan15 has updated the pull request. You must reimport the pull request before landing.

facebook-github-bot · 2022-07-04T19:49:24Z

@akankshamahajan15 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

Summary: Fix bug in O_DIRECT and io_uring when its EOF and bytes_read = 0 because of wrong check, it got added into incomplete list and gets stuck in an infinite loop as it will always return bytes_read = 0. The bug was introduced by PR #10197 and that PR is not released yet in any release branch. Pull Request resolved: #10368 Test Plan: Added new unit test Reviewed By: siying Differential Revision: D37885184 Pulled By: akankshamahajan15 fbshipit-source-id: 35b36a44b696d29b2f6f25301aa1b19547b4e03b

facebook-github-bot added the CLA Signed label Jun 17, 2022

akankshamahajan15 changed the title ~~[WIP] Provide support for direct_io with async_io~~ [WIP] Provide support for direct_reads with async_io Jun 17, 2022

akankshamahajan15 commented Jun 17, 2022

View reviewed changes

file/random_access_file_reader.cc Outdated Show resolved Hide resolved

akankshamahajan15 changed the title ~~[WIP] Provide support for direct_reads with async_io~~ Provide support for direct_reads with async_io Jun 17, 2022

akankshamahajan15 requested a review from anand1976 June 17, 2022 21:47

akankshamahajan15 marked this pull request as ready for review June 17, 2022 21:47

akankshamahajan15 force-pushed the direct_io branch from 04f8d1b to 8f9303c Compare June 18, 2022 03:09

akankshamahajan15 force-pushed the direct_io branch from 8f9303c to dd39f48 Compare June 20, 2022 18:17

akankshamahajan15 force-pushed the direct_io branch from dd39f48 to e5a20f6 Compare June 20, 2022 18:19

akankshamahajan15 commented Jun 20, 2022

View reviewed changes

file/random_access_file_reader.cc Outdated

Copy link

Contributor Author

akankshamahajan15 Jun 20, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

akankshamahajan15 force-pushed the direct_io branch from e5a20f6 to c5cd3e7 Compare June 20, 2022 18:27

akankshamahajan15 force-pushed the direct_io branch from c5cd3e7 to 62df44d Compare June 20, 2022 21:27

akankshamahajan15 force-pushed the direct_io branch from 62df44d to 5dedccf Compare June 23, 2022 19:02

anand1976 reviewed Jun 24, 2022

View reviewed changes

akankshamahajan15 force-pushed the direct_io branch from 5dedccf to c9833c4 Compare June 28, 2022 18:16

akankshamahajan15 requested a review from anand1976 June 28, 2022 18:33

akankshamahajan15 force-pushed the direct_io branch from 530afac to 9b570a6 Compare June 28, 2022 20:51

anand1976 approved these changes Jun 30, 2022

View reviewed changes

file/random_access_file_reader.cc Outdated Show resolved Hide resolved

akankshamahajan added 10 commits July 4, 2022 12:41

Provide support for direct_io with async_io

d2cb22e

Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags:

Clear some classes and reuse existing one.

7ea3032

Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags:

Fix build failure and move input/outout parameters

da769f0

Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags:

Remove rate_limiter argument in ReadAsync and add it later when it's …

ee8f439

…supported. Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags:

Update History.md

866e94f

Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags:

Fix build failures

ccbeed8

Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags:

Add comments and change variable name to make BlockPrefetch code more…

f597168

… clear. Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags:

If request is already aligned, then RandomAccessFileReader will skip the

54975c8

alignment and new scratch and use the scratch passed down by caller Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags:

Fix linter errors

6153319

Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags:

Addressed comments

0133b34

Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags:

akankshamahajan15 force-pushed the direct_io branch from 9b570a6 to 0133b34 Compare July 4, 2022 19:48

facebook-github-bot closed this in 2acbf38 Jul 6, 2022

akankshamahajan15 mentioned this pull request Jul 15, 2022

Fix hang in MultiRead with O_DIRECT and io_uring #10368

Closed

Provide support for direct_reads with async_io #10197

Provide support for direct_reads with async_io #10197

Uh oh!

Conversation

akankshamahajan15 commented Jun 17, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

akankshamahajan15 commented Jun 17, 2022

Uh oh!

Uh oh!

facebook-github-bot commented Jun 17, 2022

Uh oh!

facebook-github-bot commented Jun 18, 2022

Uh oh!

facebook-github-bot commented Jun 18, 2022

Uh oh!

facebook-github-bot commented Jun 20, 2022

Uh oh!

facebook-github-bot commented Jun 20, 2022

Uh oh!

akankshamahajan15 Jun 20, 2022

Choose a reason for hiding this comment

Uh oh!

facebook-github-bot commented Jun 20, 2022

Uh oh!

facebook-github-bot commented Jun 20, 2022

Uh oh!

akankshamahajan15 commented Jun 20, 2022

Uh oh!

facebook-github-bot commented Jun 20, 2022

Uh oh!

facebook-github-bot commented Jun 23, 2022

Uh oh!

anand1976 Jun 24, 2022

Choose a reason for hiding this comment

Uh oh!

akankshamahajan15 Jun 24, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

facebook-github-bot commented Jun 28, 2022

Uh oh!

facebook-github-bot commented Jun 28, 2022

Uh oh!

facebook-github-bot commented Jun 28, 2022

Uh oh!

facebook-github-bot commented Jun 28, 2022

Uh oh!

facebook-github-bot commented Jun 28, 2022

Uh oh!

facebook-github-bot commented Jun 28, 2022

Uh oh!

facebook-github-bot commented Jun 28, 2022

Uh oh!

anand1976 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

facebook-github-bot commented Jul 4, 2022

Uh oh!

facebook-github-bot commented Jul 4, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

akankshamahajan15 commented Jun 17, 2022 •

edited

Loading