Add an experimental mixed io mode. #61

rklaehn · 2025-03-07T07:54:21Z

The mixed IO mode is using a tokio mpsc queue to communicate with an external source/sink of data, but blocking IO for the local file io.

It is used by the big blob store refactor, where I create a dedicated tokio runtime for IO and do blocking IO on that, similar to the solution that @Frando and me have come up with for database interactions.

The feature is called experimental-mixed so that people don't expect stability. It will get a better and less scary name once it is stable.

rklaehn · 2025-03-07T07:55:31Z

cli/main.rs

@@ -1,7 +1,8 @@
+use std::{io::Write, path::PathBuf};


rklaehn · 2025-03-07T07:55:39Z

examples/encode_decode_async.rs

@@ -1,3 +1,5 @@
+use std::io;


rklaehn · 2025-03-07T07:55:59Z

examples/encode_decode_sync.rs

@@ -1,3 +1,5 @@
+use std::io;


rklaehn · 2025-03-07T07:56:07Z

src/io/error.rs

 use std::{fmt, io};

+use crate::{ChunkNum, TreeNode};


rklaehn · 2025-03-07T07:56:38Z

src/io/fsm.rs

 use iroh_io::{AsyncStreamReader, AsyncStreamWriter};
 use smallvec::SmallVec;

 pub use super::BaoContentItem;
+use super::{combine_hash_pair, DecodeError};


rklaehn · 2025-03-07T07:57:21Z

src/io/mod.rs

 pub struct Leaf {
    /// The byte offset of the leaf in the file.
    pub offset: u64,
    /// The data of the leaf.
    pub data: Bytes,
 }

+impl std::fmt::Debug for Leaf {


nicer debug instance for leaf, unrelated change

rklaehn · 2025-03-07T07:57:55Z

src/io/outboard.rs

 use std::io;

+use crate::{blake3, BaoTree, BlockSize, TreeNode};


rklaehn · 2025-03-07T07:58:20Z

src/io/sync.rs

    data: D,
    outboard: O,
    ranges: &ChunkRangesRef,
    encoded: W,
 ) -> result::Result<(), EncodeError> {
+    if ranges.is_empty() {


properly handle the case where ranges are empty

rklaehn · 2025-03-07T07:58:31Z

src/io/sync.rs

@@ -658,13 +661,12 @@ mod validate {
    use genawaiter::sync::{Co, Gen};
    use positioned_io::ReadAt;

+    use super::Outboard;


rklaehn · 2025-03-07T07:59:12Z

src/lib.rs

@@ -74,6 +74,8 @@
 //! ## Simple end to end example
 //!
 //! ```no_run
+//! use std::io;


rklaehn · 2025-03-07T07:59:35Z

src/rec.rs

@@ -439,13 +436,15 @@ mod test_support {
        (res, hash)
    }

+    /// Check that l and r of a 2-tuple are equal


fmt and doc strings

rklaehn · 2025-03-07T07:59:48Z

src/tests.rs

@@ -8,6 +8,16 @@ use bytes::Bytes;
 use proptest::prelude::*;
 use range_collections::RangeSet2;

+use super::{


rklaehn · 2025-03-07T08:00:22Z

src/tree.rs

@@ -80,6 +80,19 @@ impl BlockSize {
 }

 impl ChunkNum {
+    /// Start (inclusive) of the chunk group that this chunk is in
+    pub const fn chunk_group_start(start: ChunkNum, block_size: BlockSize) -> ChunkNum {


new fns to round up a set of ranges to chunk groups, needed for the new store

rklaehn · 2025-03-07T08:00:38Z

src/tests2.rs

@@ -6,31 +6,34 @@
 //! There is a proptest called <testname>_proptest that calls the test multiple times with random data.
 //! There is a test called <testname>_cases that calls the test with a few hardcoded values, either
 //! handcrafted or from a previous failure of a proptest.
+use std::ops::Range;


rklaehn added 5 commits February 26, 2025 10:16

add mixed mode export

0e8f0af

fmt

2a09302

Add test for new chunk fns

d157eb4

Update mixed to not copy bytes

87d384f

Make tokio dep optional

86e1d54