Skip to content

Commit 9fe86b4

Browse files
authored
feat: update quickstart to iroh-blobs 0.90 (#350)
* feat: update quickstart to iroh-blobs 0.90 * Update github stars * Prefer reusing the downloader * Fix typo
1 parent 195d9eb commit 9fe86b4

File tree

2 files changed

+87
-97
lines changed

2 files changed

+87
-97
lines changed

src/app/docs/quickstart/page.mdx

Lines changed: 86 additions & 96 deletions
Original file line numberDiff line numberDiff line change
@@ -39,7 +39,7 @@ If you're ever confused about what to import, take a look at the imports in the
3939

4040
We'll assume you've set up [rust](https://www.rust-lang.org/) and [cargo](https://doc.rust-lang.org/cargo/) on your machine.
4141

42-
Initialize a new project by running `cargo init file-transfer`, then `cd file-transfer` and install all the packages we're going to use: `cargo add iroh iroh-base tokio anyhow && cargo add iroh-blobs --features rpc`.
42+
Initialize a new project by running `cargo init file-transfer`, then `cd file-transfer` and install all the packages we're going to use: `cargo add iroh iroh-blobs tokio anyhow`.
4343

4444
From here on we'll be working inside the `src/main.rs` file.
4545

@@ -87,8 +87,10 @@ async fn main() -> anyhow::Result<()> {
8787
// connections in the iroh p2p world
8888
let endpoint = Endpoint::builder().discovery_n0().bind().await?;
8989

90-
// We initialize the Blobs protocol in-memory
91-
let blobs = Blobs::memory().build(&endpoint);
90+
// We initialize an in-memory backing store for iroh-blobs
91+
let store = MemStore::new();
92+
// Then we initialize a struct that can accept blobs requests over iroh connections
93+
let blobs = Blobs::new(&store, endpoint.clone(), None);
9294

9395
// ...
9496

@@ -102,65 +104,10 @@ Learn more about what we mean by "protocol" on the [protocol documentation page]
102104

103105
With these two lines, we've initialized iroh-blobs and gave it access to our `Endpoint`.
104106

105-
This is not quite enough to make it answer requests from the network, for that we need to configure a so-called `Router` for protocols.
106-
Similar to routers in webserver libraries, it runs a loop accepting incoming connections and routes them to the specific handler.
107-
However, instead of handlers being organized by HTTP paths, it routes based on "ALPNs".
108-
Read more about ALPNs and the router on the [protocol](/docs/concepts/protocol#alpns) and [router](/docs/concepts/router) documentation pages.
109-
110-
Now, using the `Router` we can finish the skeleton of our application integrating iroh and iroh-blobs:
107+
At this point what we want to do depends on whether we want to accept incoming iroh connections from the network or create outbound iroh connections to other nodes.
108+
Which one we want to do depends on if the executable was called with `send` as an argument or `receive`, so let's parse these two options out from the CLI arguments and match on them:
111109

112110
```rust
113-
#[tokio::main]
114-
async fn main() -> anyhow::Result<()> {
115-
// Create an endpoint, it allows creating and accepting
116-
// connections in the iroh p2p world
117-
let endpoint = Endpoint::builder().discovery_n0().bind().await?;
118-
119-
// We initialize the Blobs protocol in-memory
120-
let blobs = Blobs::memory().build(&endpoint);
121-
122-
// Now we build a router that accepts blobs connections & routes them
123-
// to the blobs protocol.
124-
let router = Router::builder(endpoint)
125-
.accept(iroh_blobs::ALPN, blobs.clone())
126-
.spawn();
127-
128-
// do *something*
129-
130-
// Gracefully shut down the router
131-
println!("Shutting down.");
132-
router.shutdown().await?;
133-
134-
Ok(())
135-
}
136-
```
137-
138-
I've also taken the liberty to make sure that we're gracefully shutting down the `Router` and all its protocols with it, in this case that's only iroh-blobs.
139-
140-
141-
## Doing something
142-
143-
So far, this code works, but doesn't actually do anything besides spinning up a node and immediately shutting down.
144-
If we put in a `tokio::time::timeout` or `tokio::signal::ctrl_c().await` in there, although it *would* actually respond to network requests for the blobs protocol, these responses are practically useless as we've stored no blobs to respond with.
145-
146-
Here's our plan for turning this into a CLI that actually does what we set out to build:
147-
1. We'll grab a [`Blobs::client`](https://docs.rs/iroh-blobs/latest/iroh_blobs/net_protocol/struct.Blobs.html#method.client) to interact with the iroh-blobs node we're running locally.
148-
2. We check the CLI arguments to find out whether you ran `cargo run -- send [PATH]` or `cargo run -- receive [TICKET] [PATH]`.
149-
3. If we're supposed to send data:
150-
- we'll use [`add_from_path`](https://docs.rs/iroh-blobs/latest/iroh_blobs/rpc/client/blobs/struct.Client.html#method.add_from_path) to index local data and make it available,
151-
- print instructions for fetching said file,
152-
- and then wait for Ctrl+C.
153-
4. If we're supposed to receive data:
154-
- we'll parse the ticket out of the CLI arguments,
155-
- download the file using [`download`](https://docs.rs/iroh-blobs/latest/iroh_blobs/rpc/client/blobs/struct.Client.html#method.download),
156-
- and copy the result to the local file system.
157-
158-
Phew okay! Here's how we'll grab an iroh-blobs client and look at the CLI arguments:
159-
160-
```rust
161-
// We use a blobs client to interact with the blobs protocol we're running locally:
162-
let blobs_client = blobs.client();
163-
164111
// Grab all passed in arguments, the first one is the binary itself, so we skip it.
165112
let args: Vec<String> = std::env::args().skip(1).collect();
166113
// Convert to &str, so we can pattern-match easily:
@@ -186,7 +133,10 @@ match arg_refs.as_slice() {
186133
}
187134
```
188135

189-
Now all we need to do is fill in the `todo!()`s one-by-one:
136+
We're also going to print some simple help text when there's no arguments or we can't parse them.
137+
138+
What's left to do now is fill in the two `todo!()`s!
139+
190140

191141
### Getting ready to send
192142

@@ -195,9 +145,9 @@ If we want to make a file available over the network with iroh-blobs, we first n
195145
<Note>
196146
What does this step do?
197147

198-
It hashes the file using [BLAKE3](https://en.wikipedia.org/wiki/BLAKE_(hash_function)) and stores a so-called ["outboard"](https://github.com/oconnor663/bao?tab=readme-ov-file#outboard-mode) for that file.
199-
This outboard file contains information about hashes of parts of this file.
200-
All of this enables some extra features with iroh-blobs like automatically verifying the integrity of the file *during* streaming, verified range downloads and download resumption.
148+
It hashes the file using [BLAKE3](https://en.wikipedia.org/wiki/BLAKE_(hash_function)) and remembers a so-called ["outboard"](https://github.com/oconnor663/bao?tab=readme-ov-file#outboard-mode) for that file.
149+
This outboard contains information about hashes of parts of this file.
150+
All of this enables some extra features with iroh-blobs like automatically verifying the integrity of the file *while it's streaming*, verified range downloads and download resumption.
201151
</Note>
202152

203153
```rust
@@ -206,81 +156,121 @@ let abs_path = std::path::absolute(&filename)?;
206156

207157
println!("Hashing file.");
208158

209-
// keep the file in place and link it, instead of copying it into the in-memory blobs database
210-
let in_place = true;
211-
let blob = blobs_client
212-
.add_from_path(abs_path, in_place, SetTagOption::Auto, WrapOption::NoWrap)
213-
.await?
214-
.finish()
215-
.await?;
159+
// When we import a blob, we get back a "tag" that refers to said blob in the store
160+
// and allows us to control when/if it gets garbage-collected
161+
let tag = store.blobs().add_path(abs_path).await?;
216162
```
217163

218-
The `WrapOption::NoWrap` is just an indicator that we don't want to wrap the file with some metadata information about its file name.
219-
We keep it simple here for now!
164+
<Note>
165+
For other use cases, there are other ways of importing blobs into iroh-blobs, you're not restricted to pulling them from the file system!
166+
You can see other options available, such as [`add_slice`](https://docs.rs/iroh-blobs/latest/iroh_blobs/api/blobs/struct.Blobs.html#method.add_slice).
167+
Make sure to also check out the options you can pass and their documentation for some interesting tidbits on performance.
168+
</Note>
220169

221-
Now, we'll print a `BlobTicket`.
222-
This ticket contains the `NodeId` of our `Endpoint` as well as the file's BLAKE3 hash.
170+
The return value `tag` contains the final piece of information such that another node can fetch a blob from us.
171+
172+
We'll use a `BlobTicket` to put the file's BLAKE3 hash and our endpoint's `NodeId` into a single copy-able string:
223173

224174
```rust
225-
let node_id = router.endpoint().node_id();
226-
let ticket = BlobTicket::new(node_id.into(), blob.hash, blob.format)?;
175+
let node_id = endpoint.node_id();
176+
let ticket = BlobTicket::new(node_id.into(), tag.hash, tag.format);
227177

228178
println!("File hashed. Fetch this file by running:");
229-
println!("cargo run --example transfer -- receive {ticket} {path}");
179+
println!(
180+
"cargo run --example transfer -- receive {ticket} {}",
181+
filename.display()
182+
);
183+
```
184+
185+
Now we've imported the file and produced instructions for how to fetch it, but we're actually not yet actively listening for incoming connections yet! (iroh-blobs won't do so unless you specifically tell it to do that.)
186+
187+
For that we'll use iroh's `Router`.
188+
Similar to routers in webserver libraries, it runs a loop accepting incoming connections and routes them to the specific handler.
189+
However, instead of handlers being organized by HTTP paths, it routes based on "ALPNs".
190+
Read more about ALPNs and the router on the [protocol](/docs/concepts/protocol#alpns) and [router](/docs/concepts/router) documentation pages.
191+
192+
In our case, we only need a single protocol, but constructing a router also takes care of running the accept loop, so that makes our life easier:
193+
194+
```rs
195+
// For sending files we build a router that accepts blobs connections & routes them
196+
// to the blobs protocol.
197+
let router = Router::builder(endpoint)
198+
.accept(iroh_blobs::ALPN, blobs)
199+
.spawn();
230200

231201
tokio::signal::ctrl_c().await?;
202+
203+
// Gracefully shut down the node
204+
println!("Shutting down.");
205+
router.shutdown().await?;
232206
```
233207

234-
And as you can see, as a final step we wait for the user to stop the file providing side by hitting `Ctrl+C` in the console.
208+
And as you can see, as a final step we wait for the user to stop the file providing side by hitting `Ctrl+C` in the console and once they do so, we shut down the router gracefully.
209+
235210

236211
### Connecting to the other side to receive
237212

238213
On the connection side, we got the `ticket` and the `path` from the CLI arguments and we can parse them into their `struct` versions.
239214

240-
With them parsed, we can call `blobs.download` with the information contained in the ticket and wait for the download to finish:
215+
With them parsed
216+
- we first construct a `Downloader` (that can help us coordinate multiple downloads from multiple peers if we'd want to)
217+
- and then call `.download` with the information contained in the ticket and wait for the download to finish:
218+
219+
<Note>
220+
Reusing the same downloader across multiple downloads can be more efficient, e.g. by reusing existing connections.
221+
In this example we don't see this, but it might come in handy for your use case.
222+
</Note>
241223

242224
```rust
243225
let filename: PathBuf = filename.parse()?;
244226
let abs_path = std::path::absolute(filename)?;
245227
let ticket: BlobTicket = ticket.parse()?;
246228

229+
// For receiving files, we create a "downloader" that allows us to fetch files
230+
// from other nodes via iroh connections
231+
let downloader = store.downloader(&endpoint);
232+
247233
println!("Starting download.");
248234

249-
blobs_client
250-
.download(ticket.hash(), ticket.node_addr().clone())
251-
.await?
252-
.finish()
235+
downloader
236+
.download(ticket.hash(), Some(ticket.node_addr().node_id))
253237
.await?;
254238

255239
println!("Finished download.");
256240
```
257241

258-
As a final step, we'll export the file we just downloaded into our blobs database to the desired file path:
242+
<Note>
243+
The return value of `.download()` is [`DownloadProgress`](https://docs.rs/iroh-blobs/latest/iroh_blobs/api/downloader/struct.DownloadProgress.html).
244+
You can either `.await` it to wait for the download to finish, or you can stream out progress events instead, e.g. if you wanted to use this for showing a nice progress bar!
245+
</Note>
246+
247+
As a final step, we'll export the file we just downloaded into our in-memory blobs database to the desired file path:
259248

260249
```rust
261250
println!("Copying to destination.");
262251

263-
blobs_client
264-
.export(
265-
ticket.hash(),
266-
abs_path,
267-
ExportFormat::Blob,
268-
ExportMode::Copy,
269-
)
270-
.await?
271-
.finish()
272-
.await?;
252+
store.blobs().export(ticket.hash(), abs_path).await?;
273253

274254
println!("Finished copying.");
275255
```
276256

277257
<Note>
278258
This first downloads the file completely into memory, then copies it from memory to file in a second step.
279259

280-
There's ways to make this work without having to store the whole file in memory, but those involve setting up `Blobs::persistent` instead of `Blobs::memory` and using `blobs.export` with `EntryMode::TryReference`.
260+
There's ways to make this work without having to store the whole file in memory!
261+
This would involve setting up an `FsStore` instead of a `MemStore` and using `.export_with_opts` with `ExportMode::TryReference`.
262+
Something similar can be done on the sending side!
281263
We'll leave these changes as an exercise to the reader 😉
282264
</Note>
283265

266+
Before we leave, we'll gracefully shut down our endpoint in the receive branch, too:
267+
268+
```rs
269+
// Gracefully shut down the node
270+
println!("Shutting down.");
271+
endpoint.close().await;
272+
```
273+
284274

285275
## That's it!
286276

src/components/GithubStars.jsx

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ export default function GithubStars(props) {
66
return (
77
<Link href="https://github.com/n0-computer/iroh" className='p-2 -mt-2 flex text-sm leading-5 fill-zinc-400 text-zinc-600 transition hover:text-zinc-900 dark:text-zinc-400 dark:hover:text-zinc-600 dark:hover:fill-zinc-600 hover:bg-black/10 rounded'>
88
<GithubIcon className="h-5 w-5" />
9-
<span className='ml-2 mt-0'>5.4k</span>
9+
<span className='ml-2 mt-0'>5.5k</span>
1010
</Link>
1111
)
1212
}

0 commit comments

Comments
 (0)