BM-615 - Add caching for assessor and set builder images #393

willemolding · 2025-03-12T04:10:30Z

Closes #388

Uses reqwest middleware to add a HTTP-aware cache. See https://docs.rs/http-cache-reqwest/latest/http_cache_reqwest/index.html. This is used only for caching requests for assessor and set-builder images at this time.

Adds caching to requests to retrieve assessor and set builder images
- Can trivially also add it to user images but this may need more discussion
Add new command line flag --cache-images to broker
Adds new command line arg --cache-dir to broker (default /tmp/broker_cache/)

linear · 2025-03-12T04:10:33Z

BM-615 Cache guest binaries downloaded from IPFS

crates/broker/src/lib.rs

willpote

Any reason we wouldn't default this to true? Otherwise it still feels like a footgun that provers will eventually run into.

github-actions · 2025-03-17T20:50:14Z

🚀 Documentation Preview

Deployment URL: https://boundless-documentation-aqg49wqr4-risczero.vercel.app

Updated at: 2025-03-20 23:43:40 UTC

willemolding · 2025-03-18T21:56:50Z

Any reason we wouldn't default this to true? Otherwise it still feels like a footgun that provers will eventually run into.

Ok yeah we can default to true. In this case maybe a --nocache flag is a nicer way to do it

) Closes #390 - Adds new public method to `boundless_market` - `with_stake_balance_alert` which takes a warning and error threshold to alert on - Adds new config fields to broker to set these on the market in `order_monitor` - Adds to existing test to check that market is correctly alerting when stake balance goes below thresholds

The error would panic and crash the instance, which was indefinitely irrecoverable. This rejects that invalid case, such that the node will continue. The part I am unsure about is the sanity check within the executor for keccak requests. This changes to error log and skip those requests, and didn't seem like a clean mechanism to fail to proof as a whole, but perhaps this should still go through to fail with the prove agent so that it errors and then perhaps the tasks get cancelled through taskdb? Haven't looked that in depth at the implications yet.

We now favor "Explorer" over "Indexer" in all comms

<img width="782" alt="image" src="https://github.com/user-attachments/assets/1fde4ea9-80f2-48e7-ba45-5563176e811c" /> --------- Co-authored-by: Sasha <[email protected]>

Automatic testing of Bento when the bento/ subdirectory is modified. --------- Co-authored-by: Austin Abell <[email protected]> Co-authored-by: Victor Graf <[email protected]>

Partially handles BM-616 Co-authored-by: Richard Howard <[email protected]>

Closes BM-619

Partially handles BM-616

We've seen transient errors when calling the Bonsai API. This adds retries to all API calls (previously we only retried calls to check job status). This also adds retries to the entire proving workflow, to better capture the case where are able to kick off a proving job and it terminates, but it terminates with a retryable internal server error. This should help with: #410 --------- Co-authored-by: Willem Olding <[email protected]> Co-authored-by: Victor Snyder-Graf <[email protected]> Co-authored-by: Rami Lukata <[email protected]> Co-authored-by: Angelo Capossele <[email protected]>

CI wasn't running any rust test on CI due to a wrong `ignored` keyword

Closes #388 Uses reqwest middleware to add a HTTP-aware cache. See https://docs.rs/http-cache-reqwest/latest/http_cache_reqwest/index.html. This is used only for caching requests for assessor and set-builder images at this time. - Adds caching to requests to retrieve assessor and set builder images - - Can trivially also add it to user images but this may need more discussion - Add new command line flag `--cache-images` to broker - Adds new command line arg `--cache-dir` to broker (default `/tmp/broker_cache/`) --------- Co-authored-by: Austin Abell <[email protected]> Co-authored-by: Cohan <[email protected]> Co-authored-by: Sasha <[email protected]> Co-authored-by: pote.eth <[email protected]> Co-authored-by: Parker Thompson <[email protected]> Co-authored-by: Victor Graf <[email protected]> Co-authored-by: Rami Lukata <[email protected]> Co-authored-by: Richard Howard <[email protected]> Co-authored-by: Angelo Capossele <[email protected]>

willemolding requested review from capossele, Wollac, zeroecco, austinabell and a team as code owners March 12, 2025 04:10

willemolding force-pushed the willem/bm-615-image-cache branch 2 times, most recently from e018f20 to 9dc3c71 Compare March 12, 2025 04:15

willemolding requested a review from nategraf March 12, 2025 04:16

willemolding force-pushed the willem/bm-615-image-cache branch from 9dc3c71 to 0b5df2b Compare March 12, 2025 04:25

austinabell reviewed Mar 12, 2025

View reviewed changes

crates/broker/src/lib.rs Outdated Show resolved Hide resolved

austinabell approved these changes Mar 13, 2025

View reviewed changes

willemolding requested a review from rlukata as a code owner March 17, 2025 20:33

willpote reviewed Mar 17, 2025

View reviewed changes

willpote approved these changes Mar 19, 2025

View reviewed changes

willemolding and others added 13 commits March 21, 2025 12:26

adds caching middleware capabilities to storage::UriHandler

c290a44

add top level config field for cache

0a7db98

remove cache from config

11bdfa0

add cache configurations to broker command line args

093bb37

bump reqwest patch version to fix fetch_mode_no_cors issue

c9aa23c

BM-625: docs: rename indexer with explorer (#395)

5e53b68

We now favor "Explorer" over "Indexer" in all comms

BM-491: feat: redirect if user has seen homepage already (#281)

99c26de

<img width="782" alt="image" src="https://github.com/user-attachments/assets/1fde4ea9-80f2-48e7-ba45-5563176e811c" /> --------- Co-authored-by: Sasha <[email protected]>

BM-628: Update CODEOWNERS (#398)

3bffef1

BM-531: Bento CI (#309)

a468450

Automatic testing of Bento when the bento/ subdirectory is modified. --------- Co-authored-by: Austin Abell <[email protected]> Co-authored-by: Victor Graf <[email protected]>

BM-616: TaskDB Optimizations (#400)

8bb7ccb

Partially handles BM-616 Co-authored-by: Richard Howard <[email protected]>

BM-634: fix bento CI for new broker env template (#405)

e8d5418

rlukata and others added 6 commits March 21, 2025 12:27

BM-619: Switch to Gnark (#399)

6123a5d

Closes BM-619

BM-632: docs: remake homepage (#403)

887b517

switch to cachine by default

05c7e39

BM-633: Add Keccak to Planner (#404)

a04cf0c

Partially handles BM-616

BM-665: Fix running tests on CI (#423)

d735ee1

CI wasn't running any rust test on CI due to a wrong `ignored` keyword

willemolding force-pushed the willem/bm-615-image-cache branch from eec0428 to d735ee1 Compare March 20, 2025 23:39

rebase with sigs

dbcc284

willemolding enabled auto-merge (squash) March 20, 2025 23:40

willemolding merged commit 6e8ba39 into main Mar 20, 2025
18 checks passed

willemolding deleted the willem/bm-615-image-cache branch March 20, 2025 23:51

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BM-615 - Add caching for assessor and set builder images #393

BM-615 - Add caching for assessor and set builder images #393

willemolding commented Mar 12, 2025

linear bot commented Mar 12, 2025

willpote left a comment

github-actions bot commented Mar 17, 2025 •

edited

Loading

willemolding commented Mar 18, 2025

BM-615 - Add caching for assessor and set builder images #393

BM-615 - Add caching for assessor and set builder images #393

Conversation

willemolding commented Mar 12, 2025

linear bot commented Mar 12, 2025

willpote left a comment

Choose a reason for hiding this comment

github-actions bot commented Mar 17, 2025 • edited Loading

willemolding commented Mar 18, 2025

github-actions bot commented Mar 17, 2025 •

edited

Loading