Support BytesSources other than the one for reducer args
#3294
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Each reducer gets its arguments through an `ArgSource`, a Unix-file-like abstraction for streams of bytes. Prior to this commit, we had an ABI designed as if it could support other args sources, but it actually hardcoded the ID of the reducer args source, and errored elsewhere. This commit extends the `BytesSource` infrastructure to support other bytes sources. This will be useful for exposing JWT payloads and HTTP responses. No other `BytesSource` uses are actually included in this commit, only the infrastructure. This commit also defines a new host call, `bytes_source_remaining_length`. This is intended to allow callers to pre-allocate a buffer correctly sized to read the entire `BytesSource` all at once. The new host function is added to a new ABI minor version, 10.1, so that old SpacetimeDB hosts can detect and reject too-new compiled modules. I have added uses of this new function to `__call_reducer__` in both Rust and C#, even though it's not strictly necessary, and I haven't removed the loop which repeatedly calls `bytes_source_read` and grows the buffer.
jsdt
reviewed
Sep 25, 2025
As @jsdt points out, `bytes::Bytes` is already a view over an underlying buffer, so keeping an additional cursor as a slice into it is silly.
rekhoff
reviewed
Sep 25, 2025
Co-authored-by: rekhoff <[email protected]> Signed-off-by: Phoebe Goldman <[email protected]>
jsdt
approved these changes
Sep 26, 2025
Centril
reviewed
Sep 26, 2025
Co-authored-by: Mazdak Farrokhzad <[email protected]> Signed-off-by: Phoebe Goldman <[email protected]>
|
Force merging for reasons discussed internally. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of Changes
Each reducer gets its arguments through an
ArgSource, a Unix-file-like abstraction for streams of bytes. Prior to this commit, we had an ABI designed as if it could support other args sources, but it actually hardcoded the ID of the reducer args source, and errored elsewhere.This commit extends the
BytesSourceinfrastructure to support other bytes sources. This will be useful for exposing JWT payloads and HTTP responses. No otherBytesSourceuses are actually included in this commit, only the infrastructure.This commit also defines a new host call,
bytes_source_remaining_length. This is intended to allow callers to pre-allocate a buffer correctly sized to read the entireBytesSourceall at once. The new host function is added to a new ABI minor version, 10.1, so that old SpacetimeDB hosts can detect and reject too-new compiled modules. I have added uses of this new function to__call_reducer__in both Rust and C#, even though it's not strictly necessary,and I haven't removed the loop which repeatedly calls
bytes_source_readand grows the buffer.API and ABI breaking changes
Adds a new ABI minor version,
spacetime_10.1. This means that old SpacetimeDB hosts will reject new compiled modules.Expected complexity level and risk
2-ish? WASM ABI code is always fiddly, but this is a pretty simple case.
Testing