feat: encode bytearray and memoryview buffers - #14
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intent
Release msgspec-toon 0.3.0b5 to fix issue 13. Encode exact bytearray and C-contiguous memoryview values directly as the same standard padded-base64 TOON scalar as exact bytes, including at root, inside Structs, and inside compact arrays, before enc_hook. Match msgspec 0.21.1 by preserving BufferError for non-contiguous memoryviews and refusing bytes and bytearray subclasses unless an enc_hook explicitly converts them. Do not silently stringify non-string mapping keys: preserve the existing policy and make every root or nested refusal name msgspec.to_builtins(..., str_keys=True) as the supported conversion route. Record bytearray, memoryview, and bytes subclasses separately in the executable support matrix; update documentation, changelog, locks, version 0.3.0b5, and advance the release performance guard to v0.3.0b4. Preserve canonical bytes, token locks, full TOON conformance, G2, G3, G5, and payload-safe errors. Validate, push, open the PR, obtain green CI, merge, and release through the repository workflow without adding an ad hoc local performance run.
What Changed
bytearrayand C-contiguousmemoryviewvalues as padded-base64 TOON scalars beforeenc_hook, while preservingBufferErrorfor non-contiguous views and hook/refusal behavior for binary subclasses.msgspec.to_builtins(..., str_keys=True); document and record the expanded behavior in the executable support matrix.0.3.0b5, update release documentation, and advance the performance guard baseline tov0.3.0b4.Risk Assessment
🚨 High: The fix still permits the original oversized pre-validation allocation through a concrete race on the repository's supported free-threaded Python target.
Testing
The full 538-fixture TOON corpus, focused buffer regressions and support matrix, canonical byte/token locks, generated changelog contract, and isolated G2 proof all passed; API evidence demonstrates the intended base64, hook, refusal, and payload-safe error behavior, while no screenshot was applicable to this non-UI codec and no ad hoc G3/G5 benchmark was run, as required.
Evidence: End-to-end buffer encode API transcript
Evidence: TOON 4.1.1 conformance results
Evidence: G2 allocation proof
Evidence: Version, lock, guard, and changelog contract transcript
Pipeline
Updates from git push no-mistakes
✅ **intent** - passed
✅ No issues found.
✅ **Rebase** - passed
✅ No issues found.
src/encode.rs:1278-exact_buffer_bytescopies buffer contents beforechecked_base64enforces the existing 2**32−1 limit. A larger bytearray is cloned here, while a memoryview is copied bytobytesand then cloned again, potentially allocating 4–8+ GiB or failing withMemoryError/process termination instead of the exact-bytesEncodeError. Check bytearray length or memoryviewnbytesthrough a shared size guard before copying.🔧 Fix: Guard buffer sizes before allocating copies
1 error still open:
src/encode.rs:1283- On the supported free-threaded Python build,len()andto_vec()are separate bytearray operations. Another thread can resize the exact bytearray beyondu32::MAXafter this check but before the snapshot, recreating the original multi-gigabyte copy-before-error path. Observe the length and copy the bytes within one object critical section, then reject based on that same snapshot length.✅ **Test** - passed
✅ No issues found.
uv run --no-sync python conformance/run.py --help(the script has no help mode, so this executed the complete pinned TOON 4.1.1 corpus)uv run --no-sync pytest -q tests/test_native_encode_types.pyuv run --no-sync pytest -qwith the four buffer-relatedtests/test_support_matrix.py::test_declared_behavior_is_actual_behavior[...]node IDsuv run --no-sync pytest -q tests/test_release_report.py::test_current_compatibility_delta_records_support_changes(initial missing-benchmark-group setup error, then passed afteruv sync --group bench --locked)uv run --no-sync pytest -q tests/test_efficiency_lock.pyuv run --no-sync python scripts/release-report.py --check-changelogmake g2Inline publicmsgspec_toon.encodeprobe covering root, Struct, compact arrays, hook precedence, non-contiguous memoryviews, binary subclasses, and root/nested mapping-key failuresParsed release-contract probe for installed/Python/Rust/lock versions,v0.3.0b4guard semantics, and generated changelog compatibilityValidated evidence JSON/transcript contracts and SHA-256 digests; confirmed a clean worktree after transient cleanup✅ **Document** - passed
✅ No issues found.
✅ **Lint** - passed
✅ No issues found.
✅ **Push** - passed
✅ No issues found.