Improve scene serialization efficiency - #759
Merged
Merged
Conversation
StateSerializer.serialize() now deduplicates byte-identical binary buffers (remapping placeholder indices onto shared buffers), streams parts through a multithreaded zstd compressor instead of concatenating a full uncompressed copy first, and unregisters before encoding so concurrently queued messages can't corrupt the output. A recording that re-adds a 4MB mesh over 20 frames shrinks from 194MB to 9.7MB and serializes ~10x faster; recordings without duplicate buffers are byte-identical to before. Client: recording/embed decode logic moves from FilePlayback.tsx into a UI-free PlaybackDecode.ts; embedded base64 decodes via native Uint8Array.fromBase64 when available; downloads stream into a preallocated buffer; the compressed payload is no longer copied before decompression. Tests: Python round-trip suite for the .viser format with an independent decoder (dedup invariants, dtype collisions, empty/non-contiguous arrays, concurrency, determinism, as_html payload), and a vitest suite decoding a real Python-generated fixture end-to-end.
The starred list display evaluated flush() before consuming the compress() generator on 3.8 (BUILD_LIST_UNPACK evaluates operands before concatenating), raising 'cannot call compress() after compressor finished'.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
StateSerializer.serialize()now deduplicates byte-identical binary buffers, shrinking recordings that re-send unchanged arrays (a 20-frame recording re-adding a 4MB mesh drops from 194MB to 9.7MB) while leaving duplicate-free scenes byte-identical.FilePlayback.tsxinto a UI-freePlaybackDecode.ts, with embedded base64 decoded via nativeUint8Array.fromBase64when available and one less full copy of both downloaded and compressed payloads..viserformat round-trip with an independent Python decoder (dedup invariants, dtype collisions, empty/non-contiguous arrays, concurrency, determinism) plus a vitest suite decoding a real Python-generated fixture end-to-end.