Repository navigation
table: use the table for messages with map fields - #477
Merged
Merged
Conversation
|
All contributors have signed the CLA ✍️ ✅ |
iainmcgin
added this pull request to stack #497
October 1, 2026 18:26
iainmcgin
marked this pull request as ready for review
October 1, 2026 18:47
zayedu
pushed a commit
to zayedu/buffa
that referenced
this pull request
Oct 2, 2026
Adds `CodecStrategy::Table`, the generator for the table codec proposed in anthropics#463. `Unrolled` stays the default. Stacked on anthropics#468 (the `buffa::table` runtime), which is stacked on anthropics#467. ```rust buffa_build::Config::new() .codec_strategy(CodecStrategy::Table) .codec_strategy_in(CodecStrategy::Unrolled, &[".wa.Message"]) ``` The plugin takes `codec_strategy=table` and repeatable `codec_strategy_in=<path>=<strategy>`. Rules match like `preserve_unknown_fields_in`: prefix, last match wins, rules over the global setting. A message with a `oneof`, `map` or group field, a custom string, bytes or collection type, `MessageSet`, or extension ranges with JSON stays unrolled, and so does every message that holds one. `table_plan.rs` computes that closure and reports it in one `TableCodecFallbackSummary` warning, silent when the user's own `Unrolled` rule is the cause. A rule that names such a message by its exact path is an error. `compile()` errors on rustc older than 1.77, and the MSRV job now also tests the table code on 1.77, so its timeout goes from 10 to 20 minutes. The 1,644 KB to 817 KB figure in anthropics#463 was measured with oneofs and maps flattened, so a real schema saves less until the follow-up that lets table messages hold unrolled, extern and well-known-type children. The conformance suite does not run under `Table`: `TestAllTypesProto3` has oneofs and maps, so its messages fall back. Parity rests on `buffa-test`, which compiles each schema twice under renamed packages and compares bytes, sizes, decoded values and errors, including on truncated, bit-flipped and noise input. Under an experiment that set all 66 `buffa-test` protos to `Table`, 602 tests pass with 119 table messages. A table message differs from an unrolled one in three ways, documented in the guide: a length past the end of its enclosing message fails at once with `UnexpectedEof`; `merge_field` cannot gather a non-contiguous buffer, so a type that another crate or run uses as a group or `DELIMITED` field must stay `Unrolled`; and `clear()` releases capacity. The generated code is tied to `buffa::table`, so regenerate it whenever `buffa` updates. About 1,300 lines are outside test files, well over the 250-line guideline. Followed by anthropics#475 (message fields that hold messages without a table), anthropics#476 (`oneof`) and anthropics#477 (`map`), stacked in that order.
azdagron
previously approved these changes
Oct 7, 2026
github-merge-queue
Bot
removed this pull request from the merge queue because a pull request earlier in the stack was removed
Oct 7, 2026
A map field is one table entry of kind Map whose descriptor holds the entry's key and value kinds and two functions instantiated for the collection type: one to iterate it, one to insert a decoded entry. The key and value are read and written through the interpreters that already exist, so a map adds one interpreter arm per pass and no code per key or value type. Decoding mirrors the unrolled codec: each entry counts against the element-memory limit, a message value counts against the recursion limit, an entry with an unknown closed-enum number is kept whole as an unknown field, and a repeated key or value in one entry keeps the last.
The planner accepts a map with the default collection and default string and bytes types, and follows the value of a map of messages as a child. A custom collection or element type still falls back.
Every key type, every value type, entries of unusual shape, closed and open enum values, the element-memory, unknown-field and recursion limits, and HashMap and BTreeMap collections.
The guide, DESIGN.md, the CodecStrategy documentation and the table module documentation listed maps among the fields that keep a message unrolled.
azdagron
approved these changes
Oct 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Messages with
mapfields use the table underCodecStrategy::Table. Amapfield is one table entry of kindMap. ItsMapVtholds the key and value kinds and two functions instantiated for the collection type (iterate, insert a decoded entry); keys and values go through the existing interpreters. A map value whose message has no table is reached through itsMessageimpl, as in #475. Stacked on #476.On
whatsapp.proto(sizetext bytes, fat LTO,panic=abort, owned types only), all 752 messages use the table (749 on #476; the three left had maps). Moving them adds 6,824 bytes atz, the map interpreters. Per map field, on a synthetic schema of 600 maps over 96 key/value type pairs, table against unrolled, in text bytes: 152 vs 284 atz, 425 vs 434 ats, 536 vs 465 at 3.Time, bare metal (c7i.metal-24xl),
opt-level=3, table over unrolled at 8, 64 and 512 entries: decode 1.51x, 1.30x, 1.34x; encode 2.22x, 2.00x, 2.03x;compute_size1.48x, 1.49x, 1.45x. One run, ±5%.Decoding mirrors unrolled code: element-memory and recursion limits, unknown closed-enum entries kept as unknown fields, and a failed merge leaves the same state. One difference:
clear()on a table message releases map capacity, where unrolled code keeps it.The loops test for a map before dispatching. Naming the map interpreters in the dispatch arms makes
opt-level=3builds 2.6% larger on a schema without maps and 4% larger onwhatsapp.proto.buffa::tablegains publicMapVt,DirectMsgVtandKindMarker(andKindSlotnow requiresKindMarker), documented as unstable support code. A custom collection, string or bytes type in a map still falls back, with the reason "has a field with a custom string, bytes or collection type".