codegen: precompute corpus-wide context once and share across generate() calls

[th0114nd]

Closes #400.

What this does

CodeGenContext::with_extern_resolution rebuilds msg_index, resolve_unboxed_variants, resolve_inlined_fields, and the per-file comment map from the whole corpus on every generate() call, even though none of that depends on effective_extern_paths — the one thing that legitimately varies per call in a workspace that generates one independent Cargo crate per proto package. See #400 for the concrete case that motivated this (a ~2900-package, one-crate-per-package workspace) and the measured impact.

Adds CodeGenConfig::shared_corpus_context: Option<Arc<SharedCorpusContext>>. SharedCorpusContext is a new opaque struct, produced once via precompute_shared_corpus_context(files, config), holding Arc-wrapped copies of the three corpus-wide values. When set, with_extern_resolution reuses them via Arc::clone instead of recomputing. None (the default) recomputes fresh every call — byte-identical to today's behavior, no change to any existing consumer.

Perf numbers

Full-corpus codegen (2898 crates), same machine (Apple M4 Max, 16 cores: 12 performance + 4 efficiency, 64 GB RAM), two runs each, alternating order to cancel drift:

	Control	With shared corpus context	Δ
Wall clock	198.0s (194.6s, 201.5s)	127.9s (127.7s, 128.1s)	1.55x faster
Total CPU (user)	2680.1s	1706.8s	36.3% less
Max RSS	10.63 GiB	10.11 GiB	4.9% less
Peak memory footprint	8.54 GiB	8.00 GiB	6.3% less

RSS and footprint both improve: reusing one Arc-shared copy across ~2900 calls costs less memory than each call independently allocating and then dropping its own fresh copy of the same data. Parallelism is consistent between variants: (user+sys)/wall ≈ 14x for both, matching this machine's 16 physical cores — the speedup comes from doing less total work, not from a change in concurrency.

Public API surface

This is worth calling out explicitly since it's a permanent addition to buffa-codegen's public surface, not just an internal change:

• New struct SharedCorpusContext — opaque (no public fields, no public methods beyond the auto-derived Debug).
• New function precompute_shared_corpus_context(files: &[FileDescriptorProto], config: &CodeGenConfig) -> SharedCorpusContext.
• New field CodeGenConfig::shared_corpus_context: Option<Arc<SharedCorpusContext>>, None by default.

CodeGenConfig is #[non_exhaustive], so this can't break any external exhaustive-literal construction. Nothing existing changes signature or behavior. The maintenance cost going forward is the usual one for an opt-in cache-sharing knob: SharedCorpusContext's precondition (same files + unboxed_oneof_fields + pointer_fields across every call sharing one context) is enforced only by documentation, not at runtime — a mismatch silently produces stale output rather than erroring. That mirrors how shared_descriptor_pool_root (#397/#399) already relies on caller discipline for its own precondition, so it's consistent with existing crate convention rather than a new category of risk.

Relationship to #397/#399

Complementary but independent. Both target the same one-crate-per-package workspace shape, but this is orthogonal to reflection/descriptor-pool embedding — no shared code path, no ordering dependency. The perf numbers above were measured with shared-fds off specifically to isolate this change's effect. Either PR can land first or stand alone.

Testing

• shared_corpus_context::shared_context_matches_fresh_computation: generate() produces byte-identical output whether shared_corpus_context is set or not, across a corpus built to exercise the real risk cases — a non-recursive inline field, a self-recursive field (asserts its view type stays boxed either way — owned message fields use MessageField<T>, which is indirected internally regardless of recursion), and a oneof variant matched by a blanket unboxed_oneof_fields rule.
• shared_corpus_context::shared_context_across_separate_per_crate_calls: the actual production pattern — one SharedCorpusContext reused across separate generate() calls that each emit a different file subset with a different extern_paths mapping for the files they don't emit. Confirms effective_extern_paths still resolves correctly per call under sharing (not just that a single combined call is internally consistent), and that one call's local config can't bleed into another's.
• Full correctness of the downstream cross-crate use case verified separately via hash-diff against a known-good baseline across the full 25,779-file corpus (not part of this repo, described in generate(): avoid recomputing corpus-wide oneof/comment analysis on every call in multi-crate workspaces #400).
• cargo fmt, cargo clippy --workspace --all-targets -- -D warnings, and cargo doc --workspace --all-features all clean; full local pre-push CI-equivalent (lint, doc, test, changelog, generated-code checks) passed; MSRV (1.75, workspace --all-targets) and the 32-bit (i686-unknown-linux-gnu) workspace check both verified locally as well.

[github-actions]

All contributors have signed the CLA ✍️ ✅
_{Posted by the CLA Assistant Lite bot.}

[th0114nd]

Update: re-measured now that #397/#399 is merged, and rebased on current main

Rebased this branch on current main (no conflicts of substance — CodeGenConfig gained both fields cleanly). Since shared_descriptor_pool_root is now merged, the realistic baseline for this workspace shape already has shared-fds on, so I re-measured with that as the comparison point instead of the original shared-fds-off isolation (same machine, same methodology — two runs each, alternating order):

	Shared-fds only (baseline)	Shared-fds + shared corpus context	Δ
Wall clock	115.1s (111.0s, 119.1s)	40.1s (40.1s, 40.1s)	2.87x faster
Total CPU (user)	1601.4s	498.8s	68.9% less
Max RSS	3.35 GiB	2.38 GiB	29.2% less

Bigger win than the original 1.55x: with shared-fds removing most of the per-crate descriptor-byte-encoding cost, the corpus-wide work this PR caches is a larger share of what's left per crate. RSS still improves, not regresses.

[iainmcgin]

[claude code]

The caching is sound — the comment map holds raw text and cross-reference resolution happens at emission against the per-call type_map, and resolve_unboxed_variants/resolve_inlined_fields read only the two rule lists — so per-call extern_paths are safe, and generate()/for_generate keep their signatures. Two things I changed before this locks in at 0.9.2, pushed as 06fba91 (rebased on main):

• The cache had no key, so a context built from another corpus or under other unboxed_oneof_fields/pointer_fields silently miscompiled — the doc said so, but the fast path should not be the one that can be wrong. SharedCorpusContext now records the file names and both rule lists and generate() returns CodeGenError::SharedCorpusContextMismatch { what } on any difference (O(files) name compares; the infallible CodeGenContext::for_generate asserts in debug builds).
• API shape: the constructor is SharedCorpusContext::new(&files, &config) rather than a crate-root free function, the type is Clone (Arcs inside) so CodeGenConfig::shared_corpus_context is Option<SharedCorpusContext> with no caller-side Arc, and Debug prints sizes rather than the corpus comment text (a {config:?} in a build.rs would otherwise dump every comment). A doctest shows the precompute-once pattern and the test suite pins Send + Sync.

Tests: the two equality tests could not fail if the shared context were ignored, so there is now a mismatch test for all three keys, the corpus carries source_code_info so the cached comment map is exercised end to end, and the Box sanity check matches Box<SelfRefView. Fragment reworded with the motivation and (#401, closes #400).

One question for you, not blocking: the pre-pass in with_extern_resolution (package/top-level name collection) and effective_extern_paths are still O(corpus) per call — did the re-measurement isolate the three cached walks, or is there a follow-up in those two?

[sylvesterkaczmarek]

SharedCorpusContext::check() treats matching file count/names as the same corpus, but the cached message-index resolutions and comment map depend on descriptor contents. Changing messages/comments in foo.proto while keeping the filename therefore passes the check and reuses stale codegen state, contrary to the documented mismatch guarantee. Please fingerprint/compare descriptor content and add a same-name/changed-schema regression test.

[th0114nd]

from me:

1. the 7aa84f7 has the same perf
2. I haven't dug exactly into what "legitimately varying" means but claude is reluctant to pursue it
3. there's some cost to doing whole FDP comparison, but its still a net positive. Not sure what level of soundness is needed here but fine with me. Fingerprinting doesn't really work because it's not part of the FDP, and re-fingerprinting it would be slower than doing byte equality. I think calling generate with &[(SHA, FDP)] would be nice but is a breaking API change. Something like comparing array addresses and length is fast and mostly accurate, but misses some edge cases like v.pop(); .push()

from the bot:

Re-measured at 7aa84f7 (full 2895-crate corpus, shared-fds-on baseline — same machine/methodology as the Sept 2 update):

	Shared-fds only (control)	Shared-fds + shared corpus context	Δ
Wall clock	110.8s (109.2, 112.3)	39.8s (40.1, 39.5)	2.79x faster
Total CPU (user)	1551.8s	501.8s	67.7% less
Max RSS	3.48 GiB	2.74 GiB	21.3% less

Matches the claimed numbers within ~4% on wall/CPU (normal noise). Performance is maintained through the rebase and the 06fba91 mismatch-check addition — no measurable per-call overhead from the name-only check() at this corpus size.

@iainmcgin — re: the pre-pass and effective_extern_paths: confirmed still O(corpus) per call (context.rs:301-351), and it's not a drop-in extension of what's already cached. Unlike the three shared fields, that pre-pass filters on effective_extern_paths itself (the per-type extern_path overrides from #111, line 322-327) — the one input this feature's contract already documents as legitimately varying per call. So sharing it isn't free the way the other three were; it'd need to be keyed additionally on the override subset, and only pays off if that subset is actually stable across a workspace's per-crate calls (plausible, since it's static config rather than the auto-generated per-package routing, but not verified here). I'd rather scope that as a follow-up than fold it into this PR.

@sylvesterkaczmarek — re: the fingerprint/regression-test request: you're right, and it's a real gap, not just a theoretical one. Pushed a fix as 2441e95: SharedCorpusContext::check() now compares the full corpus by FileDescriptorProto structural equality (it already derives PartialEq) instead of just file names, plus a regression test (same_name_changed_content_is_rejected) that mutates a message while keeping the filename and asserts the mismatch is caught.

We looked at two cheaper alternatives first and rejected both:

• A pointer/length identity check on the files slice instead of full comparison. Rejected — SharedCorpusContext carries no lifetime tying it to the caller's files, so a Vec can keep the same allocation (same pointer, same length) across an ordinary pop() + push() of a different file. That's not an adversarial edge case, it's a completely ordinary way for content to change without the pointer moving, so identity isn't sound here.
• An API change carrying a precomputed hash alongside each file (generate(&[(Sha, FileDescriptorProto)], ...)), computed once upstream instead of re-derived per call. This would be cheaper and still correct, but breaks generate()'s signature for every caller in the crate, not just shared_corpus_context users — too big a change for what this PR needs right now.

So we measured the direct fix instead:

	Name-only check() (06fba91)	Full-FDP-equality check() (2441e95)	Δ
Wall clock	38.6s	48.1s	+25% slower
User CPU	483.6s	619.4s	+28% more

So versus the shared-fds-only baseline (110.8s), this PR with the correctness fix lands at ~2.3x faster, not 2.79x — a real cost, but still a substantial win, and the only version of this feature we think is actually safe to merge. If the extra ~25% becomes worth clawing back later, the (Sha, FileDescriptorProto) API change above is the follow-up path, but we'd rather ship correct now and optimize later than ship fast now with a documented hole.

Pushed 2441e95 directly since this repo's been running on a push-then-discuss cadence this review — feel free to revert if you'd rather land this a different way.

[iainmcgin]

[claude code]

Structural comparison is the right call — a Vec reusing its allocation across a pop/push is exactly the case a name-only key misses, and 2.3× over no sharing with the full compare is still the headline. Pushed 7eea522: the new Fixed fragment described a fix to a type that first ships in this release, so a reader of the 0.9.2 notes would take it for a regression fix; folded its content (structural compare, the O(corpus) cost and your measurement) into the Added entry instead. No code change.