Skip to content

Avoid allocating duplicate-key tracking per message in fromJson - #1565

Open
intech wants to merge 1 commit into
bufbuild:mainfrom
Connectum-Framework:up/fromjson-lazy-seen
Open

intech wants to merge 1 commit into
bufbuild:mainfrom
Connectum-Framework:up/fromjson-lazy-seen

Conversation

@intech

@intech intech commented Oct 10, 2026

Copy link
Copy Markdown

fromJson allocated a Set of seen fields and a Map of seen oneofs for every decoded JSON object, only to detect duplicates. Most objects need neither:

  • A field can only be set twice in one object through two distinct keys: its proto name and its JSON name.
  • Most messages set at most one oneof.

This change:

  • Fields: a field whose JSON name differs from its proto name looks up its other key with hasOwnProperty. The Set is only created when an object holds both keys of some field.
  • Oneofs: the first oneof that is set is kept in two locals. The Map is only created when a second oneof is set.
  • Behavior: the order in which keys are read and errors are raised is unchanged:
    • the first occurrence of a duplicated field is decoded before the second is rejected;
    • a null oneof scalar still counts as present;
    • error messages are identical.

New tests cover a duplicate given by its JSON name first, an invalid first occurrence before the duplicate, and conflicts in a oneof other than the first one set.

Benchmarks

The numbers come from packages/protobuf-bench, unchanged corpus, run on a GitHub-hosted runner (Intel Xeon Platinum 8370C, Node.js 24.5.0).

  • Method: 30 pairs of fresh processes, before and after, in random order within each pair. Each process is pinned to one CPU.
  • Columns: ns/op is the median over passes of each pass's p50. Δ is the median of the per-pair ratios, and "faster in" counts the pairs in which this change was faster.
  • Threshold: cases with a Benjamini-Hochberg-corrected sign test p ≤ 0.05 and |Δ| ≥ 2.2 %:
case before (ns/op) after (ns/op) Δ ops/s faster in
fromJson/user-tiny 683 488 +38.2 % 30/30
fromJson/user-normal 2,000 1,558 +29.2 % 30/30
fromJson/repeated-message 3,432,226 3,088,395 +9.7 % 30/30
fromJson/map-message 3,615,774 3,321,766 +9.1 % 29/30
fromJson/scalar 3,576 3,283 +8.9 % 30/30
fromJson/repeated-scalar 5,681 5,466 +3.5 % 27/30

All other cases of the corpus (create, toBinary, fromBinary, toJson, BinaryWriter/*, BinaryReader/*, and fromJson/general and fromJson/map-scalar) show no significant difference.

We measured the same way on production-shaped payloads as well. The fromJson gains are:

payload gain
OTLP trace export, 100 spans +26.8 %
OTLP logs +28.6 %
OTLP metrics +23.9 %
Kubernetes pod list +21.1 %
small three-field message +26.6 %

We can share these fixtures if they are useful.

Size

from-json.js minified (esbuild): 13,809 → 13,979 B (+170 B), gzip 4,483 → 4,534 B (+51 B). None of the bundle-size entry points import fromJson, so its table does not change.

🤖 Generated with Claude Code

https://claude.ai/code/session_01JCd4TPygQp4GPxMsTMm2Bi

Every JSON object decoded by fromJson allocated a Set of seen fields and a
Map of seen oneofs, only to detect duplicates. A field can only be set twice
in one object through two distinct keys, its proto name and its JSON name,
and most messages set at most one oneof.

A field whose JSON name differs from its proto name now looks up its other
key in the object; the set is only created when an object holds both keys
of some field. The first oneof that is set is kept in two locals, and the
map is only created when a second oneof is set. The order in which keys are
read and errors are raised is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JCd4TPygQp4GPxMsTMm2Bi
@vercel

vercel Bot commented Oct 10, 2026

Copy link
Copy Markdown

@intech is attempting to deploy a commit to the bufbuild Team on Vercel.

A member of the Team first needs to authorize it.

@CLAassistant

CLAassistant commented Oct 10, 2026 •

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@intech

intech commented Oct 10, 2026

Copy link
Copy Markdown
Author

Hey! 👋

This continues the performance work discussed in #333, after the growable BinaryWriter (#1108) and the follow-ups that shipped in 2.14–2.16.

A note on how this was made: the code changes and the benchmarks were produced by an AI assistant under my direction, and I reviewed every step.

The full history the measurement method, A/A runs to calibrate the noise, CPU profiles, a review of each change, and the raw benchmark reports is in our fork:
fork PR:
Connectum-Framework#27 (benchmark reports are the bot comments on that PR)

benchmark tooling: Connectum-Framework#26

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants