Skip to content

Add allocation budget to parsing - #97

Open
anuraaga wants to merge 13 commits into
bufbuild:mainfrom
anuraaga:allocation-budget
Open

anuraaga wants to merge 13 commits into
bufbuild:mainfrom
anuraaga:allocation-budget

Conversation

@anuraaga

@anuraaga anuraaga commented Sep 7, 2026 •

Copy link
Copy Markdown
Collaborator

To improve safety for protobuf users, including when they use RPC frameworks like connect, we are implementing allocation budgets for parsing. Large schemas can have many fields allocated per message, and for a language like Python where the memory overhead per type is high, this can quickly balloon - the unit test case demonstrates it with FileDescriptorProto, which can amplify from 2-3 bytes to 700 bytes.

This adds a public API, I called it allocation_limit, the approximate bytes to cap allocation at. We want to make sure to use the same name in protobuf-es, let me know if any preference.

Some notes

  • Allocation tracking of a new protobuf message, the most important amplification factor, is precise since CPython provides the exact number
  • String tracking is quite approximate since the presence of non-ascii, or emoji, change the string representation from ascii, ucs2, ucs4. For binary, since we know the utf8 length ahead of parse, we charge it, which slightly overcharges for ucs2 and ucs4. For json where we don't, we just use the string length, which slightly undercharges for ucs2 and ucs4.
  • While it could be possible to precisely measure power-of-2 allocations in grown containers, it doesn't seem worth it, I just charge per slot
  • Header overhead will be slightly different among python versions, especially gil vs free-threaded python. Always fixed, relatively small delta

Most importantly, all approximations are at most 2-3x (strings), not the potential 300x of a large schema in a message.

Incrementing and checking a int is almost no overhead and noise in benchmarks, both native and pure python.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant