Skip to content

Pass large worker tensors through sealed memfds - #136

Open
arjunnai wants to merge 5 commits into
mlc-ai:mainfrom
arjunnai:perf/memfd-tensor-transfer
Open

arjunnai wants to merge 5 commits into
mlc-ai:mainfrom
arjunnai:perf/memfd-tensor-transfer

Conversation

@arjunnai

@arjunnai arjunnai commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

Large tensors cross the server/worker boundary by pickling through the worker's control socket, including on cache hits. This passes blobs of at least 1 MiB, both inputs and results, as sealed memfds over the same socket instead. The receiver checks the seals and exact size and maps them read-only. Small blobs, non-Linux hosts and Python builds without memfd_create (a libc fallback covers standalone builds) keep the current path. Two follow-ups reuse sealed buffers from the input cache and hash immutable upload snapshots in parallel.

Vector add, 64 MiB tensors, median of 30 trials in balanced order, sandbox off (RTX 5090 / RTX 2070):

cell main this PR
reused worker 997 / 982 ms 643 / 657 ms (−36% / −33%)
fresh worker 1335 / 1310 ms 940 / 940 ms (−30% / −28%)

4 MiB reused drops 28% / 17%. 4 KiB requests do not use the new path and are unchanged. Measured with the overhead harness from #127 on bare metal.

Testing:

  • Full test suite on main and this branch (RTX 5090 + RTX 2070): no new failures. New tests cover the fallback, descriptor cleanup after crashes and timeouts, rejected handles, and oversized results.
  • Real-kernel run with bubblewrap: transfers work inside the sandbox and the parent's descriptor count is unchanged.

arjunnai and others added 5 commits October 18, 2026 17:03
Use request-local sealed buffers and a libc fallback when Python omits memfd_create. Preserve the bytes path and close mappings before sandbox cleanup.
Send large results over the worker socket as sealed buffers. Validate size and seals before copying, and retain the HTTP response-size error.
Keep a bounded descriptor set with cache entries and duplicate handles for each send. Close cached descriptors on eviction and shutdown.
Pipeline large blob hashes across uploads and resolve them before exposing instructions or transmitting blobs. Hash only owned snapshots.
@arjunnai
arjunnai marked this pull request as ready for review October 26, 2026 01:25
Copilot AI balanced review requested due to automatic review settings October 26, 2026 01:25

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants