Skip to content

[Web] Give callers their own reference to cached shape tuples - #20501

Open
akaashrp wants to merge 2 commits into
apache:mainfrom
akaashrp:fix/web-runtime-lifetime
Open

akaashrp wants to merge 2 commits into
apache:mainfrom
akaashrp:fix/web-runtime-lifetime

Conversation

@akaashrp

@akaashrp akaashrp commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

makeShapeTuple returns the object the shape cache owns. The cache is a 256-entry LRU that disposes what it evicts, so once a prefill touches more than 256 distinct shapes a caller can hold a tuple that has already been freed. WebLLM hits this on a Radeon iGPU as Object has already been disposed followed by a device hang (mlc-ai/web-llm#844). #20130 kept evicted tuples alive until teardown, whereas here the tuple is owned by the caller.

  1. makeShapeTuple takes a reference on the cached handle and returns a new object attached to the current scope, so eviction only drops the cache's reference. Like every other function that returns a TVM object, it now requires an open scope.
  2. setDeviceLostAutoDispose(false) lets an owner that already disposes the instance itself keep it from disposing on the device-lost promise

A cache hit costs 15 to 30 ns more per call. Tests cover eviction while a caller holds the tuple and deferred disposal after device loss.

Signed-off-by: Akaash Parthasarathy <akaashrp@gmail.com>
Signed-off-by: Akaash Parthasarathy <akaashrp@gmail.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant