Skip to content

Report truncated or missing data with the right error - #25414

Open
shoumikhin wants to merge 4 commits into
pytorch:mainfrom
shoumikhin:bad-data-error-codes
Open

shoumikhin wants to merge 4 commits into
pytorch:mainfrom
shoumikhin:bad-data-error-codes

Conversation

@shoumikhin

Copy link
Copy Markdown
Contributor

What is wrong today

  • FlatTensorDataMap::load reads some short .ptd data before checking it: data shorter than the header, data shorter than the flatbuffer region the header describes, and a flatbuffer region under 8 bytes. The first 2 return the data loader's error, such as InvalidArgument. The last one reads past the region, and an empty region crashes with a FileDataLoader.
  • MergedDataMap::load refuses to merge when no map has any keys. A .ptd file with no tensors is valid, so a Module given only such a file fails to load.
  • Loading a method without the .ptd file that holds its weights fails with InvalidState (constant tensors) or InvalidExternalData (mutable tensors). Neither says the data is missing.
  • Program::load returns the data loader's error, not InvalidProgram, when the data is shorter than the 64-byte header, or shorter than the program size in an older header that has no segment_data_size.

What this changes

  1. Short .ptd data is reported as InvalidExternalData, checked before any read.
  2. Maps with no keys are accepted. Passing no map, or only null maps, is still InvalidArgument.
  3. Missing external data for constant and mutable tensors read from the .ptd file is NotFound, the code already used for a missing key. Data that a backend reads on its own is not changed.
  4. When the loader reports its size, short program data is InvalidProgram. A loader that cannot report its size still loads programs whose header does not need it. Program::load now returns the loader's error where it used to abort on a failed size query.

Results that callers can see:

Case Before After
Short .ptd data loader error InvalidExternalData
Missing .ptd, constant tensor InvalidState NotFound
Missing .ptd, mutable tensor InvalidExternalData NotFound
Short program data loader error InvalidProgram
Only .ptd files with no tensors InvalidArgument loads

So BundledModuleTest.TestNonBPFile now expects InvalidProgram.

What was tested

  • Built and ran the full C++ test suite on Linux x86_64 (CMake, Debug).
  • Each commit adds tests in FlatTensorDataMapTest, MergedDataMapTest, MethodTest, GetTensorDataPtrTest or ProgramTest that fail without its fix.
  • ProgramTest.LoadLegacyHeaderWithoutDataSize checks that a loader that cannot report its size still loads an older program (it passes with and without the change).

FlatTensorDataMap::load reports most corrupt .ptd files as
InvalidExternalData. But 3 kinds of short data were not checked before a
read:

- data shorter than the header itself,
- data shorter than the flatbuffer region that the header describes,
- a flatbuffer region shorter than the 8 bytes that hold its root offset
  and identifier.

In the first 2 cases the read fails inside the data loader, and the
loader's error comes through instead, for example InvalidArgument from
BufferDataLoader. In the last case the identifier was read before the
existing minimum-size check. With a FileDataLoader, an empty region gives
no memory at all, so loading such a file crashed the process.

load() now checks all 3 before it reads, and returns InvalidExternalData.
Errors that the loader returns for other reasons still come through
unchanged.

This changes an error code. Code that expected InvalidArgument for such a
truncated .ptd file should expect InvalidExternalData.
MergedDataMap::load skips data maps that have no keys, and then refuses to
merge with InvalidArgument when no map is left. A .ptd file with no tensors
is valid, and FlatTensorDataMap loads it. But Module merges every .ptd file
it is given, so a Module given only such a file failed to load.

Maps with no keys are now accepted and add nothing, so the merged map can
have no keys. Passing no data map at all, or only null ones, is still
InvalidArgument.
A program can keep its weights in a separate .ptd file. When a method of
such a program is loaded without that file, then for the tensors that the
runtime loads itself, it returned InvalidState for constant weights and
InvalidExternalData for mutable ones.
Neither code says what happened. Nothing is in a bad state, and no data was
malformed. The data was simply not provided.

Both cases now return NotFound, the code the runtime already returns when a
.ptd file is given but lacks a key. The error log names the tensor and says
that no .ptd data was provided.

C++ code that checked for InvalidState or InvalidExternalData after loading
a method without its .ptd data should check for NotFound instead.
Program::load reads the 64 header bytes, and then the program, whose size
the header states. When the data is shorter than either, the read fails
inside the data loader, and the loader's error comes through instead, for
example InvalidArgument from BufferDataLoader, although the cause is a
malformed program. The program size is already checked against the data
for most headers, but not for headers written before segment_data_size
existed.

When the loader reports its size, Program::load now checks both lengths
against it first and returns InvalidProgram. The program size from the
header is compared as the 64-bit value it is stored as, before it is
narrowed to size_t. Program::load now asks the loader for its size once,
and returns the loader's error where it used to abort on a failed size
query. A loader that cannot report its size still loads programs whose
header does not need it, as before.

This changes an error code. Code that expected the loader's error for such
truncated program data should expect InvalidProgram. BundledModule with a
file that is not a bundled program is one such case, so
BundledModuleTest.TestNonBPFile now expects InvalidProgram.
@shoumikhin shoumikhin added the release notes: runtime Changes related to the core runtime which loads the program methods, initializes delegates, and runs label Nov 21, 2026
Copilot AI balanced review requested due to automatic review settings November 21, 2026 14:21
@pytorch-bot

pytorch-bot Bot commented Nov 21, 2026 •

Copy link
Copy Markdown

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/25414

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 6596741 with merge base ba0d826 (image):
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Nov 21, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. release notes: runtime Changes related to the core runtime which loads the program methods, initializes delegates, and runs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants