Repository navigation
Report truncated or missing data with the right error - #25414
Open
shoumikhin wants to merge 4 commits into
Open
shoumikhin wants to merge 4 commits into
shoumikhin wants to merge 4 commits into
Conversation
FlatTensorDataMap::load reports most corrupt .ptd files as InvalidExternalData. But 3 kinds of short data were not checked before a read: - data shorter than the header itself, - data shorter than the flatbuffer region that the header describes, - a flatbuffer region shorter than the 8 bytes that hold its root offset and identifier. In the first 2 cases the read fails inside the data loader, and the loader's error comes through instead, for example InvalidArgument from BufferDataLoader. In the last case the identifier was read before the existing minimum-size check. With a FileDataLoader, an empty region gives no memory at all, so loading such a file crashed the process. load() now checks all 3 before it reads, and returns InvalidExternalData. Errors that the loader returns for other reasons still come through unchanged. This changes an error code. Code that expected InvalidArgument for such a truncated .ptd file should expect InvalidExternalData.
MergedDataMap::load skips data maps that have no keys, and then refuses to merge with InvalidArgument when no map is left. A .ptd file with no tensors is valid, and FlatTensorDataMap loads it. But Module merges every .ptd file it is given, so a Module given only such a file failed to load. Maps with no keys are now accepted and add nothing, so the merged map can have no keys. Passing no data map at all, or only null ones, is still InvalidArgument.
A program can keep its weights in a separate .ptd file. When a method of such a program is loaded without that file, then for the tensors that the runtime loads itself, it returned InvalidState for constant weights and InvalidExternalData for mutable ones. Neither code says what happened. Nothing is in a bad state, and no data was malformed. The data was simply not provided. Both cases now return NotFound, the code the runtime already returns when a .ptd file is given but lacks a key. The error log names the tensor and says that no .ptd data was provided. C++ code that checked for InvalidState or InvalidExternalData after loading a method without its .ptd data should check for NotFound instead.
Program::load reads the 64 header bytes, and then the program, whose size the header states. When the data is shorter than either, the read fails inside the data loader, and the loader's error comes through instead, for example InvalidArgument from BufferDataLoader, although the cause is a malformed program. The program size is already checked against the data for most headers, but not for headers written before segment_data_size existed. When the loader reports its size, Program::load now checks both lengths against it first and returns InvalidProgram. The program size from the header is compared as the 64-bit value it is stored as, before it is narrowed to size_t. Program::load now asks the loader for its size once, and returns the loader's error where it used to abort on a failed size query. A loader that cannot report its size still loads programs whose header does not need it, as before. This changes an error code. Code that expected the loader's error for such truncated program data should expect InvalidProgram. BundledModule with a file that is not a bundled program is one such case, so BundledModuleTest.TestNonBPFile now expects InvalidProgram.
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/25414
Note: Links to docs will display an error until the docs builds have been completed. ✅ No FailuresAs of commit 6596741 with merge base ba0d826 ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What is wrong today
FlatTensorDataMap::loadreads some short.ptddata before checking it: data shorter than the header, data shorter than the flatbuffer region the header describes, and a flatbuffer region under 8 bytes. The first 2 return the data loader's error, such asInvalidArgument. The last one reads past the region, and an empty region crashes with aFileDataLoader.MergedDataMap::loadrefuses to merge when no map has any keys. A.ptdfile with no tensors is valid, so aModulegiven only such a file fails to load..ptdfile that holds its weights fails withInvalidState(constant tensors) orInvalidExternalData(mutable tensors). Neither says the data is missing.Program::loadreturns the data loader's error, notInvalidProgram, when the data is shorter than the 64-byte header, or shorter than the program size in an older header that has nosegment_data_size.What this changes
.ptddata is reported asInvalidExternalData, checked before any read.InvalidArgument..ptdfile isNotFound, the code already used for a missing key. Data that a backend reads on its own is not changed.InvalidProgram. A loader that cannot report its size still loads programs whose header does not need it.Program::loadnow returns the loader's error where it used to abort on a failed size query.Results that callers can see:
.ptddataInvalidExternalData.ptd, constant tensorInvalidStateNotFound.ptd, mutable tensorInvalidExternalDataNotFoundInvalidProgram.ptdfiles with no tensorsInvalidArgumentSo
BundledModuleTest.TestNonBPFilenow expectsInvalidProgram.What was tested
FlatTensorDataMapTest,MergedDataMapTest,MethodTest,GetTensorDataPtrTestorProgramTestthat fail without its fix.ProgramTest.LoadLegacyHeaderWithoutDataSizechecks that a loader that cannot report its size still loads an older program (it passes with and without the change).