Cast multi-dimensional numpy arrays when inferring IterableDataset features#8319
Open
vineethsaivs wants to merge 1 commit into
Open
Cast multi-dimensional numpy arrays when inferring IterableDataset features#8319vineethsaivs wants to merge 1 commit into
vineethsaivs wants to merge 1 commit into
Conversation
…atures `_infer_features_from_batch` built the Arrow table with `pa.Table.from_pydict(batch)`, which only accepts 1-dimensional array values. When a map function yields multi-dimensional numpy arrays (e.g. an image or any 2D+ array), `IterableDataset._resolve_features()` therefore crashed with `pyarrow.lib.ArrowInvalid: Can only convert 1-dimensional array values`. Every other Arrow-table builder in this file first routes the data through `cast_to_python_objects(..., only_1d_for_numpy=True)` (see lines 173 and 184), which recursively turns multi-dimensional numpy arrays into nested lists of 1-D arrays that PyArrow accepts; the non-streaming `Dataset`/`ArrowWriter` path does the same. Route the batch through the same helper (already imported in this module) so feature inference matches the rest of the codebase. Fixes huggingface#7100. Adds a regression test asserting features resolve from a map that yields multi-dimensional numpy arrays.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
IterableDataset._resolve_features()crashes when amapfunction yields multi-dimensional numpy arrays:Feature inference goes
_resolve_features()->_infer_features_from_batch(...)->pa.Table.from_pydict(batch), and PyArrow'sfrom_pydictonly accepts 1-dimensional array values, so any 2D+ numpy array in a column crashes it.Every other place in
iterable_dataset.pythat builds an Arrow table from raw examples first routes the data throughcast_to_python_objects(..., only_1d_for_numpy=True)(lines 173 and 184), which recursively turns multi-dimensional numpy/tensor values into nested lists of 1-D arrays that PyArrow accepts. The non-streamingDataset/ArrowWriterpath does the same._infer_features_from_batchwas simply missing that cast.Fix
Route the batch through
cast_to_python_objects(batch, only_1d_for_numpy=True)beforefrom_pydict. The helper is already imported in this module, and it is a no-op for ordinary Python data, so plain batches infer identically.Fixes #7100.
Test
tests/test_iterable_dataset.py::test_iterable_dataset_resolve_features_from_multidim_numpyresolves the features of anIterableDatasetwhosemapyields multi-dimensional numpy arrays. It raisesArrowInvalidbefore the fix and passes after (resolving to a nestedList(...)).