Raise on Dataset.rename_columns name collision with an existing column#8323
Open
roli-lpci wants to merge 1 commit into
Open
Raise on Dataset.rename_columns name collision with an existing column#8323roli-lpci wants to merge 1 commit into
roli-lpci wants to merge 1 commit into
Conversation
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Dataset.rename_column(singular) already rejects a new column name that collides with anexisting column:
Dataset.rename_columns(plural, dict-based) has no equivalent check. Renaming a column to aname that collides with an existing, untouched column silently produces a
Datasetwhoseunderlying Arrow table has two columns with the same name, while
dataset.features(builtfrom a Python dict) collapses to one entry for that name. The two become inconsistent, and in
the reproduction below the renamed column's data becomes unreachable through the normal read
paths (
ds[:],to_dict(),to_pandas(), iteration, ...) with no exception raised anywhere.Since
DatasetDict.rename_columnsforwards directly toDataset.rename_columnsper split,the same corruption is reachable from
DatasetDicttoo.Fix
Add the same collision check
rename_columnalready performs, generalized to the mappingcase: a new name is rejected only if it collides with a column that isn't itself being renamed
away in the same call (so swaps like
{"a": "b", "b": "a"}and no-op renames like{"a": "a"}keep working).Test
Added to
tests/test_arrow_dataset.py::BaseDatasetTest.test_rename_columns:ValueErroron a colliding rename ({"col_1": "col_2"}wherecol_2alreadyexists and isn't being renamed).
{"col_1": "col_2", "col_2": "col_1"}) still succeeds, to confirm the fixdoesn't over-reject legitimate renames.
Confirmed the added assertions fail on
mainwithAssertionError: ValueError not raised(not a generic error) and pass with the fix (
pytest tests/test_arrow_dataset.py -k rename:2 failed on the unmodified baseline, 4 passed with the fix). A prior full run of
tests/test_arrow_dataset.pywith this change: 360 passed, 61 skipped, 1 pre-existingunrelated failure (a PyTorch-formatting test that fails identically on
mainbecause torchisn't installed in the test env).
Compatibility
Pure Python, no new imports, no version-gated syntax. Only affects the specific case where a
rename_columnsmapping's target name collides with an existing column outside the mapping;that call previously produced silently-corrupted output, so this is a bug fix, not a behavior
users could be relying on for correct results.