Skip to content

Fix Dataset.take raising IndexError when taking more elements than available#8312

Open
vidigoat wants to merge 1 commit into
huggingface:mainfrom
vidigoat:fix/dataset-take-clamp-length
Open

Fix Dataset.take raising IndexError when taking more elements than available#8312
vidigoat wants to merge 1 commit into
huggingface:mainfrom
vidigoat:fix/dataset-take-clamp-length

Conversation

@vidigoat

@vidigoat vidigoat commented Jul 8, 2026

Copy link
Copy Markdown

Description

Dataset.take(n) is implemented as self.select(range(n)), so when n is larger than the dataset length it raises IndexError: Index N out of range for dataset of size M instead of returning the whole dataset. This is inconsistent with IterableDataset.take, which clamps and returns all available elements — the two were meant to be aligned (see #6813). Dataset.skip already clamps correctly (range(n, len(self))); only take was affected.

This clamps n to len(self) so take returns the whole dataset when asked for more, matching IterableDataset.take and the "first n elements" semantics documented in the method.

Steps to reproduce (before this change)

from datasets import Dataset
Dataset.from_dict({"a": [0, 1, 2]}).take(5)   # IndexError: Index 4 out of range for dataset of size 3.

IterableDataset.take(5) on the same data returns all 3 rows.

Test plan

  • Added test_take to BaseDatasetTest (runs both in-memory and on-disk) covering fewer / exact / more-than-available / zero.
  • The n > len case fails on current main (IndexError) and passes with this change; existing behavior for n <= len is unchanged.

AI disclosure

This fix was prepared with the assistance of an AI coding assistant; the bug was reproduced and the behavior verified before submitting.

…ailable

Dataset.take(n) was implemented as self.select(range(n)), so when n exceeds the
dataset length it raised IndexError instead of returning the whole dataset. This
is inconsistent with IterableDataset.take, which clamps and returns all available
elements (the two were meant to be aligned, see huggingface#6813); Dataset.skip already
clamps correctly. Clamp n to len(self) so take returns the whole dataset when
asked for more.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant