Support batched=True in Dataset.to_dict

[vineethsaivs]

What does this do?

Dataset.to_dict(batch_size=..., batched=True) silently ignores both arguments and always returns the full dict, even though its signature, docstring ("Set to True to return a generator that yields the dataset as batches"), and -> Union[dict, Iterator[dict]] return type all promise batching. The sibling methods to_pandas and to_polars implement batching; to_dict was missing the branch.

Because it returns a plain dict instead of the promised generator, iterating the result as documented walks the dict keys and raises:

from datasets import Dataset
ds = Dataset.from_dict({"a": list(range(10))})
out = ds.to_dict(batched=True, batch_size=3)
type(out)                 # dict, not a generator
for batch in out:
    batch["a"]            # TypeError: string indices must be integers, not 'str'

Fix

Mirror the to_pandas/to_polars structure: return the full dict when batched=False, otherwise return a generator that yields batch_size-row slices (defaulting batch_size to config.DEFAULT_MAX_BATCH_SIZE). A small local closure keeps the per-batch JSON-field decoding from being duplicated across the two branches.

After the fix:

[len(b["a"]) for b in ds.to_dict(batched=True, batch_size=3)]   # [3, 3, 3, 1]

Backward compatible: batched defaults to False, so every existing .to_dict() call still returns a plain dict (verified byte-identical). Index mapping (select(...)) and Json()-feature decoding both work in batched and non-batched modes.

Test

Added a "Batched" block to tests/test_arrow_dataset.py::test_to_dict, mirroring the existing test_to_pandas/test_to_polars coverage: it asserts each yielded batch is a dict with the right columns and no more than batch_size rows. It fails before this change (the returned dict is iterated as keys) and passes after.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[lhoestq]

can you run make style to fix the code formatting ?

[vineethsaivs]

Done, ran make style and pushed. The only reformat was collapsing the batched generator return onto one line.

[lhoestq]

lgtm :)