Skip to content

fix(data): preserve empty datasets during prompt length filtering - #2462

Open
GoldArowana wants to merge 1 commit into
THUDM:mainfrom
GoldArowana:fix/empty-dataset-prompt-length-20261011
Open

GoldArowana wants to merge 1 commit into
THUDM:mainfrom
GoldArowana:fix/empty-dataset-prompt-length-20261011

Conversation

@GoldArowana

Copy link
Copy Markdown
Contributor

An empty JSONL file or a valid @[0:0] slice currently crashes during Dataset(..., max_length=8) construction because filter_long_prompt indexes origin_samples[0]. Return the original empty list before inspecting the first prompt.

This preserves existing Dataset/filter semantics: empty input is already accepted with max_length=None, and nonempty input may become empty after length filtering. It does not establish that empty data can complete rollout, evaluation, or training; downstream consumers retain their own validation.

Extend the existing registered CPU test with actual temporary JSONL files and a locally constructed Transformers fast tokenizer backed by WordLevel (no model downloads). Cover empty files and empty slices with and without a length limit, an 8-token boundary and 9-token overlong prompt, all-filtered input, empty-list identity with and without a processor, and existing ordering behavior. The file retains NUM_GPUS = 0.

Validation on Linux / Python 3.12:

  • Before the guard: python -m pytest tests/test_filter_long_prompt.py tests/test_read_file_slicing.py — 4 failed, 20 passed; exit 1, with the expected empty-input IndexErrors.
  • After the guard: the same command — 24 passed; exit 0.
  • Official CI-shaped entry: python tests/test_filter_long_prompt.py — 14 passed; exit 0, including after formatting.
  • Normal commit ran all applicable repository pre-commit hooks successfully; git diff --check passed.
  • Independent read-only review found no blocking correctness, coverage, or CPU CI issues.

Tests used CPU-only, offline settings: CUDA_VISIBLE_DEVICES='' SLIME_ACCELERATOR=cpu HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 PYTHONDONTWRITEBYTECODE=1 PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 PYTEST_ADDOPTS='-p no:cacheprovider' PYTHONPATH=.. No GPU, training, rollout service, or weight-download checks were run. This change covers a CPU data-filtering boundary.

Related PRs #2365, #2449, #2126 and #2363 and their discussions were refreshed before publication; their epoch-sampling and multimodal changes do not address this empty-input guard.

AI assistance was used for reproduction, implementation, tests, and an independent review; the reported validation commands were actually executed.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant