Repository navigation
fix(data): preserve empty datasets during prompt length filtering - #2462
Open
GoldArowana wants to merge 1 commit into
Open
GoldArowana wants to merge 1 commit into
GoldArowana wants to merge 1 commit into
Conversation
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
An empty JSONL file or a valid
@[0:0]slice currently crashes duringDataset(..., max_length=8)construction becausefilter_long_promptindexesorigin_samples[0]. Return the original empty list before inspecting the first prompt.This preserves existing Dataset/filter semantics: empty input is already accepted with
max_length=None, and nonempty input may become empty after length filtering. It does not establish that empty data can complete rollout, evaluation, or training; downstream consumers retain their own validation.Extend the existing registered CPU test with actual temporary JSONL files and a locally constructed Transformers fast tokenizer backed by WordLevel (no model downloads). Cover empty files and empty slices with and without a length limit, an 8-token boundary and 9-token overlong prompt, all-filtered input, empty-list identity with and without a processor, and existing ordering behavior. The file retains
NUM_GPUS = 0.Validation on Linux / Python 3.12:
python -m pytest tests/test_filter_long_prompt.py tests/test_read_file_slicing.py— 4 failed, 20 passed; exit 1, with the expected empty-input IndexErrors.python tests/test_filter_long_prompt.py— 14 passed; exit 0, including after formatting.git diff --checkpassed.Tests used CPU-only, offline settings:
CUDA_VISIBLE_DEVICES='' SLIME_ACCELERATOR=cpu HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 PYTHONDONTWRITEBYTECODE=1 PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 PYTEST_ADDOPTS='-p no:cacheprovider' PYTHONPATH=.. No GPU, training, rollout service, or weight-download checks were run. This change covers a CPU data-filtering boundary.Related PRs #2365, #2449, #2126 and #2363 and their discussions were refreshed before publication; their epoch-sampling and multimodal changes do not address this empty-input guard.
AI assistance was used for reproduction, implementation, tests, and an independent review; the reported validation commands were actually executed.