Refactor datapackerdataloader to be modular#19

Merged

lfengad merged 77 commits into

mainfrom

maoshengl/dataset_enhancement

Jun 12, 2026

Commits on Jun 4, 2026

Add design spec: modular dataflow refactor (DataPackerDataLoader)

foreverlms
and
claude
committed
Spec: add full dataloader coverage audit + per-recipe processor placement

foreverlms
and
claude
committed
Spec: record hollow-Processor deviation for VFM/DROID as deferred tech debt

foreverlms
and
claude
committed
Spec: parallel-experiment regression strategy + deferred deletion

foreverlms
and
claude
committed
Spec: add docs/dataflow.md tutorial as a required deliverable

foreverlms
and
claude
committed
Spec: final review fixes (processor placement consistency, status)

foreverlms
and
claude
committed
Add implementation plan 1/N: core dataflow abstraction

foreverlms
and
claude
committed
Spec: add hard resume/saving invariant + concrete regression run config

foreverlms
and
claude
committed
Add implementation plan 2/N: PoolPackingBatcher + VLM migration + regression

foreverlms
and
claude
committed
Rename new loader DataPackerDataLoader -> CosmosDataLoader (+ JointCosmosDataLoader)

foreverlms
and
claude
committed
Add implementation plans 3-6: resume, videophy2, VFM, docs/cleanup

foreverlms
and
claude
committed
feat(dataflow): add four role ABCs (DataDistributor/RawItemProcessor/SampleBatcher/BatchCollator)

foreverlms
and
claude
committed
feat(dataflow): add IdentityProcessor

foreverlms
and
claude
committed
feat(dataflow): add DefaultBatchCollator

foreverlms
and
claude
committed
feat(dataflow): add SimpleBatcher

foreverlms
and
claude
committed
feat(dataflow): add IterableDistributor with round-robin sharding

foreverlms
and
claude
committed
feat(dataflow): add MapDistributor (per-epoch shuffle + slice sharding)

foreverlms
and
claude
committed
feat(dataflow): add CosmosDataLoader orchestrator + batch_size sugar

foreverlms
and
claude
committed
test(dataflow): multi-worker disjoint-coverage integration test

foreverlms
and
claude
committed
fix(dataflow): guard MapDistributor empty dataset/shard; surface persistent_workers coercion

foreverlms
and
claude
committed
feat(dataflow): add PoolPackingBatcher (port of PackingIterableDataset engine)

foreverlms
and
claude
committed
feat(vlm): extract VLMProcessor/VLMCollator from VLMDataPacker

foreverlms
and
claude
committed
test(dataflow): golden-batch equality VLM legacy vs new loader

foreverlms
and
claude
committed
feat(vlm): add mirror experiment pre_exp012_llava_ov_datapacker_v2 on dataflow loader

foreverlms
and
claude
committed
chore(vlm): mirror TOML + launch wrapper for llava_ov dataflow regression

foreverlms
and
claude
committed
feat(dataflow): MapDistributor env-var resume fast-forward + _dp_* meta

foreverlms
and
claude
committed
feat(dataflow): CosmosDataLoader resume meta-threading + batch stamping

foreverlms
and
claude
committed
test(dataflow): resume parity vs DataLoaderStateCallback

foreverlms
and
claude
committed
fix(dataflow): revert _cached_gen; fix resume test iterator; drop stale v2 dataloader_train

foreverlms
and
claude
committed
fix(dataflow): address Codex review — size_fn len(), map-resume contiguity guard

foreverlms
and
claude
committed
feat(dataflow): add VFMListCollator (port of custom_collate_fn)

foreverlms
and
claude
committed
feat(dataflow): add SequentialPackingBatcher (port of PackingDataLoader packing)

foreverlms
and
claude
committed
feat(dataflow): add RankPartitionedDistributor (port of RankPartitionedDataLoader)

foreverlms
and
claude
committed
feat(dataflow): add MixtureDistributor (ratio-weighted homogeneous join)

foreverlms
and
claude
committed
feat(dataflow): add JointCosmosDataLoader (port of JointDataPackerDataLoader)

foreverlms
and
claude
committed
feat(vfm): vision_sft_nano_v2 mirror + golden test + TOML + launch wrapper

foreverlms
and
claude
committed
feat(videophy2): extract VideoPhy2Processor from VideoPhy2DataPacker

foreverlms
and
claude
committed
test(dataflow): golden-batch equality videophy2 legacy vs new

foreverlms
and
claude
committed
feat(videophy2): mirror experiment videophy2_sft_nano_v2 on dataflow loader

foreverlms
and
claude
committed
chore(videophy2): mirror TOML + launch wrapper for dataflow regression

foreverlms
and
claude
committed
fix(dataflow): address Codex review — SequentialPackingBatcher count-only mode, v2 TOML mirror metadata, golden-test docs

foreverlms
and
claude
committed
fix(vfm): register vision_sft_nano_v2 experiment in base config

foreverlms
and
claude
committed
fix(dataflow): VFMListCollator reproduces legacy packed-batch nesting (list[list[Tensor]] for multi-item keys)

foreverlms
and
claude
committed
docs: add docs/dataflow.md (bring-your-own-dataset tutorial for CosmosDataLoader)

foreverlms
and
claude
committed
test(dataflow): golden_vfm uses monkeypatch.setenv for gloo bootstrap (env cleanliness)

foreverlms
and
claude
committed
style(docs): apply rumdl markdown formatting (pre-commit)

foreverlms
and
claude
committed

Commits on Jun 5, 2026

Commits on Jun 8, 2026

Commits on Jun 11, 2026

Commits on Jun 12, 2026