Sync origin/main into kunlun (through #105)#4
Open
chaonanD wants to merge 8 commits into
Open
Conversation
Automated release from i4. _source_commit: `c2c7152e59e4090a61822dc22a1691e7afb44702-dirty` _dest_commit (base): `463ac142b47c637098a7e86d0631252f82b65418`
…o recipe + numerical regression test (NVIDIA#86) ## Summary Adds **DROID action-policy post-training** on Cosmos3-Nano and a **numerical regression test** for the action-policy launches — the reference reproduction recipe for the DROID policy result. ## What's included - **Lazy DROID LeRobot dataset** — `cosmos_framework/data/generator/action/datasets/cosmos3_action_lerobot.py` (streaming `BaseActionLeRobotDataset`) + a rewritten `droid_lerobot_dataset.py` on top of it, plus `droid_lerobot_dataset_config.py`. Keys the versioned merged root; `use_success_only` resolves the `success/` split; eager `_register_sources()`. - **DROID Nano recipe** — `configs/base/experiment/action/posttrain_config/action_policy_droid_nano.py` + `examples/toml/sft_config/action_policy_droid_repro.toml`: res480, `joint_pos` 8D + `use_state`, JSON action prompt (`format_prompt_as_json=True`), CPU-side color jitter. The TOML pins the GB200 reference shape — HSDP 32×8 (256 ranks), global batch 8192, lr 2e-4, 10000 iters — and trains the generation + action heads from the public Cosmos3-Nano base. - **net_ema warm-start** — `checkpoint/dcp.py`: seed `net_ema` from `net` when `net_ema` is skipped on load (fresh action heads). - **Action-policy numerical regression test** — `tests/action_policy_regression_test.py`: the action-policy analogue of `tests/launch_regression_test.py`. Deterministic 10-iter re-run of the LIBERO + DROID launches (single-node, `--deterministic`, seed 42), asserting per-arch rank-0 loss goldens with a tolerance. LIBERO golden captured on the H200 CI arch; the DROID spec skips unless its (large, out-of-CI) dataset is supplied via `DROID_ROOT`. ## Reproduction The recipe reproduces the DROID action-policy result from the public Cosmos3-Nano base. The exact training code was validated by a from-scratch 64-GPU (GB200) run to 10k iterations, and the recipe TOML now encodes that run's configuration directly. ## Companion Cookbook PR: NVIDIA/cosmos#261 — the runnable DROID/LIBERO finetune cookbook that drives this recipe. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: lfengad <liangf@nvidia.com>
…SS (NVIDIA#92) The VLM base config defaults upload_reproducible_setup=True, so a run launched through the structured-TOML flow attempts the reproducible-setup S3 upload (and wandb save_s3, which interpolate from ${upload_reproducible_setup}) by default. OSS users mostly have no S3 access and no way to turn it off from the TOML. Add a [job].upload_reproducible_setup knob (default False) that maps to the top-level config.upload_reproducible_setup: - JobConfig gains the field (default False). - PATH_REMAPS (vfm + vlm) hoist ("job","upload_reproducible_setup") to the top-level ("upload_reproducible_setup",). - load_experiment_from_toml keeps the validated model and always injects the resolved value into raw before building overrides, so the override is emitted as false even when the TOML omits it (build_hydra_overrides walks the raw dict, so an omitted field would otherwise emit nothing and the base True would win). Users opt back into S3 upload with `upload_reproducible_setup = true`. Docs (sft_config.md) + all example [job] blocks updated; adds tests/toml_config_test.py. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…sers-cosmos3 shim (NVIDIA#98) ## What Replace the `packages/diffusers-cosmos3` shim with official diffusers (≥ 0.39.0) in the checkpoint exporter, and delete the package. ## Why - diffusers 0.39.0 ships Cosmos3 natively (`Cosmos3OmniPipeline`, `Cosmos3OmniTransformer`, `Cosmos3AVAEAudioTokenizer`); the shim was a pre-upstream stand-in. - The shim's reimplemented denoising forward has a small per-step numeric drift that compounds into visibly soft t2i images — nothing should run inference through it. - The exporter's key handling was stale against the current flat key layout (`model.`-nested `q_proj`/`vae2llm` vs flat `to_q`/`proj_in`) and could not strict-load. ## Changes - Rewrite `scripts/_convert_model_to_diffusers.py` against the official classes; keep the repo-specific extensions (action-projection export, `vision_encoder/` sidecar for the transformers/vLLM consumers). - Save `sound_tokenizer/` as a real `Cosmos3AVAEAudioTokenizer` pipeline component instead of hand-writing safetensors + patching `model_index.json`. - OSS-path fixes surfaced by e2e runs: consume the HF diffusers-layout `sound_tokenizer/` directly, load vision weights from `vision_encoder/`-subfolder sources, resolve the text config via `get_text_config()` for nested `Qwen3VLConfig`. - Deps: diffusers ≥ 0.39.0 (lock 0.37.0 → 0.39.0, safetensors 0.7.0 → 0.8.0), remove the shim from extras / uv sources, drop the now-fixed diffusers audit ignores. ## Relation to upstream `scripts/convert_cosmos3_to_diffusers.py` Upstream diffusers ships a converter for the same task, and this rewrite adopts its verified key-remap table. It cannot be used directly here: - its imports target the internal i4 module namespace (`projects.cosmos3.vfm.*`), so it does not run against this repo; - it silently drops action-projection weights (it never reads the action config, and strict-load still passes because neither side has the keys); - it does not produce the dual-purpose repo this exporter ships (no `vision_encoder/` sidecar, no top-level vLLM config / unified safetensors index). ## Verification - Key remap round-trips exactly against the real diffusers 0.39.0 `Cosmos3OmniTransformer` state-dict key set (action + sound enabled), and inverts cleanly through the native inference loader's mapping. - E2E, public checkpoint: Cosmos3-Nano → export → official `Cosmos3OmniPipeline` t2i sharp; native inference re-read of the export sharp. - E2E, real training DCP: an i4-trainer 8B DPO checkpoint (flat `net.*`/`net_ema.*` keys, t2i-only) converts and generates sharp t2i through the official pipeline (driven via the inner exporter; the CLI wrapper asserts `action_gen`/`sound_gen` and targets full omni checkpoints). ## Compat notes - Previously exported repos (`model_index: Cosmos3OmniDiffusersPipeline`) still load via an explicit `Cosmos3OmniPipeline.from_pretrained(...)`. - Loading without `cosmos_guardrail` installed requires `enable_safety_checker=False` (matches upstream semantics). --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Add Cosmos3-Super VideoPhy-2 Reasoner SFT recipe. It fully fine-tunes`
Qwen3-VL-32B` using FSDP full sharding and initializes from the
Cosmos3-Super language model merged with the 32B visual tower.
- The new experiment adds a launcher and TOML recipe.
- Use a learning rate of `1e-6 `for both Nano and Super.
- Set `lr_multipliers={"model.visual": 1.0} `to override the reasoner
default of `0.1` and train the visual projector at the base learning
rate. The previous `mm_projector` and `merger` keys matched no
`Qwen3-VL` parameters because the projector is named
`model.visual.merger`, so the intended `20x` multiplier was never
applied.
- Document the Super recipe in `docs/training.md `and
`examples/README.md`.
…NVIDIA#104) Ports Cosmos Transfer (control-conditioned) SFT post-training to the OSS layout (from i4 MR !10217). ## Changes - `curator_to_sft_jsonl.py`: add `--control-type` (edge/blur/depth/seg) and `--control-path-root`. Emits `control_type` on every row and `control_path` for precomputed depth/seg; clips with no matching precomputed file are dropped with reason `missing_control_path`. Summary records the control config. - `transfer_sft_dataset.py` (new): `TransferSFTDataset(SFTDataset)` — computes edge/blur on-the-fly or loads precomputed depth/seg control videos, returns `video=[control, target]` with a shared-temporal-position `SequencePlan`; plus `get_transfer_sft_dataset()` loader. - Tests: 10 new cases covering edge/blur/depth control paths and the converter end-to-end. ## Verification - Live import of `transfer_sft_dataset` in the i4 container: OK (base class = `SFTDataset`). - Curator converter tests: **33 passed**. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Automated release from i4. _source_commit: `7392c4b59ab7c1f0a94e59fbc4a75a0f3684b66f-dirty` _dest_commit (base): `3d9c0878fd0dde76eac98161aed0493d85a036fd` Co-authored-by: lfengad <liangf@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
将
origin/main自上次同步（PR #2）以来新增的 8 个 commit 同步进kunlun分支。origin/main（tip: Release 2026-07-11 (i4 7392c4b5) NVIDIA/cosmos-framework#105 Release 2026-07-11）kunlunsync/main-into-kunlun-20260714同步方式
上次同步 PR #2 是 squash-merge，导致
git merge origin/main会把已合入的 14 个旧 commit 重新翻出并产生 ~20 处伪冲突。因此本次改用 cherry-pick 仅同步新增的 8 个 commit（463ac14..origin/main），干净应用、无冲突。本次纳入的 8 个 commit
Conflicts
Kunlun/P800/XPU 适配保留情况
git diff <sync> origin/main仅剩以下适配 delta（已核验）：backends.py、torch_sdpa.py、flash2/{init,checks,functions}.py、frontend.py、context_parallel_utils.py、checkpoint_db.py、distributed.py、device.py、device_monitor.py、qwen3guard.py、inference/args.py、conftest.py。Validation
python -m compileall cosmos_framework（python312_torch29_cuda）0 错误；无遗留冲突标记。Note
仓库使用 squash-merge，本 PR 合入后旧 commit 仍会在下次
merge时重现；建议后续同步继续用 cherry-pick 新增区间，或改用 merge-commit 合入以修正历史基线。