Sync origin/main into kunlun through 692c410#6
Merged
Conversation
Co-authored-by: Xiangyu Lu <xiangyl@nvidia.com>
NVIDIA#70) Upstream i4 refactor `4190136a09` renamed `projects/cosmos3/vfm/` to `projects/cosmos3/cosmos3/` and moved the `vlm/` subtree to `reasoner/` under configs/base, models, datasets, and utils. This release re-runs the release pipeline against that new source layout and cleans up CF-side knock-on damage. Highlights: * Release pipeline output: 459 files mapped from the new i4 layout. Files added under `cosmos_framework/**/reasoner/`; the old `cosmos_framework/**/vlm/` counterparts are removed (orphan cleanup). Also ships `callbacks/tokens_per_sec.py`, `model/tokenizer/utils/vlm_prompt_format.py`, a new `configs/base/defaults/experimental/`-excluded layout, and refreshed `data/vfm/augmentors/reasoner/*` etc. * CF-owned config move: `configs/base/vlm/{defaults,experiment}/*.py` moved to `configs/base/reasoner/{defaults,experiment}/` (5 files), with imports rewritten `cosmos_framework.configs.base.vlm.` → `cosmos_framework.configs.base.reasoner.` so `task="vlm"` still finds a live experiment tree. * `configs/toml_config/toml_config_helper.py`: retarget `task="vlm"` → `cosmos_framework/configs/base/reasoner/config.py`. * Inference config rewriters (`inference/common/config.py`, `inference/common/public_model_config.py`): add `vlm` → `reasoner` rewrite rules ahead of the general vfm rules so old checkpoint JSONs (which still ship `cosmos3._src.vfm.configs.base.defaults.vlm.*` targets) resolve to the new module paths at load time. * `configs/base/experiment/sft/models/{nano,super}_model_config.py` and the `inference/configs/model/Cosmos3-{Nano,Super}.yaml` + `examples/checkpoints/Cosmos3-Nano/model/config.json`: drop the removed `DiffusionExpertConfig` fields (`position_embedding_type`, `rope_{h,t,w}_extrapolation_ratio`) and retarget the shipped Qwen3-VL JSON path to the `reasoner/` layout. * `inference/common/public_model_config_test.py`: update stale `vlm` paths in the round-trip test fixture. * `model/vfm/mot/unified_mot_test.py`: removed (tested private helpers `_PACKAGE_ROOT` / `_resolve_packaged_config_path` that were refactored out of `unified_mot.py`). Testing (4 × NVIDIA GB200 node): * `test_launch_regression[vision_sft_nano]`: 10-iter losses match gb200 goldens exactly. * `test_launch_regression[llava_ov]`: trains 10 iters end to end; loss drifts vs. 2026-05-18 gb200 goldens (goldens need recapture). * `nano_reasoner_inference_smoke_test`: inference runs cleanly; argmax hard-gate passes; tight allclose (rtol=atol=1e-3) fails by bf16 noise vs. presumably-H100 goldens. * Unit suite (`--num-gpus=0 --levels=0`): 327 passed / 1 failed (the failure is the `convert_model_to_dcp` script test, which needs `HF_TOKEN` for the private `nvidia/Cosmos3-Experimental` snapshot). --------- Co-authored-by: lfengad <liangf@nvidia.com>
Dataset support added: 1. Fractal (FractalLeRobotDataset class) 2. RoboMind Frank dual arm (through existing RoboMINDFrankaDataset class) 3. RoboMind UR (RoboMINDURDataset class) Other changes: 1. Add corresponding stats files for the newly added datasets 2. Folder structure refactor (minor) --------- Co-authored-by: lfengad <liangf@nvidia.com>
…ness, and doc (NVIDIA#61) ## What Adds the **Cosmos3-Nano LIBERO-10 action-policy SFT** surface, mirroring the existing DROID counterpart (`action_policy_droid_nano` + toml + launcher + doc). ### Feature (net-new) - **Experiment configs** `action_policy_libero_nano` (libero_10-only) and `action_policy_libero_all_nano` (equal 4-suite mix) — gen + action heads from the public Cosmos3-Nano base. - **Dataset** `LIBEROLeRobotDataset` + `get_action_libero_sft_dataset` — frame_wise_relative rot6d, `quantile_rot`, concat_view (third-person + wrist), 20 fps. - `base_dataset` `tasks.parquet` fallback for community LIBERO layouts. - Resample-on-decode-failure guard so one undecodable packed-mp4 frame can't crash a multi-node run (matches i4 behavior). - **Closed-loop eval harness** with vectorized sim, batched `/predict_batch`, single-rank `no_dist` checkpoint load. - **Structured-prompt serving** in the policy server (`--format-prompt-as-json`), so eval matches the training prompt format; the recipe defaults to it. ### Recipe + doc — two presets (to match the Cosmos3 LIBERO-10 result) Both lr 5e-5, warmup 500, cycle 16000, global batch 2048 (HSDP 2x8): - **(A) libero_10-only** — `action_policy_libero_repro.toml` + `launch_sft_action_policy_libero.sh` (max_iter 2000). - **(B) libero-all (4-suite equal mix)** — `action_policy_libero_all_repro.toml` + `launch_sft_action_policy_libero_all.sh` (max_iter 5000; `LIBERO_ROOT` = LIBERO_LeRobot_v3 parent dir). - `docs/action_policy_libero_sft.md` documents both. ## Notes - Scoped to LIBERO only; broader action-dataloader/model changes are intentionally not included here. - Based on `main`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Xiangyu Lu <xiangyl@nvidia.com> Co-authored-by: lfengad <liangf@nvidia.com>
## What Adds `test_nano_inference_multi_control_transfer` to `tests/nano_inference_smoke_test.py`, giving the Cosmos3-Nano generator inference smoke test coverage for the **multi-control transfer** feature (two control hints blended by `multi_control_two_way_attention`). It mirrors the existing single-control transfer run (same `latency` preset, 4 ranks → cfgp=2/cp=2), but the generated spec sets **two** hints (`edge` + `blur`) with per-hint `weight` and **no** `control_path`, so both controls are computed on the fly from a single `vision_path` (the pinned public `robot_pouring.mp4` clip) and aggregated by the weighted N-pass multi-control attention. ## What it verifies - **Multi-control path executes**: 2 hints → `control_weights` set → the network routes to `multi_control_two_way_attention` (both controls computed on the fly). - **Process stays within expected invariants**: the framework's multi-control runtime asserts (weights length == #controls, packing consistency, batch_size==1) must hold or the run exits non-zero → the test fails. Plus the test's own asserts: exactly one output; both `edge` and `blur` active with a `weight` and no `control_path`; `vision_path` set; `control_guidance` and `guidance` > 1.0. - **Output is valid**: non-degenerate `vision.mp4` via the existing `_assert_video_has_content` (≥16 frames, finite pixels, pixel std > 3). Smoke-level (output validity + path executed under asserts), not numeric goldens — consistent with the rest of the module. ## Test Runs on the same 8-GPU gate as `test_nano_inference_omni`: ``` pytest -s tests/nano_inference_smoke_test.py --num-gpus=8 --levels=2 -o addopts= ``` Verified end to end on a 4-rank `latency` run (exit 0; log shows both `Computing edge control input on the fly` and `Computing blur control input on the fly`; all assertions pass). Collection confirmed both tests register (`MAX_GPUS=8`). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`lerobot` is a heavy package that pins a lot of python packages. We can lazily import it to support a lightweight env setup. Signed-off-by: Hong-Yu Chiu <hongyuc@nvidia.com> Co-authored-by: lfengad <liangf@nvidia.com>
NVIDIA#78) …paths only) Follow-up to the initial vlm→reasoner sync (PR NVIDIA#70) — this PR renames the top-level CF dest folders and rewrites every import that references them. Contents are strictly file relocations + path-string rewrites; no code-behavior edits. Dest-folder rename: * cosmos_framework/data/vfm/ → cosmos_framework/data/generator/ * cosmos_framework/model/vfm/ → cosmos_framework/model/generator/ * cosmos_framework/utils/vfm/ → cosmos_framework/utils/generator/ * cosmos_framework/data/vlm/ → cosmos_framework/data/reasoner/ * cosmos_framework/utils/vlm/ → cosmos_framework/utils/reasoner/ * cosmos_framework/configs/base/vlm/{defaults,experiment}/ → cosmos_framework/configs/base/reasoner/{defaults,experiment}/ Import-path rewrites — every reference under ``cosmos_framework.{model,data,utils}.{vfm,vlm}.*`` and ``cosmos_framework.configs.base.vlm.*`` is retargeted to the new module names, in: * Release-managed files produced by cosmos-framework-release. * CF-owned production code (inference/, callbacks/, configs/, model/, utils/, data/generator/*/, data/reasoner/*/, tools/, scripts/). * Runtime rewriters: cosmos_framework/inference/common/config.py::CONFIG_REPLACEMENTS_INVERSE and cosmos_framework/inference/common/public_model_config.py (``_canonicalize_module_path``, ``_replace_vfm_module_prefix``, ``_replace_vfm_file_prefix``, ``_module_exists`` probes, {package}-templated dest strings) — old checkpoint JSONs' canonical ``cosmos3._src.vfm.*`` / ``projects.cosmos3.vfm.*`` target strings now map to the new module locations at load time. Verified: * Fresh AST-walker sweep across ``cosmos_framework/`` → **0 dangling ``cosmos_framework.*`` imports**. * Every ``M`` (modified-content) file's diff is verified path-only (each +/- line contains a ``vfm|vlm|generator|reasoner`` substring or is whitespace). * Every ``R`` is a git-detected rename. * 4-GPU GB200 regression: ``test_launch_regression[vision_sft_nano]`` passes and matches gb200 goldens exactly. * Unit suite (venv Python 3.13): 327 passed / 1 failed (the failure is the ``convert_model_to_dcp`` script test, which needs ``HF_TOKEN`` for the private ``nvidia/Cosmos3-Experimental`` snapshot — not fixable in-repo).
…-alias remap (NVIDIA#79) ## Summary Fixes two issues in the Reasoner / VideoPhy-2 SFT "Step 2" checkpoint prep (`convert_model_to_vlm_safetensors`), so the exported VLM is fully sourced from Cosmos3-Nano, and adds a regression test. ### 1. Fix `vlm`→`reasoner` remap for public-alias file paths (`public_model_config.py`) The i4 `vlm`→`reasoner` rename was applied to the module-path remap helpers but **not** to their file-path siblings. The current released `nvidia/Cosmos3-Nano` snapshot stores paths as public URIs (`cosmos3://vfm/models/vlm/qwen3_vl/configs/...`), which route through the file-path helpers and resolved to a non-existent `cosmos_framework/model/vfm/vlm/qwen3_vl/...`, crashing checkpoint load with `FileNotFoundError`. This affects loading the released checkpoint generally (not just the converter). Added the missing `vlm`↔`reasoner` rules to both `_replace_vfm_file_prefix` (public→runtime) and `_public_string_from_runtime_file_prefix` (runtime→public), mirroring the module-path helpers; the public↔runtime round-trip is stable. ### 2. Convert the Cosmos3-Nano vision tower (`convert_model_to_vlm_safetensors.py`) The converter previously extracted only the language model and left the visual tower as the **stock Qwen3-VL** weights — because the generation model instantiates no `visual` submodule, so the 351 `vision_encoder/` tensors never appear in its state dict. Now the visual tower is loaded directly from the checkpoint's `vision_encoder/` shards (`_load_vision_state`) and overlaid as `model.visual.*`. Checkpoints without a vision tower (Text2Image / Image2Video) keep Qwen3-VL's tower unchanged. The merged export is now 100% Cosmos3-Nano-sourced (399 LM + 351 visual). ### 3. Regression test (`tests/launch_regression_test.py`) `test_convert_reasoner_converts_all_qwen_tensors` asserts: 1. merged tensor set == stock Qwen3-VL set (all included, none extra); 2. every `model.visual.*` tensor matches the Cosmos3-Nano `vision_encoder/` source bit-for-bit; 3. a non-trivial subset differs from stock Qwen3-VL (so it catches the vision-drop regression — not a no-op); 4. the language tower was overlaid too. The ~16 GB output is regenerated per run and removed via a finalizer; inputs stay in the shared HF cache. ## Test plan - ✅ New test passes on a 4-GPU node: `1 passed`, `351 visual tensors all sourced from Cosmos3-Nano; 309 differ from stock Qwen3-VL`, output cleaned up (basetemp 13K after finalizer). - ✅ End-to-end: VideoPhy-2 Reasoner SFT loads `750 keys` from the merged checkpoint and trains cleanly across 4 GPUs (loss ~0.66→~0.15 over 20 iters). - ✅ Existing `public_model_config_test.py` still passes; public↔runtime round-trip verified stable. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Fix RoboMind Dataset - RoboMINDFrankaDataset supports loading from both `robomind-franka` and `robomind-franka-dual`, but it only load `norm_stats` for `robomind-franka-dual` - With this fix it will load `norm_stats` for `robomind-franka-dual` or `robomind-franka depends on the embodiment_type.
…NVIDIA#82) ## Summary Multi-control transfer inference silently ignored per-control weights. ## Changes - **`omni_mot_model.py`**: propagate `control_weights` into `gen_data_for_packing` - **`attention.py`**: make `multi_control_two_way_attention` `torch.compile`-safe. Its per-segment lengths come from data-dependent unpadding (unbacked symints), which tripped several Dynamo guards under `fullgraph=True`: - `torch._check(k.shape[0] == v.shape[0])` — frontend K/V-length guard - `torch._check(n_q > 0)` / `torch._check(n_kv > 0)` — NATTEN `max_seqlen == 0` / `< 1` varlen guards - `torch._check(n_full == noisy_e)` — makes per-segment `[cs:ce]` slices concrete-length, fixing the in-place write shape guard - pass `cumulative_seqlen_{Q,KV}` + `max_seqlen_{Q,KV}` instead of `seqlens_{Q,KV}` to avoid the disallowed `generate_varlen_parameters` device-host sync inside the compiled region ## Test plan - [x] PAI-Bench-C multi_control eval runs end-to-end under compiled attention (previously crashed with data-dependent Dynamo guard errors). - [x] Diagnostics confirm `control_weights` reach the network and the weighted-sum path executes. - [x] A/B check: extreme weights (e.g. edge=1 vs seg=1) now produce different outputs (previously byte-identical). - [x] Single-control transfer paths unaffected (they use the static-shape `two_way_attention`). Co-authored-by: Maosheng Liao <maoshengl@nvidia.com>
Standard cosmos-framework-release pipeline run against current main (generator/reasoner dest layout). Brings the latest i4 changes into CF. Highlights: * New leaf helper shipped: cosmos_framework/utils/generator/torchcodec_video.py (mapped via a new [[overrides]] entry — it lives at the top-level projects/cosmos3/utils/, outside the bulk cosmos3/cosmos3/utils mapping, and is imported by the pkl_to_media / reasoner.bytes_to_media augmentors). * New augmentors: data/generator/augmentors/multi_reference_transform.py, torchcodec_callers_test.py. * 13 existing shipped files refreshed to match i4 head. _source_commit (i4): 546faba4 _dest_commit (cf base): 6efeea3 Co-authored-by: lfengad <liangf@nvidia.com>
## Problem On **x86_64**, the `cu130` dependency group pinned the stock natten wheel `natten==0.21.6.dev6+cu130.torch210`, which does **not** contain `sm_103a` kernels. On B300 / GB300 (compute capability 10.3) this fails at runtime with `no kernel image is available for execution on the device`. ## Change - Point x86_64 cu130 natten at `==0.21.6.dev6+cu130.torch210.gb300`, mirroring the existing aarch64 entry. That build appends `10.3 -> sm_103a` to `NATTEN_CUDA_ARCH`. - Regenerated `uv.lock` (now resolves the x86_64 gb300 wheel, sha256 `04ace5d5…`). The wheel is published in the cosmos index via nvidia-cosmos/cosmos-dependencies#60. ## Verification - `uv lock --check` passes (lock consistent with pyproject). - End-to-end: `Cosmos3-Nano` t2i inference on a B300 (sm_103) completes successfully — 960×960 image generated, no "no kernel image" error. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: lfengad <liangf@nvidia.com>
Co-authored-by: Xiangyu Lu <xiangyl@nvidia.com>
Automated release from i4. _source_commit: `c2c7152e59e4090a61822dc22a1691e7afb44702-dirty` _dest_commit (base): `463ac142b47c637098a7e86d0631252f82b65418`
…o recipe + numerical regression test (NVIDIA#86) ## Summary Adds **DROID action-policy post-training** on Cosmos3-Nano and a **numerical regression test** for the action-policy launches — the reference reproduction recipe for the DROID policy result. ## What's included - **Lazy DROID LeRobot dataset** — `cosmos_framework/data/generator/action/datasets/cosmos3_action_lerobot.py` (streaming `BaseActionLeRobotDataset`) + a rewritten `droid_lerobot_dataset.py` on top of it, plus `droid_lerobot_dataset_config.py`. Keys the versioned merged root; `use_success_only` resolves the `success/` split; eager `_register_sources()`. - **DROID Nano recipe** — `configs/base/experiment/action/posttrain_config/action_policy_droid_nano.py` + `examples/toml/sft_config/action_policy_droid_repro.toml`: res480, `joint_pos` 8D + `use_state`, JSON action prompt (`format_prompt_as_json=True`), CPU-side color jitter. The TOML pins the GB200 reference shape — HSDP 32×8 (256 ranks), global batch 8192, lr 2e-4, 10000 iters — and trains the generation + action heads from the public Cosmos3-Nano base. - **net_ema warm-start** — `checkpoint/dcp.py`: seed `net_ema` from `net` when `net_ema` is skipped on load (fresh action heads). - **Action-policy numerical regression test** — `tests/action_policy_regression_test.py`: the action-policy analogue of `tests/launch_regression_test.py`. Deterministic 10-iter re-run of the LIBERO + DROID launches (single-node, `--deterministic`, seed 42), asserting per-arch rank-0 loss goldens with a tolerance. LIBERO golden captured on the H200 CI arch; the DROID spec skips unless its (large, out-of-CI) dataset is supplied via `DROID_ROOT`. ## Reproduction The recipe reproduces the DROID action-policy result from the public Cosmos3-Nano base. The exact training code was validated by a from-scratch 64-GPU (GB200) run to 10k iterations, and the recipe TOML now encodes that run's configuration directly. ## Companion Cookbook PR: NVIDIA/cosmos#261 — the runnable DROID/LIBERO finetune cookbook that drives this recipe. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: lfengad <liangf@nvidia.com>
…SS (NVIDIA#92) The VLM base config defaults upload_reproducible_setup=True, so a run launched through the structured-TOML flow attempts the reproducible-setup S3 upload (and wandb save_s3, which interpolate from ${upload_reproducible_setup}) by default. OSS users mostly have no S3 access and no way to turn it off from the TOML. Add a [job].upload_reproducible_setup knob (default False) that maps to the top-level config.upload_reproducible_setup: - JobConfig gains the field (default False). - PATH_REMAPS (vfm + vlm) hoist ("job","upload_reproducible_setup") to the top-level ("upload_reproducible_setup",). - load_experiment_from_toml keeps the validated model and always injects the resolved value into raw before building overrides, so the override is emitted as false even when the TOML omits it (build_hydra_overrides walks the raw dict, so an omitted field would otherwise emit nothing and the base True would win). Users opt back into S3 upload with `upload_reproducible_setup = true`. Docs (sft_config.md) + all example [job] blocks updated; adds tests/toml_config_test.py. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…sers-cosmos3 shim (NVIDIA#98) ## What Replace the `packages/diffusers-cosmos3` shim with official diffusers (≥ 0.39.0) in the checkpoint exporter, and delete the package. ## Why - diffusers 0.39.0 ships Cosmos3 natively (`Cosmos3OmniPipeline`, `Cosmos3OmniTransformer`, `Cosmos3AVAEAudioTokenizer`); the shim was a pre-upstream stand-in. - The shim's reimplemented denoising forward has a small per-step numeric drift that compounds into visibly soft t2i images — nothing should run inference through it. - The exporter's key handling was stale against the current flat key layout (`model.`-nested `q_proj`/`vae2llm` vs flat `to_q`/`proj_in`) and could not strict-load. ## Changes - Rewrite `scripts/_convert_model_to_diffusers.py` against the official classes; keep the repo-specific extensions (action-projection export, `vision_encoder/` sidecar for the transformers/vLLM consumers). - Save `sound_tokenizer/` as a real `Cosmos3AVAEAudioTokenizer` pipeline component instead of hand-writing safetensors + patching `model_index.json`. - OSS-path fixes surfaced by e2e runs: consume the HF diffusers-layout `sound_tokenizer/` directly, load vision weights from `vision_encoder/`-subfolder sources, resolve the text config via `get_text_config()` for nested `Qwen3VLConfig`. - Deps: diffusers ≥ 0.39.0 (lock 0.37.0 → 0.39.0, safetensors 0.7.0 → 0.8.0), remove the shim from extras / uv sources, drop the now-fixed diffusers audit ignores. ## Relation to upstream `scripts/convert_cosmos3_to_diffusers.py` Upstream diffusers ships a converter for the same task, and this rewrite adopts its verified key-remap table. It cannot be used directly here: - its imports target the internal i4 module namespace (`projects.cosmos3.vfm.*`), so it does not run against this repo; - it silently drops action-projection weights (it never reads the action config, and strict-load still passes because neither side has the keys); - it does not produce the dual-purpose repo this exporter ships (no `vision_encoder/` sidecar, no top-level vLLM config / unified safetensors index). ## Verification - Key remap round-trips exactly against the real diffusers 0.39.0 `Cosmos3OmniTransformer` state-dict key set (action + sound enabled), and inverts cleanly through the native inference loader's mapping. - E2E, public checkpoint: Cosmos3-Nano → export → official `Cosmos3OmniPipeline` t2i sharp; native inference re-read of the export sharp. - E2E, real training DCP: an i4-trainer 8B DPO checkpoint (flat `net.*`/`net_ema.*` keys, t2i-only) converts and generates sharp t2i through the official pipeline (driven via the inner exporter; the CLI wrapper asserts `action_gen`/`sound_gen` and targets full omni checkpoints). ## Compat notes - Previously exported repos (`model_index: Cosmos3OmniDiffusersPipeline`) still load via an explicit `Cosmos3OmniPipeline.from_pretrained(...)`. - Loading without `cosmos_guardrail` installed requires `enable_safety_checker=False` (matches upstream semantics). --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Add Cosmos3-Super VideoPhy-2 Reasoner SFT recipe. It fully fine-tunes`
Qwen3-VL-32B` using FSDP full sharding and initializes from the
Cosmos3-Super language model merged with the 32B visual tower.
- The new experiment adds a launcher and TOML recipe.
- Use a learning rate of `1e-6 `for both Nano and Super.
- Set `lr_multipliers={"model.visual": 1.0} `to override the reasoner
default of `0.1` and train the visual projector at the base learning
rate. The previous `mm_projector` and `merger` keys matched no
`Qwen3-VL` parameters because the projector is named
`model.visual.merger`, so the intended `20x` multiplier was never
applied.
- Document the Super recipe in `docs/training.md `and
`examples/README.md`.
…NVIDIA#104) Ports Cosmos Transfer (control-conditioned) SFT post-training to the OSS layout (from i4 MR !10217). ## Changes - `curator_to_sft_jsonl.py`: add `--control-type` (edge/blur/depth/seg) and `--control-path-root`. Emits `control_type` on every row and `control_path` for precomputed depth/seg; clips with no matching precomputed file are dropped with reason `missing_control_path`. Summary records the control config. - `transfer_sft_dataset.py` (new): `TransferSFTDataset(SFTDataset)` — computes edge/blur on-the-fly or loads precomputed depth/seg control videos, returns `video=[control, target]` with a shared-temporal-position `SequencePlan`; plus `get_transfer_sft_dataset()` loader. - Tests: 10 new cases covering edge/blur/depth control paths and the converter end-to-end. ## Verification - Live import of `transfer_sft_dataset` in the i4 container: OK (base class = `SFTDataset`). - Curator converter tests: **33 passed**. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Automated release from i4. _source_commit: `7392c4b59ab7c1f0a94e59fbc4a75a0f3684b66f-dirty` _dest_commit (base): `3d9c0878fd0dde76eac98161aed0493d85a036fd` Co-authored-by: lfengad <liangf@nvidia.com>
Automated release from i4. _source_commit: `c0d7b0044b6eb0239bfcd13a31fbb05dc789cb22-dirty` _dest_commit (base): `0fa3ba479a7c1c5be28f81fe084b9ea544d697c9`
## Summary Ports the inference-CLI plumbing from imaginaire4 **MR 10201** (`Inference optimizations [4/n]: MXFP8 / NVFP4 PTQ`) that was missing on this side. The quantization **backend** was already synced (via the 2026-07-10 release): `QuantizationConfig` (`configs/base/defaults/quantization.py`), `apply_quantization_inplace` (`utils/generator/quantization.py`), and the `model_loader.py` hook. But the user-facing `OmniInference` entrypoint was never wired, so `--quantization-method` had no effect. This PR connects them. The missing pieces mapped exactly to MR 10201's `packages/cosmos3/cosmos3/*` files (→ `cosmos_framework/inference/*`); the `projects/cosmos3/*` files were already in. ## Changes - **`inference/common/args.py`** — add `QuantizationMethod` / `QuantizationArgs` / `QuantizationOverrides`; mix them into `SetupArgs` / `SetupOverrides`. Fields flow through `build_setup`'s `model_dump → model_validate` (same as guardrails), so no extra build call is needed. - **`inference/model.py`** — add `Cosmos3OmniConfig.quantization` property + setter and thread `quantization_config` through `from_pretrained_dcp`. Uses the **local** direct-construction style `unstructure_config(QuantizationConfig(**value))` rather than upstream's `LazyCall` wrapper, matching the sibling `parallelism`/`compile` setters in this repo. - **`inference/inference.py`** — add `_get_quantization_config` and pass it into **both** `_create` branches (`load_model_from_checkpoint` and `from_pretrained_dcp`). ## Effect - **Standard experiment DCP path** (`load_model_from_checkpoint`): quantization applied end-to-end via `apply_quantization_inplace`. ✅ - **HF `from_pretrained_dcp` path**: stores `config.quantization` without an apply hook — **matches upstream MR behavior** (not a regression introduced here). - Only supported on Blackwell + replicated inference (no FSDP sharding); guarded by the existing `model_loader.py` DP-shard check. ## Not included - `websocket_policy_server/serve_policy.py` + README from the MR — that action-eval subtree is not present in this repo. - The MR's cosmetic blank-line change in `packages/cosmos3/cosmos3/args.py`. ## Verification - `py_compile` passes on all three files. - Load-bearing pydantic logic verified in-container (isolated replica): defaults, mutable-list-default isolation, `build_quantization`, `Literal` rejection of invalid methods, MRO/inheritance, and `SetupOverrides → SetupArgs` field flow — all assertions pass. - Full module import not exercised: cosmos-framework requires Python ≥3.12 while the available i4 container env is 3.10; needs the cosmos-framework 3.13 venv to import the real module. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Summary - support Cosmos3 checkpoints with optional action and sound generation - support current and legacy AVAE/tokenizer layouts and framework-native VLM config layouts - make language-model remapping reject collisions and exclude visual weights from the text transformer - support both legacy and current action projection module names - add `--skip-vision-encoder` and make `--config-only` work with an empty output directory - generate a top-level safetensors index that matches the files actually exported - keep the converter implementation aligned with the corresponding Cosmos3 package implementation, apart from repository-specific imports and licensing - add a GPU CI regression that converts the staged Nano DCP, validates the complete Diffusers output against the public Nano reference indexes, and always removes the generated output ## Validation - converted a Cosmos3-Nano DCP checkpoint to a complete Diffusers checkpoint - verified all 10 exported safetensors files byte-for-byte against `nvidia/Cosmos3-Nano` - verified the unified index contains 1,165 tensors with no dangling file references - loaded the output with `Cosmos3OmniPipeline` and ran one-step video-plus-sound inference with finite outputs - verified pytest collection selects the new Nano Diffusers conversion test in the generator GPU workflow - Ruff check, `py_compile`, YAML parse, and `git diff --check` pass
Automated release from i4. _source_commit: `b66cfd8155158744e16d598b7515fbd0b5081a24-dirty` _dest_commit (base): `755bff48dd66b467995366796e747ffc649dd83c` Co-authored-by: lfengad <liangf@nvidia.com>
# Conflicts: # cosmos_framework/callbacks/every_n_draw_sample.py # cosmos_framework/configs/base/defaults/reasoner.py # cosmos_framework/configs/base/experiment/action/posttrain_config/action_policy_droid_nano.py # cosmos_framework/configs/base/reasoner/config.py # cosmos_framework/data/generator/action/datasets/action_sft_dataset.py # cosmos_framework/data/generator/action/datasets/base_dataset.py # cosmos_framework/data/generator/action/datasets/droid_lerobot_dataset.py # cosmos_framework/data/generator/augmentors/interleaved_image_transform.py # cosmos_framework/data/generator/augmentors/text_transforms_for_image.py # cosmos_framework/data/generator/sequence_packing/__init__.py # cosmos_framework/data/generator/sequence_packing/packers.py # cosmos_framework/data/generator/sequence_packing/temporal_causal.py # cosmos_framework/inference/inference.py # cosmos_framework/model/generator/diffusion/samplers/fixed_step.py # cosmos_framework/model/generator/diffusion/samplers/fixed_step_test.py # cosmos_framework/model/generator/mot/context_parallel_test.py # cosmos_framework/model/generator/omni_mot_model.py # cosmos_framework/model/generator/tokenizers/uniae/noncausal_4x16x16.py # cosmos_framework/model/tokenizer/evaluation/reconstruction_metrics.py # cosmos_framework/model/tokenizer/models/text_decoder.py # cosmos_framework/scripts/_convert_model_to_diffusers.py # cosmos_framework/utils/generator/model_loader.py # cosmos_framework/utils/generator/torchcodec_video.py # docs/code_structure.md # docs/sft_config.md # docs/training.md # tests/launch_regression_test.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
13 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Redo the sync from
KunlunxinAD/cosmos-framework:mainafter reverting PR #4.This PR merges the real
mainhistory directly intokunlun; it does not cherry-pick, squash, or recreate upstream commits.Sync revisions
KunlunxinAD/cosmos-framework:main692c410583a2e1271cdc355ffe2ffe574c2ae625kunlunatfc9eb349ac93c79309ed2c153e56c7307786e18facfe95c45d6b516246c1dac3a279db23099457f1Conflict resolution
Conflicting upstream files use the current
mainversions. Existing Kunlun/P800 adaptations are retained in the final 14-file diff. An automatically duplicatedCHECKPOINT_S3_EAST2block was removed during conflict review.Validation
git merge-base --is-ancestor origin/main HEAD: passedgit diff --check origin/main...HEAD: passedpython -m compileall -q conftest.py cosmos_framework: passedpre-commitworkflow: passedpytest -q: not executed past collection because the local environment is missingomegaconfuvis unavailable in the local environment