Release 2026-07-20: Cosmos3-Edge + Distillation support#119
Merged
Conversation
Register the Cosmos3-Edge checkpoint for inference: - inference/configs/model/Cosmos3-Edge.yaml: OmniMoTModel config for the 2B Edge model (reasoner nvidia/Cosmos3-Edge-Reasoner, processor/tokenizer nvidia/Cosmos3-Edge); vlm backbone disabled (weights bundled in the main checkpoint). - inference/args.py: add "Cosmos3-Edge" to _CHECKPOINTS, HF repo nvidia/Cosmos3-Edge. - inference/model.py: support Edge's diffusers layout when loading — merge the transformer/ safetensors index into the root reasoner manifest so generation-pathway tensors (*_moe_gen, add_*) load, and skip vision_encoder/* weights when the model is built without a visual tower (t2i with include_visual disabled). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add Cosmos3-Edge to the inference guide alongside Cosmos3-Nano/Super: - Models table row (all modes except audio / enable_sound). - Single-GPU Quick Start subsection: 2B omni model, single-GPU t2i example plus a batched command covering all supported modes, excluding the audio-enabled t2vs/i2vs examples (no sound tokenizer). - TOC entry. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add Cosmos3-Edge inference support (HF-backed 2B omni)
Bring public OSS main (convert_model_to_diffusers #98) into the private mirror which carries the Cosmos3-Edge inference work.
) ## Summary Implements `Nemotron3DenseVLTextForCausalLM.generate_reasoner_text` (previously `NotImplementedError`), enabling both text-only and image-conditioned reasoner-tower generation for Cosmos3-Edge. ## Changes - **`unified_mot.py`** — implement `generate_reasoner_text` covering text-only and image-conditioned prefill; lazily load a SigLIP2 vision tower from the HF repo's `vision_encoder/` on the first image prompt (`_ensure_vision_tower`, cached as `self.visual`, no DCP remap). Parameterize `_impl_generate_reasoner_text` with `prepare_multimodal_fn` so the shared decode loop serves both Qwen3-VL and Nemotron families. - **`vision_siglip2.py`** (new) — self-contained SigLIP2 vision encoder + projector stack. - **`reasoner_multimodal_utils.py`** (new) — Nemotron variant of `prepare_multimodal_reasoner_inputs`: SigLIP2 encode → `masked_scatter` → multimodal rope, no deepstack. ## Not covered Video-conditioned reasoner generation remains unimplemented (raises `NotImplementedError`). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Syncs the latest public OSS `NVIDIA/cosmos-framework` main into the private mirror. ## What's included Two commits from public main not yet in the private mirror: - `3d9c087` Add Cosmos3-Super VideoPhy-2 Reasoner SFT recipe (#101) - `6d84808` Support reasoner video input. (#25) ## Merge Clean auto-merge into current `mirror/main` (`87ce58a`), no conflicts. --------- Co-authored-by: Maosheng Liao <maoshengl@nvidia.com> Co-authored-by: Xuanmeng Zhang <simonz@nvidia.com>
Add the `Cosmos3-Edge `counterpart to the existing `Cosmos3-Nano` and `Cosmos3-Super` SFT examples, covering both post-training tasks. Both recipes target `nvidia/Cosmos3-Edge` omni release: its MoT generator tower for vision, and its SigLIP2-based reasoner tower for VideoPhy-2. Vision SFT (vision_sft_edge) T2V/I2V/V2V generator (VFM) SFT on the `Cosmos3-Edge` backbone, on the same `BridgeData2` dataset as the Nano recipe. - Add the experiment, TOML, and launcher (launch_sft_vision_edge.sh), and `optimizer/scheduler/trainer/checkpoint blocks`. The only differences are the model config (EDGE_MODEL_CONFIG, dense Nemotron backbone, no audio/sound tokenizer) and the recipe name. EMA enabled, bf16, Wan2.2 VAE, and full-shard FSDP auto-sized for 4- or 8-GPU runs. - Step 2 uses the standard `convert_model_to_dcp` DCP conversion (`BASE_CHECKPOINT_NAME=Cosmos3-Edge`). Two supporting fixes make the Edge omni checkpoint convertible: register the internal Nemotron targets (`Nemotron3DenseVLMoTConfig`, `Nemotron3DenseVLTextForCausalLM`) in the public model-config alias table, and make `convert_model_to_dcp`'s `checkpoint.json` copy conditional (Edge ships a `safetensors/Diffusers` layout with no `checkpoint.json`). Reasoner Alignment SFT (videophy2_sft_edge) - Add the experiment, TOML, and launcher. Freeze the `SigLIP2 `vision tower, train the projector and language model at 1e-6, use SDPA, and auto-size full-shard FSDP for 4- or 8-GPU runs. - Add a converter that rewrites the Edge reasoner's Diffusers-layout checkpoint as canonical HF safetensors. Require the converted snapshot through V`LM_SAFETENSORS_PATH` because model_name cannot serve as a weight fallback for Edge. - Register the `cosmos3_edge_reasoner `policy, route `Cosmos3-Edge` through the `Nemotron3DenseVL` processor, and document checkpoint preparation and launch steps. - Validated on 4x GB200: the converted snapshot loads, training loss decreases and remains stable at 1e-6, and both DCP checkpoints and HF exports save successfully.
# Conflicts: # docs/training.md # examples/README.md
Syncs the latest public OSS `NVIDIA/cosmos-framework` main into the private mirror. ## What's included New commit from public main not yet in the private mirror: - `1e139fb` feat(transfer-sft): control-conditioned SFT dataset + curator support (#104) ## Merge Clean auto-merge into current `mirror/main` (`fe0d62a`), no conflicts. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Maosheng Liao <maoshengl@nvidia.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Xuanmeng Zhang <simonz@nvidia.com>
Merge origin/main (0fa3ba4) into private mirror main
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
tune: raise vision full fine-tune SFT lr 2e-5 -> 1e-4
The lr commit (6b115bf) raised the shipped vision_sft_nano recipe to 1e-4, which shifts the 10-iter loss trajectory from iter 1 on and breaks the generator-training-regression job (loss no longer matches the goldens, which were captured at 2e-5). Pin the regression spec's optimizer.lr back to 2.0e-5 so the numerical-regression guard is decoupled from recipe-lr tuning: the committed h100 AND gb200 goldens stay valid with no GPU recapture, while the shipped recipe stays at 1e-4. Trailing --sft-toml overrides win over the TOML value (sft_config.load_experiment_from_toml). Also annotate both nano goldens. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Fix CI / CD test: pin vision_sft_nano regression to lr=2e-5 (goldens capture value)
…(Edge)
The Edge reasoner (Nemotron 3 Dense VL) supported text- and image-conditioned
generation but raised NotImplementedError on a video vision_path
(inputs/reasoner/reasoner_video.json). Everything upstream was already wired for
video — the inference layer decodes the clip, the model builds the video chat
block, Nemotron3DenseVLProcessor emits pixel_values_videos/video_grid_thw,
config.video_token_id is plumbed, and the shared _impl_generate_reasoner_text +
get_rope_index/get_placeholder_mask are medium-agnostic. Only the two Nemotron
NotImplementedError guards were missing an implementation.
Mirror the Qwen3-VL video path into the Nemotron reasoner:
- reasoner_multimodal_utils.py: add the video branch. SigLIP2 has no temporal
attention, so a video is just its frames stacked along the video_grid_thw
temporal axis; reuse the same get_image_features encoder, scatter into the
video placeholder mask, and index rope with video_grid_thw. No deepstack.
- unified_mot.py: drop the guard, extend the lazy vision-tower trigger to video,
and forward pixel_values_videos/video_grid_thw to _impl_generate_reasoner_text.
Verified end-to-end on Cosmos3-Edge (single GB200): the video path decodes
robot_pouring.mp4, encodes through SigLIP2, and generates a correct description
("The robot arm is pouring a red liquid from a kettle into a glass containing a
spoon."); the restructured image path still produces a correct caption; the
reasoner inference unit tests pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…nditioning feat(reasoner): video-conditioned generation for Nemotron 3 Dense VL (Edge)
Automated release from i4. _source_commit: `c0d7b0044b6eb0239bfcd13a31fbb05dc789cb22-dirty` _dest_commit (base): `0fa3ba479a7c1c5be28f81fe084b9ea544d697c9`
## Summary Ports the inference-CLI plumbing from imaginaire4 **MR 10201** (`Inference optimizations [4/n]: MXFP8 / NVFP4 PTQ`) that was missing on this side. The quantization **backend** was already synced (via the 2026-07-10 release): `QuantizationConfig` (`configs/base/defaults/quantization.py`), `apply_quantization_inplace` (`utils/generator/quantization.py`), and the `model_loader.py` hook. But the user-facing `OmniInference` entrypoint was never wired, so `--quantization-method` had no effect. This PR connects them. The missing pieces mapped exactly to MR 10201's `packages/cosmos3/cosmos3/*` files (→ `cosmos_framework/inference/*`); the `projects/cosmos3/*` files were already in. ## Changes - **`inference/common/args.py`** — add `QuantizationMethod` / `QuantizationArgs` / `QuantizationOverrides`; mix them into `SetupArgs` / `SetupOverrides`. Fields flow through `build_setup`'s `model_dump → model_validate` (same as guardrails), so no extra build call is needed. - **`inference/model.py`** — add `Cosmos3OmniConfig.quantization` property + setter and thread `quantization_config` through `from_pretrained_dcp`. Uses the **local** direct-construction style `unstructure_config(QuantizationConfig(**value))` rather than upstream's `LazyCall` wrapper, matching the sibling `parallelism`/`compile` setters in this repo. - **`inference/inference.py`** — add `_get_quantization_config` and pass it into **both** `_create` branches (`load_model_from_checkpoint` and `from_pretrained_dcp`). ## Effect - **Standard experiment DCP path** (`load_model_from_checkpoint`): quantization applied end-to-end via `apply_quantization_inplace`. ✅ - **HF `from_pretrained_dcp` path**: stores `config.quantization` without an apply hook — **matches upstream MR behavior** (not a regression introduced here). - Only supported on Blackwell + replicated inference (no FSDP sharding); guarded by the existing `model_loader.py` DP-shard check. ## Not included - `websocket_policy_server/serve_policy.py` + README from the MR — that action-eval subtree is not present in this repo. - The MR's cosmetic blank-line change in `packages/cosmos3/cosmos3/args.py`. ## Verification - `py_compile` passes on all three files. - Load-bearing pydantic logic verified in-container (isolated replica): defaults, mutable-list-default isolation, `build_quantization`, `Literal` rejection of invalid methods, MRO/inheritance, and `SetupOverrides → SetupArgs` field flow — all assertions pass. - Full module import not exercised: cosmos-framework requires Python ≥3.12 while the available i4 container env is 3.10; needs the cosmos-framework 3.13 venv to import the real module. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Summary - support Cosmos3 checkpoints with optional action and sound generation - support current and legacy AVAE/tokenizer layouts and framework-native VLM config layouts - make language-model remapping reject collisions and exclude visual weights from the text transformer - support both legacy and current action projection module names - add `--skip-vision-encoder` and make `--config-only` work with an empty output directory - generate a top-level safetensors index that matches the files actually exported - keep the converter implementation aligned with the corresponding Cosmos3 package implementation, apart from repository-specific imports and licensing - add a GPU CI regression that converts the staged Nano DCP, validates the complete Diffusers output against the public Nano reference indexes, and always removes the generated output ## Validation - converted a Cosmos3-Nano DCP checkpoint to a complete Diffusers checkpoint - verified all 10 exported safetensors files byte-for-byte against `nvidia/Cosmos3-Nano` - verified the unified index contains 1,165 tensors with no dangling file references - loaded the output with `Cosmos3OmniPipeline` and ran one-step video-plus-sound inference with finite outputs - verified pytest collection selects the new Nano Diffusers conversion test in the generator GPU workflow - Ruff check, `py_compile`, YAML parse, and `git diff --check` pass
Automated release from i4. _source_commit: `b66cfd8155158744e16d598b7515fbd0b5081a24-dirty` _dest_commit (base): `755bff48dd66b467995366796e747ffc649dd83c` Co-authored-by: lfengad <liangf@nvidia.com>
Merge OSS main into private main (2026-07-15)
…nse VL reasoner Backport of review cleanups made downstream in imaginaire4 to the same files here (originally added by #3 / #10). No behavioral change. - vision_siglip2.py: remove the dead get_position_embedding_fast_interpolation method (never called; forward uses get_position_embedding) and its print("wrong fast interpolation") debug statement. - vision_siglip2.py: translate the vendored Chinese comments to English. - unified_mot.py: close the two vision-tower config file handles via 'with open(...)' context managers in _ensure_vision_tower. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…-cleanups cleanup(reasoner): dead code, i18n, and file handles in Nemotron 3 Dense VL reasoner
…o the vision recipe (#18) ## Problem `bash examples/launch_sft_vision_edge.sh` (stock defaults, 8xH100) hits `loss = NaN` within the first few iterations. The exact failing iteration is nondeterministic (reported at iter 6; reproduced at iters 1-6) because the poison values come from uninitialized memory. ## Root cause Cosmos3-Edge has **no action modality** — its source config declares only vision+sound, and its released checkpoint ships **no action weights** (`action2llm`/`llm2action`/`action_modality_embed`). But the Edge vision-SFT config ran with `action_gen=True`, so: 1. `convert_model_to_dcp` builds the model with a frozen action head. On the CPU conversion path (`omni_mot_model.py:243-248`, `COSMOS_DEVICE=cpu`), `net.to_empty()` allocates the action tensors but `init_weights()` is only called for CUDA — so the action weights are **uninitialized memory**, written as-is into the DCP. 2. During training, the graph-consistency dummy forward adds `0.0 * sum(preds_action)` to the loss (`omni_mot_model.py:1063-1090`). When the uninitialized action weights contain NaN/Inf, `0.0 * NaN = NaN` (IEEE-754) poisons the loss from the start. This also explains nondeterminism across machines (e.g. finite loss on GB200): whatever garbage happens to be in the allocated pages decides the outcome. ## Fix Set `action_gen=False` for the Edge **vision SFT recipe** — matching the data it actually trains. The action pathway is never built, the dummy forward skips the action term, and existing (garbage-carrying) DCPs load cleanly because extra keys are ignored — no checkpoint re-conversion needed. **Where the flag lives:** the override is scoped to the recipe, not the shared baseline. `EDGE_MODEL_CONFIG` keeps `action_gen=True` (mirroring `Cosmos3-Edge.yaml` and the nano baseline, with a warning comment about the missing action weights), and `vision_sft_edge.py` deepcopies it into `_EDGE_VISION_MODEL_CONFIG` with `action_gen=False`. Rationale: `action_gen` should describe what a recipe trains — vision SFT has no action tokens, while a future Edge action-SFT recipe can inherit the baseline with the action head enabled instead of undoing a vision-specific default. The experiment deepcopies `_EDGE_VISION_MODEL_CONFIG` again at the `config=` site so future reuse cannot alias mutations across experiments. Nano keeps `action_gen=True` everywhere for now: its checkpoint ships real action weights (trained by the DROID action-policy recipe), so `0.0 * finite = 0` and vision SFT is unaffected. Scoping nano's vision recipe to `action_gen=False` the same way is a follow-up candidate; note it would drop the pretrained action heads from vision-SFT'd checkpoints. ## Also in this PR `videophy2_sft_edge` now defaults to `flash_attention_2`: `nemotron_siglip2`'s `_update_causal_mask` returns the mask-free (None) path only for FA2, while sdpa gets a dense 4-D mask. ~14% faster on 8xH100 (11.1 vs 12.9 s/iter) with matching loss curves over a 50-iter same-seed A/B. sdpa remains documented as the portable fallback (e.g. aarch64/Blackwell without a flash-attn wheel) via `EXTRA_TAIL_OVERRIDES`. ## Validation (8xH100, from scratch: fresh DCP conversion + training) | Run | Before | After | |---|---|---| | `launch_sft_vision_edge.sh` smoke | NaN at iter 1-6 | finite (3.29 -> 2.85 over iters 1-6) | | `launch_sft_vision_edge.sh` full 500 iters | — | 0 NaN, final loss 1.40, ckpt `iter_000000500` | | `launch_sft_videophy2_edge.sh` full (VLM path) | — | 0 NaN, exit 0, ckpt `iter_000000050` | | `videophy2_sft_edge` FA2 vs sdpa 50-iter A/B | 12.9 s/iter | 11.1 s/iter, matching loss curves | | Nano / Super vision recipes | unaffected (checkpoints ship real action weights) | unchanged | The refactor commit (`action_gen` placement) composes to a byte-identical `vision_sft_edge` training config, so the validation above still holds. Also ruled out: bf16 overflow (still NaN in fp32), data/norm issues. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
## Summary - Regenerates `cosmos-framework` from the exact merged imaginaire4 commit `a54b7da336e5ef00188cf275cea7456e723d2888` using `packages/cosmos-framework-release/release.sh --pr`. - Releases Cosmos3 DMD2 distillation training support for T2I and I2V, including student-only export support and the sampled-media callback required by inference. - Registers and configures the released 4-step Hugging Face checkpoints: - `nvidia/Cosmos3-Super-Text2Image-4Step` - `nvidia/Cosmos3-Super-Image2Video-4Step` - Carries the six cosmos-framework-owned inference/config fixes validated on private PR #13. - Adds the Framework-owned fixed-step public-config alias and student-only exporter wiring validated after imaginaire4 !10418. ## Provenance - i4 source: `a54b7da336e5ef00188cf275cea7456e723d2888` - original private base: `38cb38e77e470fbd38446c17ccfb0109b17edbde` - rebased private base: `d549fbbcb3f0065dbeeb941f9ea99d983bdf9678` - source MR: [imaginaire4 !10164](https://gitlab-master.nvidia.com/cosmos-lab/imaginaire4/-/merge_requests/10164) - release-mapping follow-up: [imaginaire4 !10388](https://gitlab-master.nvidia.com/cosmos-lab/imaginaire4/-/merge_requests/10388) - public-config/export follow-up: [imaginaire4 !10418](https://gitlab-master.nvidia.com/cosmos-lab/imaginaire4/-/merge_requests/10418) - generated release commit before rebase: `f0d02cb88ef3b244782b235f651e2df6cb78605b` - pre-rebase export/inference-tested head: `4098377f786b2729a60dc235996a0938af06b6df` - current rebased and export/inference-tested private head: `febaeb2d7a83eb65d55a4b66dea5adf9aaa4d4b6` The sampled-media callback and its test are generated from i4 by the release mapping. The temporary callback-copy commit from private PR #13 is intentionally not present. The final three-file follow-up is committed directly in Cosmos Framework because `cosmos_framework/inference/common/public_model_config.py` and `cosmos_framework/scripts/export_model.py` are Framework-owned and are not destinations in `packages/cosmos-framework-release/mapping_config.toml`. Re-running `release.sh --pr` would not update those files. ## Private-base conflict resolution The branch was rebased onto private `main` at `d549fbb`. The only conflict was `cosmos_framework/model/generator/mot/unified_mot.py`. The resolution preserves both sides: - private `main`'s complete Nemotron multimodal reasoner path, including the helper import, injectable multimodal prefill function, image/video arguments, lazy SigLIP2 vision tower, and file-handle cleanup; - the release branch's generation-tower cosine-router and auxiliary-loss-free load-balancing configuration. After resolution, `unified_mot.py` differs from private `main` only by those intended router additions. The eight rebased commits map one-for-one to the original series. GitHub now reports the PR as mergeable. ## Validation - Release mapping: 496 files copied, 0 missing; drift gate had no blocking break. - Release scrub/import/compile: passed; no internal imports, sensitive infrastructure strings, active release directives, or key-module import failures. - Rebased-head local checks at `febaeb2`: - direct interface probe confirms the Nemotron image/video arguments, injectable multimodal prefill hook, and lazy vision-tower method remain present; - bytecode compilation and all Ruff rules except the target-main-inherited `I001` import-order finding passed; - 185 focused distillation, checkpoint, trainer, callback, distributed, exporter/public-config, and MoE tests passed; - both repository pre-commit configurations and `git diff --check` passed without changing the worktree. - Rebased-head private CI: all 8 checks passed against `febaeb2`, including [GPU workflow 29489566032](https://github.com/nvidia-cosmos/cosmos-framework-private/actions/runs/29489566032) and [standalone pre-commit workflow 29489561974](https://github.com/nvidia-cosmos/cosmos-framework-private/actions/runs/29489561974). - Pre-rebase private CI: all 8 checks passed against `4098377`, including failed-only attempt 2 of [workflow 29478278528](https://github.com/nvidia-cosmos/cosmos-framework-private/actions/runs/29478278528). - Attempt 1 passed both generator jobs and pre-commit. Four unrelated jobs failed while Hugging Face returned repeated read timeouts and HTTP 504 responses for Qwen Guard, LLaVA-OneVision data, the Nano conversion checkpoint, and the Bridge synthetic-caption dataset. - No code changed. Attempt 2 passed `reasoner-inference-smoke` in 2m30s, `unittest` in 8m26s, `training-smoke` in 9m39s, and `reasoner-training-regression` in 2m18s after service recovery. - Rebased-head student-only re-export: Slurm job `1643135` re-exported the T2I and I2V iteration-6 DCP checkpoints from clean head `febaeb2`. - Each export has 1,771 tensors in 27 safetensor shards, `source_weights=ema`, and `student_only=true`. - No tensor key or sanitized config contains `teacher` or `fake_score`. - Config, index, and per-shard sizes match the previously validated exports for the same DCPs. - Rebased-head exact released-model inference: job `1643135` completed `0:0` in `00:15:44` on one GB200 node with 4 GPUs. - T2I: `nvidia/Cosmos3-Super-Text2Image-4Step`, 768p, guidance 1.0, seed 1, successful JPEG. - I2V: `nvidia/Cosmos3-Super-Image2Video-4Step`, 480p, 121 frames, guidance 1.0, seed 1, successful MP4. - Both used fixed-step SDE schedule `[1.0, 0.9375, 0.8333333333333334, 0.625, 0.0]`. - T2I SHA-256: `e3d991f8726de166707cc2c15a9aea71ea1756ce5db60bab9f5d963e084f0a02`. - I2V SHA-256: `1463de9f2865c13b0bc6d7a5d0f26b6e70283805fc1085ced18fd3d2c54a7633`. - Both artifacts are bit-for-bit identical to the prior canonical private runs. ## Scope and safety - T2I and I2V only. - Smoke validation only; no production-quality reproduction claim. - No model or artifact was uploaded to Hugging Face. - This is the internal safety/credential-review PR. No public GitHub PR has been created. - Rebased-head CI, student-only export, and exact T2I/I2V inference revalidation are complete; private review and merge remain pending.
#21) The 2026-07-16 repo renewal (28a0b8e -> be935d6) deleted all remote-code .py files and switched metadata to native model_type=cosmos3_edge (transformers main only). This migrates both Edge recipes to framework-internal modeling on transformers 4.57.6: - new cosmos_framework/model/generator/reasoner/cosmos3_edge/: dense-only vendored port of the remote-code reasoner (no Mamba -> mamba-ssm prereq gone), registered via AutoConfig/AutoModelForImageTextToText, canonical 670-key state dict; vision-feature parity vs the old stack is bit-identical - safetensors_loader: direct index-following load from the snapshot layout (root manifest -> transformer/ + vision_encoder/ shards, 28-paired -> canonical remap) + sha256 guard on the vision shard (G2) - cosmos3_edge_processing: native port of the deleted processor, keeping the original pixel-patch layout (raster rows, (py,px,c)) -- transformers-main's native processor emits a different layout that corrupts vision features with these weights; goldens test pins the convention (G1) - convert_edge_reasoner_to_vlm_safetensors deleted; VLM_SAFETENSORS_PATH now optional (weights direct-load from nvidia/Cosmos3-Edge, ungated) - docs: converter step removed; reasoner tower documented as bit-identical to nvidia/Cosmos3-Edge-Reasoner@590c1c0 (old "differs" caveat disproven) Validated on 4xGB200 (jobs 1651362/1651363, snapshot b79964e): videophy2 500-iter loss curve statistically identical to the pre-migration baseline (max |dMA25| 0.0188, mean 0.0056; census 233/2,022,966,784 exact match); vision 500-iter 499/499 finite, 5/5 saves, MA10 2.86->1.69 on the retrained upstream generation weights. 22 new CPU tests.
…ce) (#22) ## Summary The post-merge GPU Tests run on `main` (run [29571676065](https://github.com/nvidia-cosmos/cosmos-framework-private/actions/runs/29571676065)) failed in `reasoner-training-regression`: `uvx hf@latest download` died with ``` No solution found when resolving tool dependencies: hf==1.24.0 depends on huggingface-hub==1.24.0 ... unsatisfiable ``` **Diagnosis:** `uvx pkg@latest` refreshes only `pkg`'s own index metadata, not its dependencies'. Upstream published `hf` 1.24.0 (pinning the equally new `huggingface-hub` 1.24.0) mid-run; the runner saw the fresh `hf` metadata against a stale cached `huggingface-hub` index. Both versions exist on PyPI — the pair is satisfiable, only the cached metadata was behind. The failed job passed on re-run with no code change, confirming a transient race. **Fix (minimal):** at every test-infra `hf` download call site, retry once with `uvx --refresh` prepended on any nonzero exit (fresh index metadata; also doubles as a plain retry for transient network errors): - `tests/launch_regression_test.py` — retry loop inside the existing `_hf_download()` helper - `tests/nano_training_smoke_test.py` — small `_hf_download(args, log)` wrapper around `_run`; both call sites use it - `tests/action_policy_regression_test.py` — same wrapper; both call sites use it - `tests/_stage_h100_inputs.sh` — 2-line `_hf_download()` bash function; all three call sites use it No behavior change on the happy path; on second failure the existing error paths fire unchanged. ## Test plan - [x] `python -m py_compile` on the three test files; `bash -n` on the stage script - [x] Both command shapes exercised live (`uvx hf@latest download ...` and `uvx --refresh hf@latest download ...`) against a small public file - [ ] PR GPU Tests run exercises every changed download path on the real runner
…new_tokens (#23) Two related Cosmos3-Edge reasoner fixes. ## 1. Vision tower config fallback (`unified_mot.py`) Cosmos3-Edge reasoner with a **vision prompt** (`reasoner_image` / `reasoner_video`) crashed at the first vision prompt: ``` huggingface_hub.errors.EntryNotFoundError: 404 Client Error. Entry Not Found for url: https://huggingface.co/nvidia/Cosmos3-Edge/resolve/main/vision_encoder/config.json ``` raised from `Nemotron3DenseVLTextForCausalLM._ensure_vision_tower`. The public `nvidia/Cosmos3-Edge` repo ships `vision_encoder/model.safetensors` (weights) but **not** `vision_encoder/config.json` — the SigLIP2 + projector spec is folded into the top-level `config.json`, where the projector field is `merger_intermediate_size` rather than `merger_intermedia`. Fix: wrap the `vision_encoder/config.json` fetch in `try/except EntryNotFoundError`, fall back to the top-level `config.json`, and accept either projector key name. **Behavior-preserving** for checkpoints that ship a standalone `vision_encoder/config.json` (the `try` succeeds → original path). Only reachable from the reasoner vision tower; generation/action modes go through the VAE tokenizer and never reach this code. ## 2. Raise reasoner default `max_new_tokens` 64 → 1024 (`defaults/reasoner/sample_args.json`) The reasoner runtime default capped generation at 64 new tokens. Edge's reasoner is a chain-of-thought model, so at 64 the decode was truncated mid-thought, before the final answer. `max_new_tokens` is only a ceiling (decode stops at EOS), so a generous default is free for cases that stop early. Measured natural completion lengths on Cosmos3-Edge: | task | tokens to natural stop | |------|------------------------| | reasoner_video | ~101 | | reasoner_image | ~149 | | reasoner (text-only CoT) | ~638 | 64 and even 256 truncate the verbose text-only chain-of-thought; 1024 lets all three terminate naturally with their final answer. ## Verification Ran all `inputs/reasoner/*.json` + `inputs/omni/*.json` on Cosmos3-Edge (4× GB200). - Before fix 1: `reasoner_image` / `reasoner_video` 404-crash. After: both produce coherent captions. - Before fix 2 (64 cap): captions cut off mid-reasoning. After (1024): full CoT + final answer for all three, e.g. *"The robot arm is pouring a red liquid from a kettle into a glass containing a spoon."* - All other modes (t2i/t2v/i2v/v2v, action forward/inverse dynamics, policy) unaffected. Audio modes (i2vs/t2vs) skipped — Edge has no sound tokenizer. > Fix 1 rebases commit `32983f9` from `pzeren/edge-reasoner-review-cleanups` onto current `main`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: liang.feng <liangf@nvidia.com>
) ## Summary Adds the Cosmos3-Edge DROID action-policy post-training recipe, porting the internal reference configuration to the OSS stack. - `action_policy_droid_edge`: derived from `action_policy_droid_nano` via deepcopy, with the Edge deltas — `EDGE_MODEL_CONFIG` backbone, `use_und_k_norm_for_gen=True` (the generator qk-norm fix, trained from the base checkpoint), lr 5e-4 with linear decay to 0.4x, idle-frame prompt conditioning, global batch 8192. - `examples/toml/sft_config/action_policy_droid_edge.toml` and `examples/launch_sft_action_policy_droid_edge.sh`: run-level scalars and launch wrapper for the reference reproduction (100k iterations, GB200 HSDP 32x8 reference shape). - Naming normalization for the action-policy assets, aligned with the registered experiment names (`action_policy_{droid,libero_10,libero_all}_{nano,edge}`): example TOMLs and launch shells renamed accordingly; `docs/action_policy_droid_posttrain.md` extended to cover both tiers. All references updated across docstrings, docs, launchers, and tests. The base checkpoint is `nvidia/Cosmos3-Edge` converted with `convert_model_to_dcp`. Note: the current HF export does not include the `k_norm_und_for_gen` tensors; the recipe tolerates this via `strict_resume=False` (fresh near-identity initialization). ## Validation - Lint and config-load validation pass; resolved values match the internal reference configuration. - Training-curve validation is in progress on GB200 at the reference batch geometry. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…-path) (#25) ## What The **Export checkpoint to Hugging Face safetensors** section of `docs/training.md` only showed the default 3-arg `export_model` command. That command **fails for Cosmos3-Edge**: - Edge's SigLIP2 vision tower is **frozen during SFT and not saved into the DCP** (the training checkpoint holds only the trainable LM + adapters). - The public `nvidia/Cosmos3-Edge` config carries a **sanitized** `backbone_path` (`s3://bucket/...`), so the default `--vit` auto-download raises `ValueError: Invalid checkpoint path: s3://bucket/...`. Users following the doc verbatim hit this wall with no guidance. ## Change Add a `### ViT / vision tower (Cosmos3-Edge)` subsection giving the two Edge-safe export forms: - **(a) `--no-vit`** — generation-only export (T2V/I2V/V2V/T2I); the vision tower isn't used by the generation path. - **(b) `--vit-checkpoint-path <snapshot>/vision_encoder`** — complete model incl. the vision tower, needed for reasoner image/video inference. Notes that Nano/Super are unaffected (their `backbone_path` resolves normally). Docs-only; no code change. ## Verification Ran the full chain end-to-end on GB200 with the current `main` code (no code changes): - 1-iter `vision_sft_edge` → training DCP → `export_model` (both forms) → HF safetensors ✅ - t2i inference with the exported model → **byte-identical** to base `Cosmos3-Edge` (export is lossless) ✅ - reasoner+image inference with the `--vit-checkpoint-path` model → correct image-grounded description (vision tower works) ✅ 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Simon Zhang <simonz@nvidia.com>
### Summary Wire up serving for the **Cosmos3-Edge-Policy-DROID** checkpoint in the RoboLab policy server, alongside the existing Cosmos3-Nano policy. The Edge policy differs from Nano in its config/tokenizer export and prompt formatting, so this change registers its target aliases, adds a structured-JSON prompt path, and repairs the VAE registry key its checkpoint conversion emits. ### Changes **`cosmos_framework/inference/common/public_model_config.py`** - Register the `Nemotron3DenseVLMoTConfig` / `Nemotron3DenseVLTextForCausalLM` target aliases used by the Edge policy, under the canonical `nemotron3_dense_vl_*` names. - Remove the stale `nemotron_3_dense_vl_*` (extra-underscore) aliases previously added in #5. They keyed the *same* target paths, so the duplicate keys collided in the `_TARGET_ALIASES` literal and shadowed the canonical names in the inverted `_ALIAS_TARGETS` map — causing `KeyError: 'nemotron3_dense_vl_text_for_causal_lm'` when loading the published Edge checkpoint config. **`cosmos_framework/scripts/action_policy_server_robolab.py`** - Add the Edge policy HF repo mapping (`nvidia/Cosmos3-Edge-Policy-DROID`) to `_ROBOLAB_POLICY_HF_REPOSITORIES`. - Add a `format_prompt_as_json` server arg to serve structured JSON prompts matching training; thread it into both the default `ActionTransformPipeline` fallback and the instantiated dataset config. - JSON-encode `ai_caption` when the transform produces it as a dict. **`cosmos_framework/utils/checkpoint_db.py`** - Re-form the bare relative Wan2.2 VAE path exported by the Edge checkpoint (empty `bucket_name`) into an `s3://bucket/...` registry key so it resolves through the registry and auto-downloads from HF, matching Nano's path. Local files still take precedence.
### Summary Documentation update to introduce the **Cosmos3-Edge-Policy-DROID** model alongside the existing **Cosmos3-Nano-Policy-DROID** in the policy server guide. Previously the docs only covered the Nano variant; the guide is now generalized to cover both post-trained DROID policy models. ### Changes **`docs/action_policy_droid_server.md`** - Retitled the guide from *Cosmos3-Nano-Policy-DROID Server* to the model-agnostic *Cosmos3-Policy-DROID Server*. - Added an intro section listing the two available post-trained DROID policy models: 1. [Cosmos3-Nano-Policy-DROID](https://huggingface.co/nvidia/Cosmos3-Nano-Policy-DROID) 2. [Cosmos3-Edge-Policy-DROID](https://huggingface.co/nvidia/Cosmos3-Edge-Policy-DROID) - Split the "start the policy server" instructions into per-model steps, adding the Edge command with its required flags (`--checkpoint_path nvidia/Cosmos3-Edge-Policy-DROID` and `--format-prompt-as-json True`). **`README.md`** - Updated the Policy Server doc reference from *Cosmos3-Nano-Policy-DROID* to *Cosmos3-Policy-DROID* to match the generalized guide. --------- Co-authored-by: lfengad <liangf@nvidia.com>
…ownload (#28) ## Summary Testing the DROID recipes against the documented public `nvidia/Cosmos3-DROID` download fails at dataloader init with `ValueError: Unknown version: 'success'`: the dataset loader resolves the DROID schema from the directory basename against its version registry, and the documented `DROID_ROOT=.../Cosmos3-DROID/success` matches no registry entry. The public download's layout (`success/` + `failure/`) matches the `droid_plus_lerobot_640x360_20260412` registry entry exactly — only the directory name must match. Documentation-only fix: - `docs/action_policy_droid_posttrain.md`: stage the download into a directory named `droid_plus_lerobot_640x360_20260412` and point `DROID_ROOT` at that parent directory (not `.../success`). - `examples/launch_sft_action_policy_droid_{nano,edge}.sh`: defaults and dataset checks updated to the same convention. No framework code changes. Pairs with the same fix in the cookbook (nvidia-cosmos/cosmos-private#22). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
## What Syncs the Cosmos3 **Edge diffusers converter** (i4 MR 10504) and **distilled-model diffusers** support (i4 MR 10417) into `cosmos-framework`, with imports adapted to the `cosmos_framework` layout. Existing (non-Edge / non-distilled) convert-to-diffusers behavior is preserved. ## Changes - `_convert_model_to_diffusers.py` / `convert_model_to_diffusers.py`: distilled FlowMatchEuler scheduler, Edge reasoner + vision sidecar, metadata-only tokenizer, `modular_model_index.json`, `EdgePolicyMetadata`. - `export_model.py`: write action-policy `policy` metadata to `checkpoint.json` (guarded on `is_edge AND action_gen` — non-action Edge exports unaffected). - `droid_lerobot_dataset.py`: `EMBODIMENT_TYPE = "droid_lerobot"`. - `docs/training.md`: new **Convert to Diffusers** section. ## Repo-local adaptations - **Signature-gate** `hidden_act` / `qk_norm_for_text` / `rope_axes_dim` on the diffusers `Cosmos3OmniTransformer` so conversion runs on the current diffusers build and auto-enables the fields once diffusers exposes them. *(temporary — droppable once the Edge-capable diffusers ships)* - **Auto-skip** `vision_encoder/` when the checkpoint has no reasoner ViT weights (warn instead of erroring); `--skip-vision-encoder` remains an explicit override. - Allow **non-action Edge** conversion (policy block written only for action policies). - Recognize the **public-format** export config in `_is_edge_exported_checkpoint`. - Fail fast with a clear message when the diffusers build lacks the Edge transformer API. *(temporary)* - Download the Edge reasoner root `model.safetensors.index.json` (was required but missing from `allow_patterns`). ## Testing (GB200, real runs) **Nano** (`vision_sft_nano`): 1-iter SFT → `export_model` → `convert_to_diffusers` → inference. - Export complete (7 shards, manifest, index). t2i inference `status=success`. - Standard diffusers conversion → **UniPC** scheduler pipeline; t2i inference from the diffusers output `status=success`. - **Distilled path**: injecting a `fixed_step_sampler_config` (sde) into the export and converting auto-detects distilled → **FlowMatchEulerDiscreteScheduler** (`stochastic_sampling=True`, `t_list` persisted) + `modular_model_index.json` with `Cosmos3DistilledModularPipeline`, `is_distilled=True`, `distilled_sigmas`. Without the config it still exports UniPC (no regression). **Edge** (`vision_sft_edge`): 1-iter SFT → `export_model` → inference. - Export self-contained (`vision_encoder/` bundle + manifest). t2i and reasoner-image inference `status=success` (reasoner correctly describes the image). - **Edge → diffusers verified end-to-end against `huggingface/diffusers` main (0.40.0.dev0)**, which exposes the Edge transformer API (`hidden_act`/`qk_norm_for_text`/`rope_axes_dim`): produces a complete `Cosmos3OmniPipeline` (transformer/ + vae/ + scheduler/ + text_tokenizer/ + vision_encoder/, root index 670 weights, no missing shards). On the currently-pinned diffusers it fails fast with a clear upgrade hint. - Note: diffusers main also needs `huggingface_hub` 1.x + `transformers` 5.x; verified via a `PYTHONPATH` overlay without touching the pinned env. The temporary shim / capability-check become no-ops once the pinned diffusers is bumped to an Edge-capable build. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
## Summary Removes the Cosmos3-Edge DROID action-policy recipe pending end-to-end validation; it will be resubmitted once validated: - `action_policy_droid_edge` experiment, its registration, `examples/toml/sft_config/action_policy_droid_edge.toml`, and `examples/launch_sft_action_policy_droid_edge.sh`. - Edge references in `docs/action_policy_droid_posttrain.md` and the action-recipe block of `docs/training.md` (docs return to Nano-only). Kept from #24/#28: the action asset naming normalization (`action_policy_{droid,libero_10,libero_all}_{nano,edge}` scheme for TOMLs/launchers/docs), the DROID dataset-path guidance for the public download, the FILTER_DIR guard, and the LIBERO conversion step. Unrelated Edge work (#25, #26, #27, #29) is untouched. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…k imaginaire4!10546) (#31) ## What Picks GitLab `cosmos-lab/imaginaire4!10546` ("Adding k norm", author @atjoshi) into cosmos-framework. Adds the **missing generator-only K normalization** (`k_norm_und_for_gen`) to the Diffusers transformer for **Cosmos3-Edge**. Source MR: https://gitlab-master.nvidia.com/cosmos-lab/imaginaire4/-/merge_requests/10546 ## Changes (`cosmos_framework/scripts/_convert_model_to_diffusers.py`) 1. **`_remap_language_model_state_dict`** — no longer drops `.k_norm_und_for_gen.` keys, so the tensor is carried into the shared Diffusers transformer weights (previously it was excluded from the root shared-weight index). 2. **`convert_model_to_diffusers`** — reads `use_und_k_norm_for_gen` from the exported VLM instance config (default `False`), logs it, and passes it to the `Cosmos3OmniTransformer` builder. ## Port note (difference from the source MR) imaginaire4 passes `use_und_k_norm_for_gen` as a direct constructor kwarg. This repo's converter gates newer constructor args through `optional_transformer_kwargs` (passed only when the installed Diffusers `Cosmos3OmniTransformer.__init__` accepts them, else warns) — its companion `qk_norm_for_text` already lives there. I threaded `use_und_k_norm_for_gen` through the same gate for consistency and backward-compatibility with older Diffusers builds. Behavior on a current Diffusers build is identical to the source MR. ## Verification - `python -m py_compile` clean. - Diff is a faithful 1:1 of the MR's two hunks, adapted to the builder shape.
## Summary The updated `nvidia/Cosmos3-Edge` HF checkpoint (`main` @ `f7f180c`) adds a per-layer `k_norm_und_for_gen` RMSNorm on the generation pathway — its `transformer/config.json` now sets `use_und_k_norm_for_gen: true` and each layer ships a new `layers.*.self_attn.k_norm_und_for_gen.weight`. The inference generator config `Cosmos3-Edge.yaml` only set `qk_norm_for_text: false` and omitted `use_und_k_norm_for_gen`, so the model was built with `k_norm_und_for_gen=None` (the module is gated in `unified_mot.py` by `use_und_k_norm_for_gen AND qk_norm_for_diffusion AND not qk_norm_for_text`). The new K-norm weights were **silently dropped at load**, and inference ran with unnormalized `K_und` that dominates the joint-attention softmax → **severely blurry generations** across t2i and other omni modes. This one-line change sets `use_und_k_norm_for_gen: true`, aligning the inference config with the checkpoint. It mirrors the reasoner-side configs (`Cosmos3EdgeReasoner_VLM_GCP_Config_*_UndKNorm` in `configs/base/defaults/reasoner.py`) which already enable it. `Cosmos3-Edge.yaml` is the only Edge inference config; Nano/Super use `qk_norm_for_text: true` and correctly do not (and must not) set this flag. ## Verification - t2i (`inputs/omni/t2i.json`, seed 0): output goes from blurry → sharp. - Full sweep of `inputs/omni/*.json` + `inputs/reasoner/*.json` (14 tasks, audio i2vs/t2vs skipped per Edge) completes with all `rc=0`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… index + single-rank DCP (#33) ## Summary Two fixes in `cosmos_framework/inference/model.py` that let `nvidia/Cosmos3-Edge-Policy-DROID` (and similar diffusers-format policy checkpoints) load and run, on single- and multi-GPU. ### 1. Tolerate a malformed root weight index (`_diffusers_weight_map`) The generator-only `k_norm_und_for_gen` tensor belongs to the transformer component and should be sourced from its own index under the diffusers name `layers.N...`. The converter is supposed to exclude it from the root `model.safetensors.index.json` (see `_convert_model_to_diffusers._remap_language_model_state_dict`). Checkpoints exported **before that converter fix** — e.g. `nvidia/Cosmos3-Edge-Policy-DROID` — leak it into the root index under the raw **model-internal** name `layers.layers.N...` (double `layers`) instead of the diffusers `layers.N...`. Those names point at tensors absent from the shards, so `read_metadata` aborts with a `KeyError` ("shard is missing 28 indexed tensor(s)"). Fix: drop root-index `k_norm_und_for_gen` entries and defer to the (correctly named) transformer index. The weight data itself is fine in the shards; only the root manifest was wrong. ### 2. Avoid the DCP collective path on a single rank (`from_pretrained_dcp`) The diffusers `dcp.load` ran the cross-rank gather even under a 1-rank process group (which the RoboLab policy server initializes via `maybe_init_distributed`). That pickles the load planner / step result and fails with `TypeError: cannot pickle code objects` — which also **masked** the real `KeyError` above. Pass `no_dist=True` when `world_size <= 1`; `world_size > 1` keeps the collective path unchanged. ## Root cause of the checkpoint bug `nvidia/Cosmos3-Edge-Policy-DROID` was exported by a converter predating the k-norm root-index fix, so its root index carries the un-remapped `layers.layers.N` k-norm keys. The correct `nvidia/Cosmos3-Edge` checkpoint has these as `layers.N`. (Re-exporting the DROID checkpoint with the current converter is the upstream fix; this PR makes the loader tolerant either way.) ## Verification - **DROID**: loads and produces a finite `[32, 8]` action chunk on 1 GPU; loads cleanly across **world_size=2 and world_size=4** (collective path). - **No regression** — checked every cached diffusers checkpoint at `refs/main`: - `Cosmos3-Edge` (k-norm on): 28/28 k-norm keys still load via the transformer index; t2i output is **byte-identical** to before this change (deterministic seed 0). - `Cosmos3-Nano`, `Cosmos3-Super`, `Cosmos3-Super-*-4Step` (k-norm off): 0 k-norm keys — the filter is a no-op. - `Cosmos3-Super-Image2Video`, `Cosmos3-Super-Text2Image`, `Cosmos3-Experimental`: non-diffusers format — this code path isn't taken. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: lfengad <liangf@nvidia.com>
…7-19 # Conflicts: # README.md
## What Adopts the HF K-norm restoration (nvidia/Cosmos3-Edge PR #30, revision `f7f180c2`: 28 trained `k_norm_und_for_gen` tensors + `use_und_k_norm_for_gen: true`) in the **training/SFT** side of the framework — the counterpart to #31 (export) and #32 (inference). Without this, the documented Edge SFT flow silently drops the restored norms: `EDGE_MODEL_CONFIG` never set the flag, so the model builds without the modules and the model-driven DCP warm start ignores the 28 tensors — training runs un-normalized gen→und cross-attention against weights trained WITH the norm (the same defect #32 fixed for inference, where it caused severely blurry generations). Separately, `videophy2_sft_edge` breaks loudly on any fresh resolve of the updated snapshot: the root index grew 670→698 keys and the indexed-snapshot loader rejects the 28 unmapped gen-only keys at startup. ## Changes - **`edge_model_config.py`**: set `use_und_k_norm_for_gen=True` (matches `Cosmos3-Edge.yaml` / #32); vision_sft_edge now builds the norm modules and loads the trained weights (verified: live TOML resolution gate=True; strict DCP load of a re-converted base loads all 28). - **`vision_sft_edge` experiment + TOML**: add `k_norm_und_for_gen` to `optimizer.keys_to_select` (upstream parity: imaginaire4 !10218 trains it alongside `moe_gen`; it was the only gen-side norm left frozen). Note: under this recipe's bf16 param storage at lr=1e-4, per-step Adam updates on ~unit-magnitude norm weights round below the bf16 ULP — this affects all such params equally (e.g. `k_norm_moe_gen` moved ≤4.8e-4 over 500 iters) and is a pre-existing recipe characteristic, documented in the repo-root SPEC. - **`checkpoint/dcp.py`**: actionable error when a strict load misses `k_norm_und_for_gen` keys (old base → re-run `convert_model_to_dcp` against `f7f180c2`+; pre-flip SFT resume → restart from a re-converted base). Unit-tested both ways (hint fires only for k-norm misses). - **`safetensors_loader.py`**: indexed-snapshot detection recognizes-and-skips the gen-only `k_norm_und_for_gen` root-index keys (no module in the reasoner); genuinely unknown keys still fail loudly. Fixes `videophy2_sft_edge` and `convert_model_to_vlm_safetensors` against post-`f7f180c2` snapshots. Unit-tested; real-snapshot bijection test updated for both pre/post-#30 indexes. - **`inference/model.py`**: revision hint when a flag-ON model meets a pre-restoration diffusers snapshot. Unit-tested. - **`docs/training.md`**: base-checkpoint refresh requirement, the two distinct framework minimums (conversion: #32 / `8cf08b4`; training + videophy2: this PR), and the new error text meaning. ## Validation (4×GB200, base re-converted from `f7f180c2`) - **500-iter stock `launch_sft_vision_edge.sh`**: `Done with training`, 499/499 loss points, 0 NaN; MA25 curve tracks the prior-base reference with a consistent small improvement (1.83→1.78 at iter 500). Strict load (`strict_resume=True`) proves the 28 norms load from the checkpoint (not fresh-init). - **Negative test**: old norm-less DCP + this recipe → aborts before iteration 1 with the actionable re-convert hint (verbatim in log). - **videophy2 6-iter smoke** against the raw post-#30 snapshot: indexed loader maps 670 keys, skips 28, loads 670 tensors, optimizer groups match the previously validated run (233 trainable/670), trains clean. - **GPU probes**: 4/4 CUDA und-k-norm forward-semantics tests pass; gradient probes confirm the norm receives gradients in real training (the bf16-rounding note above explains why its weights are quantization-stable at this LR). - Unit tests: indexed-loader 16/16, both new error-hint tests, `ruff` clean. Full investigation record: `SPEC_edge_knorm_hf_pr30_adoption.md` (repo root, untracked). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: lfengad <liangf@nvidia.com>
Sync the latest `main` from the public OSS remote (`NVIDIA/cosmos-framework`) into private `main`. ## OSS commits brought in (2) - `4698c77` docs: surface bundled Agent Skills in README (#116) - `5048fb1` feat(action): add human hand pose dataset (#71) ## Auto-merged (no conflict) `#71` files: `cosmos_framework/data/generator/action/datasets/{__init__.py, human_hand_pose_lerobot_dataset.py}`, `.../action/domain_utils.py`, `.../normalizer_stats/human_hand_pose_lerobot_stats.json`. ## Conflict resolved — `README.md` (Reference table) Both sides touched the same table. Resolution: - **Kept** the OSS `#116` addition: the new `AGENTS.md + Agent Skills` row. - **Kept private's** Policy Server naming `Cosmos3-Policy-DROID` (not OSS's `Cosmos3-Nano-Policy-DROID`), per maintainer decision. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Hans Yang <hayang@nvidia.com> Co-authored-by: Hans Yang <hayang@nvidia.com> Co-authored-by: redwood56 <tgalda@nvidia.com> Co-authored-by: redwood56 <redwood56@users.noreply.github.com>
Automated private release from i4. _source_commit: `edfc054c0bebab005b1657e70434586e42544fa7` _dest_commit (base): `888f9eb952bb1988d68ead575306ebf4263bacc3` Validation performed locally on the generated private checkout: - `packages/cosmos-framework-release/release.sh ... --pr`: succeeded; 500 mapped files, no drift BREAKs, 42 changed + 2 new target files before post-release formatting cleanup - `uv run ruff check <changed .py files>`: passed after mechanical generated-tree import sorting - `uv run ruff format --check <changed .py files>`: passed - `git diff --check`: passed - scrub of changed files for internal distill paths/secrets: no `projects.cosmos3.interactive`, `cosmos3_interactive`, `nv-00-10206`, GitLab URL, private key, or lustre path remains; remaining hits are existing private-main `s3://nv-cosmos-checkpoint-us-east-2/...` in reasoner defaults and expected `HF_TOKEN` env lookup - `uv run --extra train pytest --capture=no cosmos_framework/model/generator/distillation/common_loss_test.py cosmos_framework/model/generator/distillation/dmd2_rf_test.py cosmos_framework/model/generator/distillation/optimizer_test.py`: 111 passed Notes: - This PR is private-safety staging only; no public GitHub PR has been generated. - The i4 release manifest generated by this run updates `_source_commit`, `_dest_commit`, and adds mappings for `inference_text_kv_memory.py` and `cosine_router_test.py`; that i4 manifest update is still local pending acceptance of this private release diff.
lfengad
changed the title
…ths (#38) Sanitizes two Edge test skipif guards that hardcoded an internal lustre path (username + `cosmos-framework-private/outputs/...`). They now read `COSMOS3_EDGE_EXPORT_DIR` / `COSMOS3_EDGE_SNAPSHOT_DIR` from the env; unset still auto-skips, so behavior is unchanged. This mirrors the same commit added to the public sync PR (#119) so private `main` and public `main` stay identical after the sync. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
lfengad
force-pushed
the
merge-oss-main-2026-07-19
branch
from
July 20, 2026 16:01
6b01c91 to
fb807e4
Compare
Collaborator
Author
|
Resynced this branch to the current private `main` tip (`fb807e4`) — now 47 commits ahead of `origin/main`, still a clean fast-forward (no conflicts). Two commits added since the last push:
After merge, public `main` file content == private `main`. Rescanned the new commits — no secrets/tokens/internal paths. Merge with "Create a merge commit" to keep histories aligned. |
qmiao-hub
self-requested a review
July 20, 2026 16:07
lfengad
enabled auto-merge (squash)
July 20, 2026 16:07
qmiao-hub
approved these changes
Jul 20, 2026
yaoxu-crypto
approved these changes
Jul 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
9 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Brings the public
mainup to the current privatemainin a single sync. The headline additions are Cosmos3-Edge support (inference + post-training) and a Distillation training path.Highlights
Cosmos3-Edge
max_new_tokensdefault raised.nvidia/Cosmos3-Edge; ViT export (--no-vit/--vit-checkpoint-path) documented.Distillation
DistillationTrainertraining path (inherits the base trainer, overrides only the optimizer hooks) with tests.Other
COSMOS3_EDGE_EXPORT_DIR/COSMOS3_EDGE_SNAPSHOT_DIR) — no internal paths ship publicly.Sync details
origin/main;origin/mainis a direct ancestor → clean fast-forward, no conflicts.mainfile content == privatemain.Merge method
Merge with "Create a merge commit" (preserve all commits) so public history stays aligned with private and future OSS syncs remain clean.
🤖 Generated with Claude Code