natten: add gb300 x86_64 wheel to v1.5.0 index#60
Merged
Conversation
Companion to #58 (aarch64). Adds the sm_103 (Blackwell Ultra / GB300) x86_64 build of NATTEN 0.21.6.dev6 to the v1.5.0 index. Built with CUDA 13.0.2 + torch 2.10.0+cu130, NATTEN_CUDA_ARCH appended with 10.3 -> sm_103a. Verified on B300 (sm_103): na2d runs, no 'no kernel image' error. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
lfengad
added a commit
to NVIDIA/cosmos-framework
that referenced
this pull request
Jul 3, 2026
## Problem On **x86_64**, the `cu130` dependency group pinned the stock natten wheel `natten==0.21.6.dev6+cu130.torch210`, which does **not** contain `sm_103a` kernels. On B300 / GB300 (compute capability 10.3) this fails at runtime with `no kernel image is available for execution on the device`. ## Change - Point x86_64 cu130 natten at `==0.21.6.dev6+cu130.torch210.gb300`, mirroring the existing aarch64 entry. That build appends `10.3 -> sm_103a` to `NATTEN_CUDA_ARCH`. - Regenerated `uv.lock` (now resolves the x86_64 gb300 wheel, sha256 `04ace5d5…`). The wheel is published in the cosmos index via nvidia-cosmos/cosmos-dependencies#60. ## Verification - `uv lock --check` passes (lock consistent with pyproject). - End-to-end: `Cosmos3-Nano` t2i inference on a B300 (sm_103) completes successfully — 960×960 image generated, no "no kernel image" error. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: lfengad <liangf@nvidia.com>
erichudev
pushed a commit
to KunlunxinAD/cosmos-framework
that referenced
this pull request
Jul 7, 2026
* Add Docker build CI workflow (NVIDIA#73) Co-authored-by: Xiangyu Lu <xiangyl@nvidia.com> * Release: sync from i4 (vlm → reasoner rename) + inference/config fixes (NVIDIA#70) Upstream i4 refactor `4190136a09` renamed `projects/cosmos3/vfm/` to `projects/cosmos3/cosmos3/` and moved the `vlm/` subtree to `reasoner/` under configs/base, models, datasets, and utils. This release re-runs the release pipeline against that new source layout and cleans up CF-side knock-on damage. Highlights: * Release pipeline output: 459 files mapped from the new i4 layout. Files added under `cosmos_framework/**/reasoner/`; the old `cosmos_framework/**/vlm/` counterparts are removed (orphan cleanup). Also ships `callbacks/tokens_per_sec.py`, `model/tokenizer/utils/vlm_prompt_format.py`, a new `configs/base/defaults/experimental/`-excluded layout, and refreshed `data/vfm/augmentors/reasoner/*` etc. * CF-owned config move: `configs/base/vlm/{defaults,experiment}/*.py` moved to `configs/base/reasoner/{defaults,experiment}/` (5 files), with imports rewritten `cosmos_framework.configs.base.vlm.` → `cosmos_framework.configs.base.reasoner.` so `task="vlm"` still finds a live experiment tree. * `configs/toml_config/toml_config_helper.py`: retarget `task="vlm"` → `cosmos_framework/configs/base/reasoner/config.py`. * Inference config rewriters (`inference/common/config.py`, `inference/common/public_model_config.py`): add `vlm` → `reasoner` rewrite rules ahead of the general vfm rules so old checkpoint JSONs (which still ship `cosmos3._src.vfm.configs.base.defaults.vlm.*` targets) resolve to the new module paths at load time. * `configs/base/experiment/sft/models/{nano,super}_model_config.py` and the `inference/configs/model/Cosmos3-{Nano,Super}.yaml` + `examples/checkpoints/Cosmos3-Nano/model/config.json`: drop the removed `DiffusionExpertConfig` fields (`position_embedding_type`, `rope_{h,t,w}_extrapolation_ratio`) and retarget the shipped Qwen3-VL JSON path to the `reasoner/` layout. * `inference/common/public_model_config_test.py`: update stale `vlm` paths in the round-trip test fixture. * `model/vfm/mot/unified_mot_test.py`: removed (tested private helpers `_PACKAGE_ROOT` / `_resolve_packaged_config_path` that were refactored out of `unified_mot.py`). Testing (4 × NVIDIA GB200 node): * `test_launch_regression[vision_sft_nano]`: 10-iter losses match gb200 goldens exactly. * `test_launch_regression[llava_ov]`: trains 10 iters end to end; loss drifts vs. 2026-05-18 gb200 goldens (goldens need recapture). * `nano_reasoner_inference_smoke_test`: inference runs cleanly; argmax hard-gate passes; tight allclose (rtol=atol=1e-3) fails by bf16 noise vs. presumably-H100 goldens. * Unit suite (`--num-gpus=0 --levels=0`): 327 passed / 1 failed (the failure is the `convert_model_to_dcp` script test, which needs `HF_TOKEN` for the private `nvidia/Cosmos3-Experimental` snapshot). --------- Co-authored-by: lfengad <liangf@nvidia.com> * [Cosmos3 OSS]Add more action datasets (NVIDIA#72) Dataset support added: 1. Fractal (FractalLeRobotDataset class) 2. RoboMind Frank dual arm (through existing RoboMINDFrankaDataset class) 3. RoboMind UR (RoboMINDURDataset class) Other changes: 1. Add corresponding stats files for the newly added datasets 2. Folder structure refactor (minor) --------- Co-authored-by: lfengad <liangf@nvidia.com> * Add Cosmos3-Nano LIBERO-10 action-policy SFT recipe, config, eval harness, and doc (NVIDIA#61) ## What Adds the **Cosmos3-Nano LIBERO-10 action-policy SFT** surface, mirroring the existing DROID counterpart (`action_policy_droid_nano` + toml + launcher + doc). ### Feature (net-new) - **Experiment configs** `action_policy_libero_nano` (libero_10-only) and `action_policy_libero_all_nano` (equal 4-suite mix) — gen + action heads from the public Cosmos3-Nano base. - **Dataset** `LIBEROLeRobotDataset` + `get_action_libero_sft_dataset` — frame_wise_relative rot6d, `quantile_rot`, concat_view (third-person + wrist), 20 fps. - `base_dataset` `tasks.parquet` fallback for community LIBERO layouts. - Resample-on-decode-failure guard so one undecodable packed-mp4 frame can't crash a multi-node run (matches i4 behavior). - **Closed-loop eval harness** with vectorized sim, batched `/predict_batch`, single-rank `no_dist` checkpoint load. - **Structured-prompt serving** in the policy server (`--format-prompt-as-json`), so eval matches the training prompt format; the recipe defaults to it. ### Recipe + doc — two presets (to match the Cosmos3 LIBERO-10 result) Both lr 5e-5, warmup 500, cycle 16000, global batch 2048 (HSDP 2x8): - **(A) libero_10-only** — `action_policy_libero_repro.toml` + `launch_sft_action_policy_libero.sh` (max_iter 2000). - **(B) libero-all (4-suite equal mix)** — `action_policy_libero_all_repro.toml` + `launch_sft_action_policy_libero_all.sh` (max_iter 5000; `LIBERO_ROOT` = LIBERO_LeRobot_v3 parent dir). - `docs/action_policy_libero_sft.md` documents both. ## Notes - Scoped to LIBERO only; broader action-dataloader/model changes are intentionally not included here. - Based on `main`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Remove eval CLI references (NVIDIA#5) Co-authored-by: Xiangyu Lu <xiangyl@nvidia.com> Co-authored-by: lfengad <liangf@nvidia.com> * Add multi-control transfer inference smoke test (NVIDIA#77) ## What Adds `test_nano_inference_multi_control_transfer` to `tests/nano_inference_smoke_test.py`, giving the Cosmos3-Nano generator inference smoke test coverage for the **multi-control transfer** feature (two control hints blended by `multi_control_two_way_attention`). It mirrors the existing single-control transfer run (same `latency` preset, 4 ranks → cfgp=2/cp=2), but the generated spec sets **two** hints (`edge` + `blur`) with per-hint `weight` and **no** `control_path`, so both controls are computed on the fly from a single `vision_path` (the pinned public `robot_pouring.mp4` clip) and aggregated by the weighted N-pass multi-control attention. ## What it verifies - **Multi-control path executes**: 2 hints → `control_weights` set → the network routes to `multi_control_two_way_attention` (both controls computed on the fly). - **Process stays within expected invariants**: the framework's multi-control runtime asserts (weights length == #controls, packing consistency, batch_size==1) must hold or the run exits non-zero → the test fails. Plus the test's own asserts: exactly one output; both `edge` and `blur` active with a `weight` and no `control_path`; `vision_path` set; `control_guidance` and `guidance` > 1.0. - **Output is valid**: non-degenerate `vision.mp4` via the existing `_assert_video_has_content` (≥16 frames, finite pixels, pixel std > 3). Smoke-level (output validity + path executed under asserts), not numeric goldens — consistent with the rest of the module. ## Test Runs on the same 8-GPU gate as `test_nano_inference_omni`: ``` pytest -s tests/nano_inference_smoke_test.py --num-gpus=8 --levels=2 -o addopts= ``` Verified end to end on a 4-rank `latency` run (exit 0; log shows both `Computing edge control input on the fly` and `Computing blur control input on the fly`; all assertions pass). Collection confirmed both tests register (`MAX_GPUS=8`). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Lazily import `lerobot` in datasets. (NVIDIA#76) `lerobot` is a heavy package that pins a lot of python packages. We can lazily import it to support a lightweight env setup. Signed-off-by: Hong-Yu Chiu <hongyuc@nvidia.com> Co-authored-by: lfengad <liangf@nvidia.com> * Release: vfm → generator, vlm → reasoner dest rename (moves + import … (NVIDIA#78) …paths only) Follow-up to the initial vlm→reasoner sync (PR NVIDIA#70) — this PR renames the top-level CF dest folders and rewrites every import that references them. Contents are strictly file relocations + path-string rewrites; no code-behavior edits. Dest-folder rename: * cosmos_framework/data/vfm/ → cosmos_framework/data/generator/ * cosmos_framework/model/vfm/ → cosmos_framework/model/generator/ * cosmos_framework/utils/vfm/ → cosmos_framework/utils/generator/ * cosmos_framework/data/vlm/ → cosmos_framework/data/reasoner/ * cosmos_framework/utils/vlm/ → cosmos_framework/utils/reasoner/ * cosmos_framework/configs/base/vlm/{defaults,experiment}/ → cosmos_framework/configs/base/reasoner/{defaults,experiment}/ Import-path rewrites — every reference under ``cosmos_framework.{model,data,utils}.{vfm,vlm}.*`` and ``cosmos_framework.configs.base.vlm.*`` is retargeted to the new module names, in: * Release-managed files produced by cosmos-framework-release. * CF-owned production code (inference/, callbacks/, configs/, model/, utils/, data/generator/*/, data/reasoner/*/, tools/, scripts/). * Runtime rewriters: cosmos_framework/inference/common/config.py::CONFIG_REPLACEMENTS_INVERSE and cosmos_framework/inference/common/public_model_config.py (``_canonicalize_module_path``, ``_replace_vfm_module_prefix``, ``_replace_vfm_file_prefix``, ``_module_exists`` probes, {package}-templated dest strings) — old checkpoint JSONs' canonical ``cosmos3._src.vfm.*`` / ``projects.cosmos3.vfm.*`` target strings now map to the new module locations at load time. Verified: * Fresh AST-walker sweep across ``cosmos_framework/`` → **0 dangling ``cosmos_framework.*`` imports**. * Every ``M`` (modified-content) file's diff is verified path-only (each +/- line contains a ``vfm|vlm|generator|reasoner`` substring or is whitespace). * Every ``R`` is a git-detected rename. * 4-GPU GB200 regression: ``test_launch_regression[vision_sft_nano]`` passes and matches gb200 goldens exactly. * Unit suite (venv Python 3.13): 327 passed / 1 failed (the failure is the ``convert_model_to_dcp`` script test, which needs ``HF_TOKEN`` for the private ``nvidia/Cosmos3-Experimental`` snapshot — not fixable in-repo). * Convert Cosmos3-Nano vision tower in reasoner VLM export + fix public-alias remap (NVIDIA#79) ## Summary Fixes two issues in the Reasoner / VideoPhy-2 SFT "Step 2" checkpoint prep (`convert_model_to_vlm_safetensors`), so the exported VLM is fully sourced from Cosmos3-Nano, and adds a regression test. ### 1. Fix `vlm`→`reasoner` remap for public-alias file paths (`public_model_config.py`) The i4 `vlm`→`reasoner` rename was applied to the module-path remap helpers but **not** to their file-path siblings. The current released `nvidia/Cosmos3-Nano` snapshot stores paths as public URIs (`cosmos3://vfm/models/vlm/qwen3_vl/configs/...`), which route through the file-path helpers and resolved to a non-existent `cosmos_framework/model/vfm/vlm/qwen3_vl/...`, crashing checkpoint load with `FileNotFoundError`. This affects loading the released checkpoint generally (not just the converter). Added the missing `vlm`↔`reasoner` rules to both `_replace_vfm_file_prefix` (public→runtime) and `_public_string_from_runtime_file_prefix` (runtime→public), mirroring the module-path helpers; the public↔runtime round-trip is stable. ### 2. Convert the Cosmos3-Nano vision tower (`convert_model_to_vlm_safetensors.py`) The converter previously extracted only the language model and left the visual tower as the **stock Qwen3-VL** weights — because the generation model instantiates no `visual` submodule, so the 351 `vision_encoder/` tensors never appear in its state dict. Now the visual tower is loaded directly from the checkpoint's `vision_encoder/` shards (`_load_vision_state`) and overlaid as `model.visual.*`. Checkpoints without a vision tower (Text2Image / Image2Video) keep Qwen3-VL's tower unchanged. The merged export is now 100% Cosmos3-Nano-sourced (399 LM + 351 visual). ### 3. Regression test (`tests/launch_regression_test.py`) `test_convert_reasoner_converts_all_qwen_tensors` asserts: 1. merged tensor set == stock Qwen3-VL set (all included, none extra); 2. every `model.visual.*` tensor matches the Cosmos3-Nano `vision_encoder/` source bit-for-bit; 3. a non-trivial subset differs from stock Qwen3-VL (so it catches the vision-drop regression — not a no-op); 4. the language tower was overlaid too. The ~16 GB output is regenerated per run and removed via a finalizer; inputs stay in the shared HF cache. ## Test plan - ✅ New test passes on a 4-GPU node: `1 passed`, `351 visual tensors all sourced from Cosmos3-Nano; 309 differ from stock Qwen3-VL`, output cleaned up (basetemp 13K after finalizer). - ✅ End-to-end: VideoPhy-2 Reasoner SFT loads `750 keys` from the merged checkpoint and trains cleanly across 4 GPUs (loss ~0.66→~0.15 over 20 iters). - ✅ Existing `public_model_config_test.py` still passes; public↔runtime round-trip verified stable. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * [Cosmos3 OSS] Fix RoboMind Dataset (NVIDIA#81) Fix RoboMind Dataset - RoboMINDFrankaDataset supports loading from both `robomind-franka` and `robomind-franka-dual`, but it only load `norm_stats` for `robomind-franka-dual` - With this fix it will load `norm_stats` for `robomind-franka-dual` or `robomind-franka depends on the embodiment_type. * Fix multi-control transfer: honor control weights under torch.compile (NVIDIA#82) ## Summary Multi-control transfer inference silently ignored per-control weights. ## Changes - **`omni_mot_model.py`**: propagate `control_weights` into `gen_data_for_packing` - **`attention.py`**: make `multi_control_two_way_attention` `torch.compile`-safe. Its per-segment lengths come from data-dependent unpadding (unbacked symints), which tripped several Dynamo guards under `fullgraph=True`: - `torch._check(k.shape[0] == v.shape[0])` — frontend K/V-length guard - `torch._check(n_q > 0)` / `torch._check(n_kv > 0)` — NATTEN `max_seqlen == 0` / `< 1` varlen guards - `torch._check(n_full == noisy_e)` — makes per-segment `[cs:ce]` slices concrete-length, fixing the in-place write shape guard - pass `cumulative_seqlen_{Q,KV}` + `max_seqlen_{Q,KV}` instead of `seqlens_{Q,KV}` to avoid the disallowed `generate_varlen_parameters` device-host sync inside the compiled region ## Test plan - [x] PAI-Bench-C multi_control eval runs end-to-end under compiled attention (previously crashed with data-dependent Dynamo guard errors). - [x] Diagnostics confirm `control_weights` reach the network and the weighted-sum path executes. - [x] A/B check: extreme weights (e.g. edge=1 vs seg=1) now produce different outputs (previously byte-identical). - [x] Single-control transfer paths unaffected (they use the static-shape `two_way_attention`). Co-authored-by: Maosheng Liao <maoshengl@nvidia.com> * Release: sync from internal （clean run) (NVIDIA#83) Standard cosmos-framework-release pipeline run against current main (generator/reasoner dest layout). Brings the latest i4 changes into CF. Highlights: * New leaf helper shipped: cosmos_framework/utils/generator/torchcodec_video.py (mapped via a new [[overrides]] entry — it lives at the top-level projects/cosmos3/utils/, outside the bulk cosmos3/cosmos3/utils mapping, and is imported by the pkl_to_media / reasoner.bytes_to_media augmentors). * New augmentors: data/generator/augmentors/multi_reference_transform.py, torchcodec_callers_test.py. * 13 existing shipped files refreshed to match i4 head. _source_commit (i4): 546faba4 _dest_commit (cf base): 6efeea3 Co-authored-by: lfengad <liangf@nvidia.com> * deps: use natten gb300 (sm_103) wheel for x86_64 cu130 (NVIDIA#84) ## Problem On **x86_64**, the `cu130` dependency group pinned the stock natten wheel `natten==0.21.6.dev6+cu130.torch210`, which does **not** contain `sm_103a` kernels. On B300 / GB300 (compute capability 10.3) this fails at runtime with `no kernel image is available for execution on the device`. ## Change - Point x86_64 cu130 natten at `==0.21.6.dev6+cu130.torch210.gb300`, mirroring the existing aarch64 entry. That build appends `10.3 -> sm_103a` to `NATTEN_CUDA_ARCH`. - Regenerated `uv.lock` (now resolves the x86_64 gb300 wheel, sha256 `04ace5d5…`). The wheel is published in the cosmos index via nvidia-cosmos/cosmos-dependencies#60. ## Verification - `uv lock --check` passes (lock consistent with pyproject). - End-to-end: `Cosmos3-Nano` t2i inference on a B300 (sm_103) completes successfully — 960×960 image generated, no "no kernel image" error. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: lfengad <liangf@nvidia.com> * Allow config aliases as public targets (NVIDIA#4) Co-authored-by: Xiangyu Lu <xiangyl@nvidia.com> --------- Signed-off-by: Hong-Yu Chiu <hongyuc@nvidia.com> Co-authored-by: Xiangyu Lu <169013972+xlu451@users.noreply.github.com> Co-authored-by: Xiangyu Lu <xiangyl@nvidia.com> Co-authored-by: yy-code-nv <yangyangt@nvidia.com> Co-authored-by: lfengad <liangf@nvidia.com> Co-authored-by: Liangkai Zhang <liangkaiz@nvidia.com> Co-authored-by: LiangHao <hliangac@connect.ust.hk> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Hongyu, Chiu <hongyuc@nvidia.com> Co-authored-by: Trung Pham <trungp@nvidia.com> Co-authored-by: Maosheng Liao <maoshengl@nvidia.com> Co-authored-by: pengcuo <pzeren@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Companion to #58 (aarch64). Adds the sm_103 (Blackwell Ultra / GB300) x86_64 build of NATTEN
0.21.6.dev6to the v1.5.0 index.Changes
natten-0.21.6.dev6+cu130.torch210.gb300-cp313-cp313-linux_x86_64.whlindocs/v1.5.0/natten/index.html(sha25604ace5d591b2775149d7a9adc8f3e6099b18fedad364c48e70b4a998ee598b9c).The build-infra changes (
LOCAL_VERSION_SUFFIX,NATTEN_CUDA_ARCH+=10.3) already landed in #58, so this PR only adds the index entry.Build
0.21.6.dev6(07c82f5)torch==2.10.0+cu130, python 3.13NATTEN_CUDA_ARCH="8.0;8.6;9.0;10.0;12.0;10.3"-> NATTEN maps10.3 -> sm_103aVerification
cuobjdump libnatten.socontainssm_103a(alongside sm_80/86/90a/100a/120)na2don a B300 (sm_103): completes, no "no kernel image" error🤖 Generated with Claude Code