Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
4 changes: 2 additions & 2 deletions .agents/skills/cosmos3-codebase-nav/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ All paths below are relative to this file's location (`.agents/skills/cosmos3-co

- `cosmos_framework/` — main training package (data, model, trainer, callbacks, checkpoint, utils, …).
- `cosmos_framework/configs/base/experiment/` — vfm (generator) experiment SKUs referenced by `[train.train_policy].experiment` in the recipe TOMLs.
- `cosmos_framework/configs/base/vlm/experiment/` — vlm (reasoner) experiment SKUs.
- `cosmos_framework/configs/base/reasoner/experiment/` — vlm (reasoner) experiment SKUs.
- `cosmos_framework/inference/` — inference subpackage (args, model, inference engine, defaults, Ray serving, common helpers).
- `cosmos_framework/scripts/` — top-level entry-point scripts (train, inference, eval, export_model, convert_model_to_dcp, upsample_prompts, caption_from_video, captions_to_sft_jsonl, action_policy_server, …). Invoked as `python -m cosmos_framework.scripts.<name>`.
- `examples/toml/sft_config/<recipe>.toml` + `examples/launch_sft_<recipe>.sh` — paired SFT recipes (training entry-point input). The shell sources `examples/_sft_launcher_common.sh`, which forwards into `cosmos_framework.scripts.train --sft-toml=...`.
Expand All @@ -45,7 +45,7 @@ All paths below are relative to this file's location (`.agents/skills/cosmos3-co
| SFT recipe TOMLs (paired with `examples/launch_sft_*.sh`) | `../../../examples/toml/sft_config/<recipe>.toml` |
| SFT pydantic schema (validates the recipe TOML) | `../../../cosmos_framework/configs/toml_config/sft_config.py` |
| Training experiment SKUs (vfm) | `../../../cosmos_framework/configs/base/experiment/` |
| Training experiment SKUs (vlm / reasoner) | `../../../cosmos_framework/configs/base/vlm/experiment/` |
| Training experiment SKUs (vlm / reasoner) | `../../../cosmos_framework/configs/base/reasoner/experiment/` |
| Example inputs | `../../../inputs/omni/t2i.json`, `../../../inputs/omni/t2v.json`, `../../../inputs/omni/i2v.json`, … |

Available modality modes for defaults: `text2image`, `text2video`, `image2video`, `image2image`, `video2video`, `forward_dynamics`, `inverse_dynamics`, `policy`.
Expand Down
4 changes: 2 additions & 2 deletions .claude/skills/cosmos3-codebase-nav/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ All paths below are relative to this file's location (`.claude/skills/cosmos3-co

- `cosmos_framework/` — main training package (data, model, trainer, callbacks, checkpoint, utils, …).
- `cosmos_framework/configs/base/experiment/` — vfm (generator) experiment SKUs referenced by `[train.train_policy].experiment` in the recipe TOMLs.
- `cosmos_framework/configs/base/vlm/experiment/` — vlm (reasoner) experiment SKUs.
- `cosmos_framework/configs/base/reasoner/experiment/` — vlm (reasoner) experiment SKUs.
- `cosmos_framework/inference/` — inference subpackage (args, model, inference engine, defaults, Ray serving, common helpers).
- `cosmos_framework/scripts/` — top-level entry-point scripts (train, inference, eval, export_model, convert_model_to_dcp, upsample_prompts, caption_from_video, captions_to_sft_jsonl, action_policy_server, …). Invoked as `python -m cosmos_framework.scripts.<name>`.
- `examples/toml/sft_config/<recipe>.toml` + `examples/launch_sft_<recipe>.sh` — paired SFT recipes (training entry-point input). The shell sources `examples/_sft_launcher_common.sh`, which forwards into `cosmos_framework.scripts.train --sft-toml=...`.
Expand All @@ -45,7 +45,7 @@ All paths below are relative to this file's location (`.claude/skills/cosmos3-co
| SFT recipe TOMLs (paired with `examples/launch_sft_*.sh`) | `../../../examples/toml/sft_config/<recipe>.toml` |
| SFT pydantic schema (validates the recipe TOML) | `../../../cosmos_framework/configs/toml_config/sft_config.py` |
| Training experiment SKUs (vfm) | `../../../cosmos_framework/configs/base/experiment/` |
| Training experiment SKUs (vlm / reasoner) | `../../../cosmos_framework/configs/base/vlm/experiment/` |
| Training experiment SKUs (vlm / reasoner) | `../../../cosmos_framework/configs/base/reasoner/experiment/` |
| Example inputs | `../../../inputs/omni/t2i.json`, `../../../inputs/omni/t2v.json`, `../../../inputs/omni/i2v.json`, … |

Available modality modes for defaults: `text2image`, `text2video`, `image2video`, `image2image`, `video2video`, `forward_dynamics`, `inverse_dynamics`, `policy`.
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/gpu-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -256,13 +256,13 @@ jobs:
run: |
export LD_LIBRARY_PATH=
uv run --all-extras --group=cu128-train torchrun --nproc_per_node=2 -m pytest -v \
cosmos_framework/model/vfm/mot/cfgp_ar_test.py -o addopts=
cosmos_framework/model/generator/mot/cfgp_ar_test.py -o addopts=

- name: Distributed unit tests - context_parallel (torchrun, 4 ranks)
run: |
export LD_LIBRARY_PATH=
uv run --all-extras --group=cu128-train torchrun --nproc_per_node=4 -m pytest -v \
cosmos_framework/model/vfm/mot/context_parallel_test.py -o addopts=
cosmos_framework/model/generator/mot/context_parallel_test.py -o addopts=

# Clear everything the suite writes into the working tree (all gitignored
# scratch): pytest tmp dirs (DCP checkpoint, logs), the script-test
Expand Down
14 changes: 7 additions & 7 deletions NOTICE
Original file line number Diff line number Diff line change
Expand Up @@ -15,18 +15,18 @@ Notices:
Copyright 2025 The Qwen Team and The HuggingFace Inc. team.
Copyright 2025 HuggingFace Inc. team.
Files:
cosmos_framework/model/vfm/vlm/qwen3_vl/configuration_qwen3_vl.py
cosmos_framework/model/vfm/vlm/qwen3_vl/qwen3_vl.py
cosmos_framework/model/vfm/vlm/qwen3_vl/utils.py
cosmos_framework/model/vfm/vlm/qwen3_vl/video_processing_qwen3_vl.py
cosmos_framework/model/generator/vlm/qwen3_vl/configuration_qwen3_vl.py
cosmos_framework/model/generator/vlm/qwen3_vl/qwen3_vl.py
cosmos_framework/model/generator/vlm/qwen3_vl/utils.py
cosmos_framework/model/generator/vlm/qwen3_vl/video_processing_qwen3_vl.py

ByteDance-Seed Bagel / Qwen2 tokenizer
Source: https://github.com/ByteDance-Seed/Bagel
License: Apache-2.0
Notice:
Copyright 2024 The Qwen Team and The HuggingFace Inc. team.
File:
cosmos_framework/model/vfm/tokenizers/tokenization_qwen2.py
cosmos_framework/model/generator/tokenizers/tokenization_qwen2.py

HuggingFace Diffusers UniPC scheduler
Source: https://github.com/huggingface/diffusers
Expand All @@ -35,7 +35,7 @@ Notices:
Copyright 2024 TSAIL Team and The HuggingFace Team. All rights reserved.
Copyright 2024-2025 The Alibaba Wan Team Authors. All rights reserved.
File:
cosmos_framework/model/vfm/diffusion/samplers/fm_solvers_unipc.py
cosmos_framework/model/generator/diffusion/samplers/fm_solvers_unipc.py

Detectron2 lazy config helpers
Source: https://github.com/facebookresearch/detectron2
Expand All @@ -51,4 +51,4 @@ License: GPL-3.0
Notice:
Upstream source files do not contain an explicit copyright notice.
Files:
cosmos_framework/model/vfm/utils/taylorseer.py
cosmos_framework/model/generator/utils/taylorseer.py
4 changes: 2 additions & 2 deletions cosmos_framework/callbacks/compile_tokenizer.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
"""Training callback that defers AOT compilation of the VAE tokenizer.

The actual compilation logic lives in
:meth:`~cosmos_framework.model.vfm.tokenizers.wan2pt2_vae_4x16x16.Wan2pt2VAEInterface.compile_encode`.
:meth:`~cosmos_framework.model.generator.tokenizers.wan2pt2_vae_4x16x16.Wan2pt2VAEInterface.compile_encode`.
This module provides a :class:`CompileTokenizer` callback that invokes it
at the right point during training (after ``compile_after_iterations``
steps, to avoid NCCL timeouts during CUDA/cuDNN warm-up).
Expand All @@ -27,7 +27,7 @@

from cosmos_framework.utils import log
from cosmos_framework.utils.callback import Callback
from cosmos_framework.model.vfm.omni_mot_model import OmniMoTModel
from cosmos_framework.model.generator.omni_mot_model import OmniMoTModel


class CompileTokenizer(Callback):
Expand Down
2 changes: 1 addition & 1 deletion cosmos_framework/callbacks/every_n_draw_sample.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
from cosmos_framework.utils import distributed, log, misc
from cosmos_framework.utils.easy_io import easy_io
from cosmos_framework.tools.visualize.video import save_img_or_video
from cosmos_framework.utils.vfm.data_utils import slice_data_batch
from cosmos_framework.utils.generator.data_utils import slice_data_batch


def resize_image(image: torch.Tensor, size: int = 1024) -> torch.Tensor:
Expand Down
2 changes: 1 addition & 1 deletion cosmos_framework/callbacks/expert_heatmap.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
from cosmos_framework.model._base import ImaginaireModel
from cosmos_framework.trainer import ImaginaireTrainer
from cosmos_framework.utils import distributed
from cosmos_framework.model.vfm.reasoner.qwen3_vl_moe.qwen3_vl_moe import Qwen3VLMoeTextSparseMoeBlock
from cosmos_framework.model.generator.reasoner.qwen3_vl_moe.qwen3_vl_moe import Qwen3VLMoeTextSparseMoeBlock


def compute_expert_heatmap(vfm: torch.nn.Module) -> dict[str, torch.Tensor]:
Expand Down
2 changes: 1 addition & 1 deletion cosmos_framework/callbacks/hf_export.py
Original file line number Diff line number Diff line change
Expand Up @@ -133,7 +133,7 @@ def on_save_checkpoint(self, model: Any, state_dict: dict[str, Any]) -> None:
return

# Deferred import to avoid circular dependency at module load time.
from cosmos_framework.model.vfm.vlm_model import VLMModel
from cosmos_framework.model.generator.vlm_model import VLMModel

if not isinstance(model, VLMModel):
# The legacy vlm/train.py path passes model_parts: list[nn.Module] (raw HF
Expand Down
2 changes: 1 addition & 1 deletion cosmos_framework/callbacks/moe_specialization_callback.py
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@
from cosmos_framework.model._base import ImaginaireModel
from cosmos_framework.trainer import ImaginaireTrainer
from cosmos_framework.utils import distributed
from cosmos_framework.model.vfm.reasoner.qwen3_vl_moe.qwen3_vl_moe import Qwen3VLMoeTextSparseMoeBlock
from cosmos_framework.model.generator.reasoner.qwen3_vl_moe.qwen3_vl_moe import Qwen3VLMoeTextSparseMoeBlock


def _get_device_mesh(vfm: torch.nn.Module):
Expand Down
2 changes: 1 addition & 1 deletion cosmos_framework/callbacks/moe_stability_callback.py
Original file line number Diff line number Diff line change
Expand Up @@ -107,7 +107,7 @@
from cosmos_framework.model._base import ImaginaireModel
from cosmos_framework.trainer import ImaginaireTrainer
from cosmos_framework.utils import distributed
from cosmos_framework.model.vfm.reasoner.qwen3_vl_moe.qwen3_vl_moe import Qwen3VLMoeTextSparseMoeBlock
from cosmos_framework.model.generator.reasoner.qwen3_vl_moe.qwen3_vl_moe import Qwen3VLMoeTextSparseMoeBlock


def _effective_experts(
Expand Down
2 changes: 1 addition & 1 deletion cosmos_framework/callbacks/norm_monitor.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
from cosmos_framework.utils import distributed, log, misc
from cosmos_framework.utils.callback import Callback
from cosmos_framework.utils.easy_io import easy_io
from cosmos_framework.data.vfm.sequence_packing.runtime import get_gen_seq
from cosmos_framework.data.generator.sequence_packing.runtime import get_gen_seq

try:
from apex.contrib.layer_norm import FastLayerNorm
Expand Down
2 changes: 1 addition & 1 deletion cosmos_framework/callbacks/sequence_packing_padding.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
from cosmos_framework.callbacks.every_n import EveryN
from cosmos_framework.model._base import ImaginaireModel
from cosmos_framework.trainer import ImaginaireTrainer
from cosmos_framework.data.vfm.sequence_packing.runtime import get_padding_stats
from cosmos_framework.data.generator.sequence_packing.runtime import get_padding_stats


class SequencePackingPadding(EveryN):
Expand Down
2 changes: 1 addition & 1 deletion cosmos_framework/callbacks/tokens_per_sec.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@
``MFUCallback`` is written for the OmniMoT/VFM network -- it reads
``model.net.language_model.config`` and per-modality token counts
(``output_batch["und_token_length"]`` etc.). ``VLMModel``
(``cosmos_framework/model/vfm/vlm_model.py``) is a plain HF wrapper: it exposes
(``cosmos_framework/model/generator/vlm_model.py``) is a plain HF wrapper: it exposes
``self.model`` (no ``.net``) and its ``training_step`` returns only
``{"loss", "loss_avg", "labels"}``. ``MFUCallback`` would therefore silently
no-op (token length is ``None``) or fail on ``model.net``. This callback instead
Expand Down
2 changes: 1 addition & 1 deletion cosmos_framework/callbacks/training_stats.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@
from cosmos_framework.utils import distributed
from cosmos_framework.utils.callback import Callback
from cosmos_framework.callbacks.wandb_log import _LossRecord
from cosmos_framework.data.vfm.action.domain_utils import EMBODIMENT_TO_DOMAIN_ID
from cosmos_framework.data.generator.action.domain_utils import EMBODIMENT_TO_DOMAIN_ID

# Build inverse mapping: domain_id -> embodiment_type. First occurrence wins when multiple embodiment names share the
# same domain id.
Expand Down
2 changes: 1 addition & 1 deletion cosmos_framework/checkpoint/dcp.py
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,7 @@
from cosmos_framework.model._base import ImaginaireModel
from cosmos_framework.utils import callback, distributed, log, misc
from cosmos_framework.utils.easy_io import easy_io
from cosmos_framework.utils.vfm.rand_state import get_rand_state_dict, set_rand_state_dict
from cosmos_framework.utils.generator.rand_state import get_rand_state_dict, set_rand_state_dict


class ModelWrapper(Stateful):
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -35,9 +35,9 @@ class ActivationCheckpointingConfig:
Read sites:

- MoT path consumes every field — see
cosmos_framework/model/vfm/mot/parallelize_unified_mot.py.
cosmos_framework/model/generator/mot/parallelize_unified_mot.py.
- VLM path consumes only ``mode`` (and only ``"full"`` enables
checkpointing) — see cosmos_framework/model/vfm/vlm_model.py.
checkpointing) — see cosmos_framework/model/generator/vlm_model.py.
"""

# AC mode:
Expand Down
2 changes: 1 addition & 1 deletion cosmos_framework/configs/base/defaults/model.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
from cosmos_framework.utils.lazy_config import LazyCall as L
from cosmos_framework.configs.base.defaults.model_config import OmniMoTModelConfig
from cosmos_framework.configs.base.defaults.parallelism import ParallelismConfig
from cosmos_framework.model.vfm.omni_mot_model import OmniMoTModel
from cosmos_framework.model.generator.omni_mot_model import OmniMoTModel

MOT_DDP_CONFIG = dict(
trainer=dict(
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -21,17 +21,17 @@

Original YAML reference target paths use the ``cosmos3._src.vfm.*`` namespace
(the OSS-release form of ``projects.cosmos3.vfm.*``); inside this released
tree the same modules live under ``cosmos_framework.data.vfm.*``.
tree the same modules live under ``cosmos_framework.data.generator.*``.
"""

from hydra.core.config_store import ConfigStore

from cosmos_framework.configs.base.defaults.reasoner import create_qwen2_tokenizer_with_download
from cosmos_framework.data.vfm.joint_dataloader import (
from cosmos_framework.data.generator.joint_dataloader import (
PackingDataLoader,
RankPartitionedDataLoader,
)
from cosmos_framework.data.vfm.local_datasets.sft_dataset import get_sft_dataset
from cosmos_framework.data.generator.local_datasets.sft_dataset import get_sft_dataset
from cosmos_framework.utils.lazy_config import LazyCall as L

# ---------------------------------------------------------------------------
Expand Down
2 changes: 1 addition & 1 deletion cosmos_framework/configs/base/defaults/optimizer.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
from cosmos_framework.utils.lazy_config import PLACEHOLDER
from cosmos_framework.utils.lazy_config import LazyCall as L
from cosmos_framework.utils.config_helper import ConfigStore
from cosmos_framework.utils.vfm.optimizer import build_lr_scheduler, build_optimizer
from cosmos_framework.utils.generator.optimizer import build_lr_scheduler, build_optimizer

OPTIMIZER_KWARGS: dict[str, Any] = dict(
# Learning rate for the optimizer.
Expand Down
Loading