Remove eval CLI references

Author: Xiangyu Lu

.agents/skills/cosmos3-codebase-nav/SKILL.md

@@ -95,7 +95,6 @@ Users can also supply a custom defaults file per-request via the `defaults_file`
 | ---------------------------------- | -------------------------------------------------------------------------------------------- |
 | Batch inference                    | `python -m cosmos_framework.scripts.inference`                                               |
 | Training                           | `python -m cosmos_framework.scripts.train --sft-toml=examples/toml/sft_config/<recipe>.toml` |
-| Action evaluation                  | `python -m cosmos_framework.scripts.eval`                                                    |
 | Online serving (Ray)               | `python -m cosmos_framework.inference.ray.serve`                                             |
 | Submit to Ray server               | `python -m cosmos_framework.inference.ray.submit`                                            |
 | Gradio UI                          | `python -m cosmos_framework.inference.ray.gradio`                                            |

AGENTS.md

@@ -7,7 +7,7 @@ Read this file first — it is the canonical map for navigating the Cosmos repos
 - **Training infrastructure** — top-level subpackages under `cosmos_framework/` (data, model, trainer, callbacks, checkpoint, …).
 - **Inference infrastructure** — `cosmos_framework/inference/` (Diffusers / Transformers / vLLM-friendly inference core, online serving via Ray + Gradio).
 - **Backend packages** — `packages/{diffusers,transformers,vllm}-cosmos3/` provide library-style shims that load Cosmos3 checkpoints into the respective ecosystems.
-- **Entry-point scripts** — `cosmos_framework/scripts/` (`train.py`, `inference.py`, `eval.py`, `export_model.py`, …) invoked as `python -m cosmos_framework.scripts.<name>`. Primary training entry point: `cosmos_framework.scripts.train` driven by a structured, pydantic-validated TOML interface (`--sft-toml=<recipe-toml>`); the schema lives at [`cosmos_framework/configs/toml_config/sft_config.py`](./cosmos_framework/configs/toml_config/sft_config.py) and the canonical recipe pattern is documented in [`examples/README.md`](./examples/README.md).
+- **Entry-point scripts** — `cosmos_framework/scripts/` (`train.py`, `inference.py`, `export_model.py`, …) invoked as `python -m cosmos_framework.scripts.<name>`. Primary training entry point: `cosmos_framework.scripts.train` driven by a structured, pydantic-validated TOML interface (`--sft-toml=<recipe-toml>`); the schema lives at [`cosmos_framework/configs/toml_config/sft_config.py`](./cosmos_framework/configs/toml_config/sft_config.py) and the canonical recipe pattern is documented in [`examples/README.md`](./examples/README.md).
 
 > All paths below are relative to the repository root (the directory containing `pyproject.toml`, the `cosmos_framework/` Python package, and `packages/`).
 

cosmos_framework/inference/_test.py

@@ -13,7 +13,6 @@
 from typing_extensions import Self
 
 from cosmos_framework.data.vfm.utils import VIDEO_RES_SIZE_INFO
-from cosmos_framework.inference.metrics import compute_action_mse, compute_psnr
 from cosmos_framework.inference.args import (
     IMAGE_ONLY_RESOLUTIONS,
     AspectRatio,
@@ -27,6 +26,7 @@
 from cosmos_framework.inference.common.checkpoints import CHECKPOINTS, DATASETS, DatasetConfig
 from cosmos_framework.inference.fixtures.args import MAX_GPUS
 from cosmos_framework.inference.fixtures.script import INPUT_DIR, ScriptConfig, ScriptRunner, script_test
+from cosmos_framework.inference.metrics import compute_action_mse, compute_psnr
 from cosmos_framework.utils.checkpoint_db import CheckpointConfig
 
 _CURRENT_DIR = Path(__file__).parent.absolute()
@@ -394,15 +394,6 @@ def __call__(self, runner: ScriptRunner, cfg: ScriptConfig) -> dict[str, str]:
             marks=marks,
             get_env=get_env,
         ),
-        ScriptConfig(
-            name=f"eval_{name}",
-            script=_TEST_DIR / "eval.sh",
-            levels=(0, 2),
-            gpus=(1, MAX_GPUS, MAX_GPUS),
-            marks=marks,
-            get_env=get_env,
-            after_script=_action_after_script,
-        ),
     ]
 
 

docs/code_structure.md

@@ -37,7 +37,7 @@ Cosmos/
 ├── cosmos_framework/             # Main package (training infra + inference subpackage)
 │   ├── inference/      # Inference subpackage (model, args, defaults, Ray serving, common helpers, SFT experiment configs)
 │   └── ...             # Training-infra subpackages: data, model, trainer, callbacks, checkpoint, …
-│   └── scripts/        # CLI entry-point scripts: train.py, _train.py, inference.py, eval.py, export_model.py, …
+│   └── scripts/        # CLI entry-point scripts: train.py, _train.py, inference.py, export_model.py, …
 ├── packages/           # Backend shim packages: diffusers-cosmos3, transformers-cosmos3, vllm-cosmos3
 ├── docs/               # User documentation (you are here)
 ├── docker/             # Dockerfiles for reproducible environments
@@ -196,7 +196,7 @@ Add new worker types as sibling subpackages — each owns its own startup, messa
 - `tests/` — pytest tests, mirroring the `cosmos_framework/` package layout.
 - `examples/` — runnable end-to-end examples; see `examples/README.md`.
 - `docker/` — Dockerfiles and image build helpers; see `docker/README.md`.
-- `cosmos_framework/scripts/` — CLI entry-point scripts (`train.py`, `inference.py`, `eval.py`, `export_model.py`, …); invoke as `python -m cosmos_framework.scripts.<name>`. Primary training entry point: `cosmos_framework.scripts.train` driven by a structured, pydantic-validated TOML interface (`--sft-toml=<recipe-toml>`); recipe TOMLs live under [`examples/toml/sft_config/`](../examples/toml/sft_config/) and the schema is defined in [`cosmos_framework/configs/toml_config/sft_config.py`](../cosmos_framework/configs/toml_config/sft_config.py) — see [`examples/README.md`](../examples/README.md) and [`docs/training.md`](./training.md).
+- `cosmos_framework/scripts/` — CLI entry-point scripts (`train.py`, `inference.py`, `export_model.py`, …); invoke as `python -m cosmos_framework.scripts.<name>`. Primary training entry point: `cosmos_framework.scripts.train` driven by a structured, pydantic-validated TOML interface (`--sft-toml=<recipe-toml>`); recipe TOMLs live under [`examples/toml/sft_config/`](../examples/toml/sft_config/) and the schema is defined in [`cosmos_framework/configs/toml_config/sft_config.py`](../cosmos_framework/configs/toml_config/sft_config.py) — see [`examples/README.md`](../examples/README.md) and [`docs/training.md`](./training.md).
 - `packages/` — library-style backend shims: `packages/{diffusers,transformers,vllm}-cosmos3/`, each installable independently.
 
 ## Where to Add New Code

docs/setup.md

@@ -216,7 +216,6 @@ If there is no attention backend wheel for your torch/cuda versions, you can bui
 Optional package extras:
 
 - `train`: Training infrastructure (FSDP, parallelism, checkpointing, datasets)
-- `eval`: Evaluation harnesses for trained checkpoints
 
 #### CUDA Variants