Cover every G0 and G1 detection variant in the benchmark harness

README.md

@@ -6,6 +6,33 @@ The harness records the exact LibreYOLO version and commit in each emitted
 result JSON. For public submissions, validate the result JSON rather than
 assuming the local editable install points at the intended branch.
 
+## G0/G1 detection campaign
+
+The harness covers every detect size in LibreYOLO's G0/G1 registry. For the
+current 60-variant inventory, pin LibreYOLO to
+`6a0ccc3a0e579948011d5a55200309c4082e61e7`; v1.5.0 predates TinyFormer.
+
+```bash
+va-bench list --groups g0 g1
+va-bench run --groups g0 g1 --coco-dir /path/to/coco-mini500 --device cuda
+```
+
+Selection reads the installed library's groups and detect sizes, and fails
+if the harness is missing any of them. Other tasks are not selected. A model
+failure is reported, the remaining models still run, and the command exits
+nonzero so a partial sweep cannot be mistaken for a complete one.
+
+Registration is separate from checkpoint availability. YOLO9-P2 t/s have no
+published COCO checkpoints; their VisDrone preview is not a COCO replacement.
+They require locally COCO-trained `LibreYOLO9P2t.pt` / `LibreYOLO9P2s.pt` files
+via `--weights-dir`. With PyTorch this directory must contain each selected
+checkpoint; the harness rejects wrong families, sizes, tasks and COCO labels.
+No random-weight benchmark is published in place of a missing checkpoint.
+
+See [the campaign inventory](docs/g0-g1-campaign.md) for the full list, the
+58-model publicly downloadable batch, version pins and verification limits.
+New P2/TinyFormer entries cover COCO inference, not RF100-VL training recipes.
+
 ## Reproducibility
 
 Every emitted JSON carries a `repro` block so a third party can reproduce the
@@ -59,7 +86,7 @@ The registry covers 67 open LibreYOLO detection variants:
 | PicoDet | 3 | Yes | Yes |  |
 | EC / EdgeCrafter | 4 | Yes | Yes |  |
 | RTMDet | 5 | Yes | Yes |  |
-| YOLO-NAS | 3 | Yes | Yes | LibreYOLO's open retrained weights. |
+| YOLO-NAS | 3 | Yes | Yes | Deci checkpoints via the library's checksum-verified CDN path; original weight terms apply. |
 
 DAMO-YOLO is not registered.
 

docs/g0-g1-campaign.md

@@ -0,0 +1,100 @@
+# G0/G1 detection campaign
+
+Inventory source: LibreYOLO `6a0ccc3a0e579948011d5a55200309c4082e61e7` (dev, includes TinyFormer).
+Groups classify families; this campaign selects the detect task only.
+
+60 variants across 13 families. 58 have public checkpoint endpoints; P2 t/s need locally COCO-trained checkpoints.
+
+| Group | Family | Sizes | COCO checkpoint |
+| --- | --- | --- | --- |
+| g0 | yolo9 | t, s, m, c | Public download endpoint verified |
+| g0 | rfdetr | n, s, m, l | Public download endpoint verified |
+| g1 | yolo9_e2e | t, s, m, c | Public download endpoint verified |
+| g1 | yolo9_p2 | t, s | Local trained weights required |
+| g1 | ec | s, m, l, x | Public download endpoint verified |
+| g1 | rtdetr | r18, r34, r50, r50m, r101, l, x | Public download endpoint verified |
+| g1 | rtdetrv2 | r18, r34, r50, r50m, r101 | Public download endpoint verified |
+| g1 | rtdetrv4 | s, m, l, x | Public download endpoint verified |
+| g1 | dfine | n, s, m, l, x | Public download endpoint verified |
+| g1 | deim | n, s, m, l, x | Public download endpoint verified |
+| g1 | deimv2 | atto, femto, pico, n, s, m, l, x | Public download endpoint verified |
+| g1 | tinyformer | s, m, l, x, xl | Public download endpoint verified |
+| g1 | yolonas | s, m, l | Public download endpoint verified |
+
+## Setup
+
+Pin the library, not a moving dev branch. v1.5.0 does not include TinyFormer.
+
+```bash
+python -m pip install "libreyolo @ git+https://github.com/LibreYOLO/libreyolo.git@6a0ccc3a0e579948011d5a55200309c4082e61e7"
+va-bench list --groups g0 g1
+```
+
+Use a clean harness checkout containing this change and record its exact commit.
+Keep the same PyTorch/CUDA environment for the whole Spark campaign. Start by
+repeating YOLO9-T and RF-DETR-S so the new library pin has anchor measurements.
+
+## Public checkpoint batch (58 variants)
+
+```bash
+va-bench run --models \
+  yolov9t yolov9s yolov9m yolov9c rfdetr-n rfdetr-s rfdetr-m rfdetr-l \
+  yolov9e2e-t yolov9e2e-s yolov9e2e-m yolov9e2e-c ec-s ec-m ec-l ec-x \
+  rtdetr-r18 rtdetr-r34 rtdetr-r50 rtdetr-r50m rtdetr-r101 rtdetr-l rtdetr-x rtdetrv2-r18 \
+  rtdetrv2-r34 rtdetrv2-r50 rtdetrv2-r50m rtdetrv2-r101 rtdetrv4-s rtdetrv4-m rtdetrv4-l rtdetrv4-x \
+  dfine-n dfine-s dfine-m dfine-l dfine-x deim-n deim-s deim-m \
+  deim-l deim-x deimv2-atto deimv2-femto deimv2-pico deimv2-n deimv2-s deimv2-m \
+  deimv2-l deimv2-x tinyformer-s tinyformer-m tinyformer-l tinyformer-x tinyformer-xl yolonas-s \
+  yolonas-m yolonas-l \
+  --coco-dir /path/to/coco-mini500 --output-dir results/spark-g0-g1 \
+  --device cuda --format pytorch \
+  --dataset-id LibreYOLO/coco-val2017-mini500 \
+  --dataset-revision 6c5d8d10dab92cf6f2cd476e1d5509ab5786cb88
+```
+
+No --limit. Keep conf=0.001, IoU=0.6, max_det=300 and each model's native input
+size. Run on an otherwise idle Spark. Inspect every model failure; a sweep exits
+nonzero if any model failed, even though remaining models are attempted.
+
+## Full inventory including P2
+
+```bash
+va-bench run --groups g0 g1 --coco-dir /path/to/coco-mini500 --device cuda
+```
+
+This requests all 60 variants and explicitly reports P2 missing-checkpoint errors
+unless its canonical files exist locally. For a separate P2 run:
+
+```bash
+va-bench run --models yolov9p2-t yolov9p2-s --weights-dir /path/to/coco-trained-p2 \
+  --coco-dir /path/to/coco-mini500 --device cuda
+```
+
+Do not use the 10-class VisDrone preview or transferred/randomly initialized P2
+heads as COCO benchmarks. The harness checks checkpoint family, detect task,
+size, 80-class count, and COCO class names/order. Training evidence for supplied
+weights remains the contributor's responsibility.
+
+## Evidence and limits
+
+- All 60 native variants passed CPU preprocessing, forward, and postprocessing
+  at native input resolution with initialized weights on this pinned library.
+- Published YOLO9-E2E-T and TinyFormer-S checkpoints completed the harness on
+  two actual COCO images, including evaluation and weight hashing. Smoke outputs
+  were not submitted as benchmark measurements.
+- Public Hugging Face checkpoint endpoints were checked for 55 variants and
+  the library's Deci CDN endpoints for the three YOLO-NAS variants.
+- Full COCO accuracy, all 58 published checkpoint loads, and Spark/CUDA execution
+  have not been independently reproduced here. These are the contributor run.
+- YOLO-NAS checkpoints retain Deci's weight terms. TinyFormer checkpoints have
+  Apache-2.0 plus DINOv3 terms. The harness links existing library downloads; it
+  does not redistribute weights or relabel their licenses.
+- Registration of P2/TinyFormer is for inference; no RF100-VL fine-tuning recipe
+  is claimed by this change.
+
+## Submission
+
+The corresponding website catalogue change must land before new-family JSONs
+validate. Keep harness JSONs unchanged, with hashes and commands intact. Full
+COCO accuracy references remain separate from mini500 latency runs; never splice
+metrics from different runs without a separate provenance contract.

tests/test_groups.py

@@ -0,0 +1,82 @@
+"""Group selection, local weight routing, and incomplete sweep failures."""
+
+from types import SimpleNamespace
+
+import pytest
+
+from va_bench import benchmark, cli, models
+
+
+def test_group_selection_uses_canonical_family_and_every_size(monkeypatch):
+    required = {("yolo9", "t"), ("yolo9_p2", "s"), ("tinyformer", "xl")}
+    monkeypatch.setattr(models, "_required_group_pairs", lambda groups: required)
+    assert models.list_models(["g0", "g1"]) == ["tinyformer-xl", "yolov9p2-s", "yolov9t"]
+
+
+def test_group_selection_rejects_silent_coverage_gaps(monkeypatch):
+    monkeypatch.setattr(models, "_required_group_pairs", lambda groups: {("tinyformer", "future")})
+    with pytest.raises(ValueError, match="future"):
+        models.list_models(["g1"])
+
+
+def test_run_selection_is_mutually_exclusive():
+    with pytest.raises(SystemExit) as exc:
+        cli.main(["run", "--groups", "g0", "--all", "--coco-dir", "unused"])
+    assert exc.value.code == 2
+
+
+def test_partial_sweep_exits_nonzero_but_finishes_other_models(monkeypatch, tmp_path):
+    calls = []
+    saved = []
+
+    def run(**kwargs):
+        calls.append(kwargs["model_key"])
+        if kwargs["model_key"] == "yolov9t":
+            raise RuntimeError("bad checkpoint")
+        return {}
+
+    monkeypatch.setattr(benchmark, "benchmark_model", run)
+    monkeypatch.setattr("va_bench.output.save_result", lambda result, output: saved.append(result))
+    with pytest.raises(SystemExit) as exc:
+        cli.main(["run", "--models", "yolov9t", "rfdetr-s", "--coco-dir", str(tmp_path)])
+    assert exc.value.code == 1
+    assert calls == ["yolov9t", "rfdetr-s"]
+    assert len(saved) == 1
+
+
+def test_local_checkpoint_is_used_and_coco_identity_checked(monkeypatch, tmp_path):
+    from libreyolo.utils.general import COCO_CLASSES
+
+    names = dict(enumerate(COCO_CLASSES))
+    path = tmp_path / "LibreYOLO9P2t.pt"
+    path.write_bytes(b"fixture")
+    calls = []
+    fake = SimpleNamespace(FAMILY="yolo9_p2", task="detect", size="t", nb_classes=80, names=names)
+
+    def load(**kwargs):
+        calls.append(kwargs)
+        return fake
+
+    monkeypatch.setattr("libreyolo.LibreYOLO", load)
+    assert models.load_model("yolov9p2-t", "cpu", tmp_path)[0] is fake
+    assert calls[0]["model_path"] == str(path)
+    fake.nb_classes = 10
+    with pytest.raises(ValueError, match="80 classes"):
+        models.load_model("yolov9p2-t", "cpu", tmp_path)
+    fake.nb_classes = 80
+    fake.names = dict(reversed(list(names.items())))
+    fake.names[0] = "not-person"
+    with pytest.raises(ValueError, match="class names/order"):
+        models.load_model("yolov9p2-t", "cpu", tmp_path)
+
+
+def test_missing_p2_does_not_load_random_or_visdrone_weights(monkeypatch, tmp_path):
+    monkeypatch.chdir(tmp_path)
+    with pytest.raises(FileNotFoundError, match="no published COCO checkpoint"):
+        models.load_model("yolov9p2-t")
+
+
+def test_pytorch_dispatch_forwards_local_weight_directory(monkeypatch, tmp_path):
+    monkeypatch.setattr(benchmark, "_benchmark_pytorch", lambda *args, **kwargs: kwargs)
+    result = benchmark.benchmark_model("yolov9p2-t", tmp_path, weights_dir=tmp_path)
+    assert result["weights_dir"] == tmp_path

tests/test_models.py

@@ -35,7 +35,7 @@ def test_registry_has_expected_models():
         assert any(k.startswith(fam) for k in keys)
     assert {"yolonas-s", "yolonas-m", "yolonas-l"} <= set(keys)
     assert not any(k.startswith("damoyolo-") for k in keys)
-    assert len(keys) == 67
+    assert len(keys) == 74
     assert Counter(spec.family for spec in MODEL_REGISTRY.values()) == {
         "deim": 5,
         "deimv2": 8,
@@ -50,6 +50,8 @@ def test_registry_has_expected_models():
         "yolonas": 3,
         "yolov9": 4,
         "yolov9-e2e": 4,
+        "yolov9-p2": 2,
+        "tinyformer": 5,
         "yolox": 6,
     }
 

tests/test_rf100vl_train.py

@@ -78,7 +78,8 @@ def test_generate_data_yaml_uses_all_splits_and_sorted_category_ids(tmp_path):
 
 
 def test_every_campaign_family_has_a_protocol_recipe():
-    families = set(list_families()) - {"yolov9-e2e"}
+    # COCO inference registration does not promise an RF100-VL training recipe.
+    families = set(list_families()) - {"yolov9-e2e", "yolov9-p2", "tinyformer"}
     for family in families:
         recipe = rf100vl_train.load_recipe(
             rf100vl_train.recipe_path_for_family(family),

va_bench/benchmark.py

@@ -28,6 +28,7 @@
 from .hardware import collect_all as collect_hw
 from .hardware import get_runtime_device_name
 from .models import (
+    G0_G1_REFERENCE_COMMIT,
     check_params,
     count_onnx_params,
     get_spec,
@@ -127,7 +128,7 @@
     90,
 ]
 
-SUPPORTED_LIBREYOLO_COMMIT = "24143e52339d71570ae3207ee60ada202a731200"
+SUPPORTED_LIBREYOLO_COMMIT = G0_G1_REFERENCE_COMMIT
 REQUIRED_PYTORCH_MODEL_API = (
     "_get_input_size",
     "_preprocess",
@@ -160,7 +161,8 @@ def benchmark_model(
         fmt: Backend format: "pytorch" (default), "onnx", or "tensorrt".
         weights_dir: Directory containing user-supplied ONNX files (.onnx)
             or TensorRT engines (.engine + .engine.json sidecar).
-            Required when fmt="onnx" or fmt="tensorrt", ignored otherwise.
+            Required when fmt="onnx" or fmt="tensorrt"; optional local .pt
+            checkpoint directory for PyTorch.
         device: Device string ("auto", "cuda", "mps", "cpu").
         conf: Confidence threshold for predictions.
         iou: IoU threshold for NMS.
@@ -184,6 +186,7 @@ def benchmark_model(
             verbose,
             dataset_id=dataset_id,
             dataset_revision=dataset_revision,
+            weights_dir=weights_dir,
         )
     if fmt == "onnx":
         if weights_dir is None:
@@ -380,6 +383,7 @@ def _benchmark_pytorch(
     verbose: bool,
     dataset_id: str | None = None,
     dataset_revision: str | None = None,
+    weights_dir: str | Path | None = None,
 ) -> dict[str, Any]:
     coco_dir = Path(coco_dir)
 
@@ -389,7 +393,7 @@ def _benchmark_pytorch(
         print(f"Benchmarking: {spec.display_name} ({spec.key}) [PyTorch]")
         print(f"{'=' * 70}")
 
-    model, _ = load_model(model_key, device=device)
+    model, _ = load_model(model_key, device=device, weights_dir=weights_dir)
     _assert_supported_pytorch_model_api(model)
     actual_device = model.device
     imgsz = model._get_input_size()
@@ -518,7 +522,7 @@ def _benchmark_pytorch(
         weights=build_weights_repro(
             weight_file=spec.weight_file,
             resolved_path=getattr(model, "model_path", None),
-            source="libreyolo-managed",
+            source="user-supplied" if weights_dir else "libreyolo-managed",
         ),
     )
     return assemble_result(