[new-model] Open-H3-Regenerate-2K: 768P H3 draft to 2560x1440, served on the OpenAI video routes

docs/inference/open_h3_regenerate_2k.md

@@ -0,0 +1,133 @@
+# Open-H3-Regenerate-2K
+
+`OpenH3Regenerate2KPipeline` regenerates a 768P [MiniMax-H3](https://huggingface.co/MiniMaxAI/MiniMax-H3) draft video
+(1344x768, 121 frames, 24 fps) at **2560x1440** with three refine steps. It is video to video: the draft is the input,
+and the output is a silent 2K MP4. FastVideo ports NVlabs'
+[Open-H3-Regenerate-2K](https://huggingface.co/Efficient-Large-Model/Open-H3-Regenerate-2K)
+([project page](https://nvlabs.github.io/Sana/Open-H3-Regenerate-2K/),
+[reference code](https://github.com/NVlabs/Sana/tree/sol-engine/models/Open-H3-Regenerate-2K)); the dense BF16
+`standard_fwd` path of that repo is the parity reference. The accelerated `sol_engine_fwd` path (Sol-Attn, MXFP8, CUDA
+graphs) and the fast 24-layer VAE decoder are not ported.
+
+```text
+draft video -> H3 VAE encode -> latent upscaler (2x, 3D conv) -> 3 Euler refine steps -> H3 VAE decode -> 2560x1440
+                                                 H3 DiT + merged LoRA + Sharp curve AdaLN branches
+```
+
+## What runs
+
+| Step | Details |
+|---|---|
+| Draft encode | `ffmpeg` resamples to 24 fps, cover-scales (bilinear) and centre-crops to 1344x768 (`draft_size: 768p`, or 1024x576 for `576p`), holds the last frame to 121 frames plus 3 copies. The FP32 H3 VAE posterior is sampled under seed 42. |
+| Latent upscale | The author's 3D-conv upscaler (`fastvideo/models/upsamplers/minimax_h3_latent_upscaler.py`), BF16, to the 160x90 latent grid. |
+| Refine | Euler on sigmas `[0.909375, 0.857143, 0.666667, 0]`, CFG 1, model time `1 - sigma`, initial state `(1 - 0.909375) * upscaled + 0.909375 * noise`. `num_inference_steps: 2` uses `[0.7, 0.494118, 0]` (softer, about 30% faster). |
+| Transformer | The base MiniMax-H3 DiT (the same module as the other H3 pipelines) with the released merged LoRA (FP32, rank 576 on 312 linears, scale 1) and 50 LMS Sharp curve AdaLN branches. |
+| Decode | The released H3 VAE; the output is trimmed to 121 frames. |
+
+The merged LoRA loads through the standard H3 community-LoRA path as a constructor-time adapter: the pipeline sets
+`lora_path` to `<model>/lora/open_h3_regenerate_2k_lora.safetensors` when you do not pass one. The Sharp branches take an
+8-D timestep curve (`lora/adaln_t_table.bin`, 1025x8) instead of the layer input, so they cannot be folded into weights;
+`sharp_curve.py` adds them to the AdaLN projection outputs with forward hooks.
+
+**Audio.** The reference feeds an encoded-silence audio stream to the DiT as conditioning and writes the video with no
+audio track (`-an`); it neither regenerates nor passes through the draft's audio. FastVideo does the same: the draft's
+soundtrack is ignored and the output MP4 has no audio.
+
+## Weights
+
+The release repo keeps the unmodified MiniMax-H3 components (a copy of `MiniMaxAI/MiniMax-H3` at `6818f6c`) under
+`base_model/` next to `lora/` and `latent_upsampler/`. Point `model_path` at the Hub id or at a local copy of that layout.
+`adaln_t_table.bin` is base-derived and is not redistributed in the release repo: put the 32,800-byte file (SHA-256
+`c02a6c11888297688c1e6278185ea1f947023acfc69f9003bbcdcec9a229a8e7`) at `lora/adaln_t_table.bin`. The reference
+`download.py` fetches it by byte range from `Comfy-Org/MiniMax-H3` at revision
+`e5eb578a89295337b8ff433a035929ce0279e0b6` (tensor `adaln_t_table` of
+`diffusion_models/minimax_h3_ref2va_pruned_int8_convrot.safetensors`) and verifies the hash.
+
+A GB200/B200-class GPU is needed (about 150 GB with the DiT, the 2K activations and the VAE).
+
+## Usage
+
+```bash
+python examples/inference/basic/basic_open_h3_regenerate_2k.py \
+    --model-path Efficient-Large-Model/Open-H3-Regenerate-2K \
+    --video draft_768p.mp4 --prompt "A red fox trots through fresh snow" --output outputs/regen2k
+```
+
+The prompt conditions the refine steps (use the draft's own prompt). Drafts from other video models are better encoded at
+576P (`--draft-size 576p`). `--steps 2` runs the two-step variant. The output is always 2560x1440; the first 121 frames of
+the draft are used.
+
+## Serving
+
+```bash
+fastvideo serve --config examples/serving/openai_open_h3_regenerate_2k.yaml
+```
+
+`examples/serving/openai_open_h3_regenerate_2k.yaml` loads the release on one GPU, sets `media_roots`, and pins the 2K
+defaults. The draft arrives as the typed `video_reference` of `POST /v1/videos/sync` (the response body is the MP4) or of
+`POST /v1/videos` (an async job; poll `GET /v1/videos/{id}` and download `GET /v1/videos/{id}/content`):
+
+```bash
+curl -sS -X POST http://localhost:8000/v1/videos/sync \
+  -H 'Content-Type: application/json' \
+  -d '{
+        "prompt": "A red fox trots through fresh snow",
+        "video_reference": {"video_url": "https://example.com/draft_768p.mp4"},
+        "seed": 1766580932
+      }' \
+  -o regen_2k.mp4 -D -
+```
+
+The response headers carry `X-Request-Id`, `X-Model`, `X-Inference-Time-S` and `X-Stage-Durations`. Rules:
+
+- Exactly one `video_reference` (or the legacy `video_path` / `video_url`) and no image or audio references.
+- `http(s)` URLs must resolve to public addresses unless `server.allow_private_media_urls` is set. A local path is accepted
+  only inside a directory listed in `server.media_roots`. `data:` URLs are refused for video references; this policy is
+  unchanged.
+- `size`, `width` and `height` may be omitted or `2560x1440`; `num_frames` may be omitted or 124 (the frame count of the
+  VAE grid; the video has 121 frames); `fps` must be 24. `seconds` is not accepted.
+- `num_inference_steps` is 3 (default) or 2; `seed` sets the refine noise.
+
+## Parity with the reference
+
+Measured on one GB200 (tray hpc-rack-1-5, BF16, dense attention) against the reference `standard_fwd` with the released
+VAE, same draft, seed (1766580932), prompt and VAE posterior seed (42). The reference's noise is a CUDA generator draw
+that FastVideo reproduces (the same `torch.Generator(device="cuda")` seed), so no injected noise was needed.
+
+| Stage / quantity | Fox draft (FastVideo H3 t2v, 768P) | Dance draft (576P, project page) |
+|---|---|---|
+| Draft latent, upscaled latent, DiT input | bit-identical | bit-identical |
+| One DiT forward (LoRA + Sharp), relative L2 | 1.1e-2 | 2.6e-2 |
+| Refined latent, relative L2 | 1.3e-2 | 3.7e-2 |
+| Decoded video PSNR / SSIM (21 frames) | 45.2 dB / 0.993 | 42.1 dB / 0.992 |
+
+The differences come from text-encoder rounding (relative L2 1e-3 on the prompt embeddings) and BF16 kernel choices;
+the reference reports the same order (1.4% relative L2 on the refined latent between its own runs).
+
+Warm timings, one GB200 (seconds):
+
+| | Upscale | Refine (3 steps) | VAE decode | Total |
+|---|---:|---:|---:|---:|
+| Reference `standard_fwd` | 0.87 | 71.6 | 15.0 | 87.5 |
+| FastVideo (end to end, incl. draft encode 5-10 s and decode to host) | 5.9-10.7 (encode + upscale + silence) | 69.5 | 25.0 | 107.5 |
+
+The first request builds nothing extra but loads weights (about 70 s once). The decode stage is slower than the
+reference because it also copies 121 frames to host memory and the MP4 encode is outside both columns.
+
+## Licences
+
+The release is not covered by one licence; its `README.md` says so, and the runtime code's licence does not replace the
+model licences. FastVideo only loads these weights and does not redistribute them.
+
+- The refiner LoRA, the fast H3 VAE decoder and `base_model/` (a copy of `MiniMaxAI/MiniMax-H3` at `6818f6c`: DiT, VAEs,
+  text encoder) follow the
+  [MiniMax H3 Community License](https://huggingface.co/MiniMaxAI/MiniMax-H3/blob/6818f6c32d12b210915e44ad56a4228c2608f160/LICENSE),
+  including its territory, use and redistribution terms (`base_model/LICENSE`, `NOTICE`). The Qwen3-VL text encoder's
+  Apache-2.0 text ships as `LICENSE-Qwen.txt`.
+- The latent upscaler follows its own licence:
+  [LBH-123-AI/Comfyui_Minimax_h3_latent_Upscaler](https://github.com/LBH-123-AI/Comfyui_Minimax_h3_latent_Upscaler)
+  (MIT, per the reference code) and its [weights](https://huggingface.co/LBH-123-AI/Minimax_h3_latent_Upscaler)
+  (`latent_upsampler/LICENSE`, Apache-2.0). `minimax_h3_latent_upscaler.py` is a port of that node's network.
+- The merged LoRA includes [LightX2V FL2VA Turbo](https://huggingface.co/lightx2v/Minimax-h3-Turbo) and
+  [LMS Sharp](https://huggingface.co/Alissonerdx/Minimax-H3-ComfyUI), which keep their own licences; the Sharp
+  `adaln_t_table` is read from [Comfy-Org/MiniMax-H3](https://huggingface.co/Comfy-Org/MiniMax-H3).

docs/inference/support_matrix.md

@@ -68,6 +68,7 @@ column links a runnable script in `examples/inference/basic/` where one exists.
 | matrixgame | `FastVideo/Matrix-Game-2.0-Base-Distilled-Diffusers`<br />`FastVideo/Matrix-Game-2.0-GTA-Distilled-Diffusers`<br />`FastVideo/Matrix-Game-2.0-TempleRun-Distilled-Diffusers`<br />`FastVideo/Matrix-Game-2.0-Base-Diffusers`<br />`FastVideo/Matrix-Game-2.0-GTA-Diffusers`<br />`FastVideo/Matrix-Game-2.0-TempleRun-Diffusers`<br />`mignonjia/mg_longtuning_distilled_zelda`<br />`mignonjia/mg_sf_distilled_zelda_1k_steps`<br />`mignonjia/mg_sf_distilled_zelda`<br />`mignonjia/mg_causal_zelda`<br />`mignonjia/mg_bidirectional_zelda` | I2V | [basic_matrixgame2.py](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_matrixgame2.py) |
 | matrixgame | `FastVideo/Matrix-Game-3.0-Base-Distilled-Diffusers` | I2V | [basic_matrixgame3.py](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_matrixgame3.py) |
 | minimax_h3 | `MiniMaxAI/MiniMax-H3` | T2V, I2V | [T2VA](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_minimax_h3_t2v.py)<br />[FL2VA](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_minimax_h3_fl2va.py)<br />[Ref2VA](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_minimax_h3_ref2va.py) |
+| open_h3_regenerate_2k | `Efficient-Large-Model/Open-H3-Regenerate-2K` | — | [basic_open_h3_regenerate_2k.py](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_open_h3_regenerate_2k.py) |
 | sd35 | `stabilityai/stable-diffusion-3.5-medium` | T2I | [basic_sd35_t2i.py](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_sd35_t2i.py) |
 | qwen_image21 | `Qwen/Qwen-Image-2.1` | T2I, I2I | [basic_qwen_image21.py](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_qwen_image21.py) |
 | stable_audio | `FastVideo/stable-audio-open-1.0-Diffusers` | T2V | [basic_stable_audio.py](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_stable_audio.py) |
@@ -121,6 +122,9 @@ and distilled) generate video with audio from text, optionally plus an image (se
 `kandinsky6_sr` IDs upscale an existing video (see the
 [Video SR guide](kandinsky6_sr.md)).
 
+**Note (Open-H3-Regenerate-2K)**: `open_h3_regenerate_2k` regenerates an existing 768P MiniMax-H3 draft video at
+2560x1440 (video to video, silent output); see the [Open-H3-Regenerate-2K guide](open_h3_regenerate_2k.md).
+
 **Note (LTX-2.5)**: T2V and I2V produce synchronized video and audio. The
 official `Lightricks/LTX-2.5` repository uses gated, split component files, so
 it must first be converted to FastVideo's component layout. See the

examples/inference/basic/basic_open_h3_regenerate_2k.py

@@ -0,0 +1,87 @@
+# SPDX-License-Identifier: Apache-2.0
+"""Regenerate a 768P MiniMax-H3 draft video at 2560x1440 with Open-H3-Regenerate-2K.
+
+The draft is VAE-encoded (the first 121 frames at 24 fps, 1344x768), its latent is upscaled, and three refine steps
+of the H3 transformer with the released LoRA run at 2560x1440. The output video is silent. See
+docs/inference/open_h3_regenerate_2k.md.
+
+    python examples/inference/basic/basic_open_h3_regenerate_2k.py --video draft.mp4 --prompt "..." --output out/
+"""
+
+from __future__ import annotations
+
+import argparse
+import time
+from pathlib import Path
+
+from fastvideo import VideoGenerator
+from fastvideo.api import (
+    ComponentConfig,
+    EngineConfig,
+    GenerationRequest,
+    GeneratorConfig,
+    InputConfig,
+    OffloadConfig,
+    OutputConfig,
+    PipelineSelection,
+    SamplingConfig,
+)
+
+DEFAULT_MODEL = "Efficient-Large-Model/Open-H3-Regenerate-2K"
+
+
+def parse_args() -> argparse.Namespace:
+    parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
+    parser.add_argument("--model-path", default=DEFAULT_MODEL, help="release repo id or a local copy of it")
+    parser.add_argument("--video", required=True, help="768P draft video (URL-free local path)")
+    parser.add_argument("--prompt", required=True, help="text the refine steps are conditioned on")
+    parser.add_argument("--output", type=Path, default=Path("outputs/open_h3_regenerate_2k"))
+    parser.add_argument("--steps", type=int, choices=(2, 3), default=3, help="refine steps (3 is the released setting)")
+    parser.add_argument("--seed", type=int, default=1766580932)
+    parser.add_argument("--attention-backend", default="FLASH_ATTN")
+    parser.add_argument("--draft-size", choices=("768p", "576p"), default="768p",
+                        help="size the draft is encoded at; 576p for drafts from other video models")
+    parser.add_argument("--runs", type=int, default=1, help="repeat the request to measure warm timings")
+    return parser.parse_args()
+
+
+def main() -> None:
+    args = parse_args()
+    args.output.mkdir(parents=True, exist_ok=True)
+    generator = VideoGenerator.from_config(
+        GeneratorConfig(
+            model_path=args.model_path,
+            engine=EngineConfig(num_gpus=1, offload=OffloadConfig(dit=False, dit_layerwise=False, text_encoder=True, vae=False)),
+            pipeline=PipelineSelection(
+                components=ComponentConfig(),
+                experimental={
+                    "attention_backend": args.attention_backend,
+                    "draft_size": args.draft_size,
+                },
+            ),
+        ))
+    try:
+        for run in range(args.runs):
+            started = time.perf_counter()
+            result = generator.generate(
+                GenerationRequest(
+                    prompt=args.prompt,
+                    inputs=InputConfig(video_path=args.video),
+                    sampling=SamplingConfig(height=1440,
+                                            width=2560,
+                                            num_frames=124,
+                                            fps=24,
+                                            num_inference_steps=args.steps,
+                                            guidance_scale=1.0,
+                                            seed=args.seed),
+                    output=OutputConfig(output_path=str(args.output / f"regen2k_{run}.mp4"),
+                                        save_video=True,
+                                        return_frames=False),
+                ))
+            print(f"run {run}: {time.perf_counter() - started:.1f}s -> {result.video_path}")
+    finally:
+        generator.shutdown()
+
+
+if __name__ == "__main__":
+    main()

examples/serving/openai_open_h3_regenerate_2k.yaml

@@ -0,0 +1,43 @@
+# OpenAI-compatible Open-H3-Regenerate-2K server (one GPU with about 150 GB free, e.g. B200 / GB200).
+# POST /v1/videos/sync (or /v1/videos) with a 768P H3 draft as video_reference; the response is the 2560x1440 MP4.
+# Local draft paths are accepted only inside server.media_roots; http(s) URLs must resolve to public addresses unless
+# server.allow_private_media_urls is set. data: URLs are always refused. See docs/inference/open_h3_regenerate_2k.md.
+generator:
+  model_path: Efficient-Large-Model/Open-H3-Regenerate-2K
+  engine:
+    num_gpus: 1
+    offload:
+      dit: false
+      dit_layerwise: false
+      text_encoder: true
+      vae: false
+      pin_cpu_memory: true
+    compile:
+      enabled: false
+      vae_enabled: false
+  pipeline:
+    experimental:
+      attention_backend: FLASH_ATTN
+      # 768p for MiniMax-H3 drafts; 576p encodes drafts from other video models at 1024x576.
+      draft_size: 768p
+
+server:
+  host: 0.0.0.0
+  port: 8000
+  output_dir: outputs/openai_open_h3_regenerate_2k
+  served_model_name: open-h3-regenerate-2k
+  media_roots:
+    - /data/drafts
+  media_reference_max_bytes: 536870912
+
+default_request:
+  sampling:
+    height: 1440
+    width: 2560
+    num_frames: 124
+    fps: 24
+    num_inference_steps: 3
+    guidance_scale: 1.0
+    seed: 1766580932
+  output:
+    return_frames: false

fastvideo/configs/pipelines/open_h3_regenerate_2k.py

@@ -0,0 +1,30 @@
+# SPDX-License-Identifier: Apache-2.0
+"""Pipeline configuration for Open-H3-Regenerate-2K (768P H3 draft -> 2560x1440)."""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+
+from fastvideo.configs.pipelines.minimax_h3 import MiniMaxH3PipelineConfig
+
+DRAFT_SIZE_CHOICES = ("768p", "576p")
+
+
+@dataclass
+class OpenH3Regenerate2KPipelineConfig(MiniMaxH3PipelineConfig):
+    """MiniMax-H3 policy plus the refine-stage knobs.
+
+    ``draft_size`` is the size the draft video is VAE-encoded at: 768p (1344x768) for MiniMax-H3 drafts, 576p
+    (1024x576) for drafts from other video models. ``refine_start_sigma`` re-noises the upscaled draft to another
+    level; the later steps then follow the shift-6 rule (None keeps the released grid).
+    """
+
+    draft_size: str = "768p"
+    refine_start_sigma: float | None = None
+
+    def check_pipeline_config(self) -> None:
+        super().check_pipeline_config()
+        if self.draft_size not in DRAFT_SIZE_CHOICES:
+            raise ValueError(f"draft_size must be one of {DRAFT_SIZE_CHOICES}, got {self.draft_size!r}.")
+        if self.refine_start_sigma is not None and not 0.0 < self.refine_start_sigma < 1.0:
+            raise ValueError(f"refine_start_sigma must be in (0, 1), got {self.refine_start_sigma}.")

fastvideo/entrypoints/openai/request_adapter.py

@@ -41,6 +41,10 @@ class RequestAdaptationError(ValueError):
     """The transport request cannot be represented by the loaded pipeline."""
 
 
+OPEN_H3_REGENERATE_2K_FAMILY = "open_h3_regenerate_2k"
+OPEN_H3_REGENERATE_2K_SIZE = (2560, 1440)
+OPEN_H3_REGENERATE_2K_NUM_FRAMES = 124
+
 _VIDEO_SUFFIXES = frozenset({".avi", ".m4v", ".mkv", ".mov", ".mp4", ".mpeg", ".mpg", ".webm"})
 
 
@@ -381,6 +385,13 @@ def _apply_reference_inputs(
     if request.task is not None and model_family != "minimax_h3":
         raise RequestAdaptationError("The task selector is only defined for MiniMax-H3 servers.")
 
+    if model_family == OPEN_H3_REGENERATE_2K_FAMILY:
+        if len(videos) != 1 or images or audios:
+            raise RequestAdaptationError("Open-H3-Regenerate-2K takes exactly one video_reference (the 768P draft) "
+                                         "and no image or audio references.")
+        kwargs["video_path"] = videos[0]
+        return
+
     if model_family == "minimax_h3" and (videos or audios):
         raise RequestAdaptationError("MiniMax-H3 video/audio references require a server configured with "
                                      "override_pipeline_cls_name=MiniMaxH3Ref2VAModularPipeline.")
@@ -522,6 +533,16 @@ def build_generation_request(
     if height is not None and (not isinstance(height, int) or height <= 0):
         raise RequestAdaptationError(f"height must be a positive integer, got {height!r}")
 
+    if model_family == OPEN_H3_REGENERATE_2K_FAMILY:
+        if (width, height) not in ((None, None), OPEN_H3_REGENERATE_2K_SIZE):
+            raise RequestAdaptationError("Open-H3-Regenerate-2K always renders 2560x1440; omit size/width/height "
+                                         f"or pass 2560x1440, got {width}x{height}.")
+        if kwargs.get("num_frames", OPEN_H3_REGENERATE_2K_NUM_FRAMES) != OPEN_H3_REGENERATE_2K_NUM_FRAMES:
+            raise RequestAdaptationError("Open-H3-Regenerate-2K regenerates the draft's first 121 frames (24 fps); "
+                                         "omit seconds and num_frames.")
+        if kwargs["fps"] != 24:
+            raise RequestAdaptationError(f"Open-H3-Regenerate-2K requires fps=24, got {kwargs['fps']}.")
+
     if model_family == "minimax_h3":
         from fastvideo.pipelines.basic.minimax_h3.packing import (
             MINIMAX_H3_CANVAS_MULTIPLE,