Rui Zhao1,* Kaiming Yang1,* Jifeng Zhu1,† Siyang Chen1,† Ziqi Wang1
Weijia Wu1 Kevin Qinghong Lin2 Heng Wang3 Mike Zheng Shou1,‡
1Show Lab, National University of Singapore 2University of Oxford 3Tencent
*Equal contribution †Equal contribution (second authors) ‡Corresponding author
Turn a generated manipulation video into a 3D robot trajectory, execute it in simulation, and measure whether the imagined task actually works.
- 🎬 Video → trajectory → execution. One reproducible pipeline from a generated video to robot action and task-success evaluation.
- 🧪 101 frozen tasks. Curated RoboCasa scenes with fixed state, camera, first-frame generation input, protocol, and ground-truth references.
- 🤖 Model-agnostic evaluation. Import one video or a complete model batch without changing benchmark data or pipeline code.
- 🔌 Replaceable model backends. Swap the video generator, VLM judge, detector, segmenter, tracker, depth model, or pose model without changing the benchmark pipeline.
- 📦 Reproducible release. Relative-path workspace configuration, pinned providers, reviewed dependency patches, immutable inputs, and exact resume.
- 2026-09 - Code release: Explore the Dream.exe codebase, with an end-to-end pipeline for video-to-trajectory extraction, robot execution, and evaluation. Start with the bundled quickstart example, then follow the installation and benchmark guides to reproduce the full workflow.
- 2026-09 - Benchmark & released models: Find our 101-task Dream.exe benchmark, fine-tuned DVD depth models, and Wan2.2 image-to-video LoRA checkpoints on Hugging Face. The video release includes paired 2K and 7K high/low noise adapters; download and run them with the upstream Wan2.2 I2V A14B base model.
- 2026-06 - Accepted as a Spotlight at the ICML 2026 FoGen Workshop. 🎉
The 101 tasks cover single-object manipulation, multi-object interaction, and long-horizon multi-stage tasks. Generated videos are evaluation subjects; execution videos, extracted trajectories, actions, and metrics are stored only as experiment outputs. Ground-truth video/action/depth remain immutable references in the benchmark.
- 🎬 Generate & track. Start from a scene image and task prompt, then track the end-effector and manipulated object throughout the generated video.
- 📐 Lift into 3D. Our LoRA-adapted DVD estimates depth, calibrated with the initial scene depth; camera parameters lift tracked pixels into world coordinates.
- 🦾 Recover actions. Calibrate the end-effector trajectory to the robot's control point, estimate rotation, and infer grasp/release timing from end-effector–object motion.
- ✅ Execute & evaluate. Combine position, rotation, and gripper state into a 7D trajectory, then execute it with closed-loop control in simulation to measure task success.
See Section 3.3 and Appendix C of our paper for details.
Getting started has two deliberately separate steps: install everything needed once, then run one benchmark case that is already in the repository. The full 101-case benchmark is not needed for this path.
Follow INSTALL.md from top to bottom. It installs the main Conda environment, the default execution providers, the required public model assets, and the Dream.exe DVD checkpoint used by the bundled case. The guide also separates these requirements from full-benchmark data and optional models.
The default depth model is our DVD LoRA fine-tuned on Dream.exe benchmark data. The installation downloads only the bundled case's checkpoint, alongside the required shared model dependencies.
The case data and released Kling 3.0 input video are already included through Git LFS. You do not edit a workspace or download the full benchmark before the first run.
From the repository root, run one command:
dream-exe run \
--workspace examples/quickstart/workspace.json \
--spec examples/quickstart/run.jsonThis executes video2traj → action → simulation → evaluation for
rc_cheesybread_ep000001 / Kling3.0 / standard. A successful run prints a
completed run report and writes only below .dream-exe/quickstart/; the bundled
benchmark inputs remain unchanged.
If the command stops during preflight, use dream-exe doctor --workspace examples/quickstart/workspace.json --case rc_cheesybread_ep000001 to identify a
missing provider, checkpoint, or Git LFS file. The
bundled-case guide explains the output tree,
the three-input comparison, saved metrics, and troubleshooting.
The following paths are alternatives after the bundled case; they are not a single sequence that every user must complete.
| Goal | Additional requirement | Continue with |
|---|---|---|
| Run one or all 101 benchmark cases | Download the full benchmark data; add released result inputs only for exact reproduction | Benchmark and reproduction |
| Evaluate videos from your Video Generation Model | Use the included Wan2.2 adapter, our released Wan2.2 LoRAs, import MP4s, or connect a hosted/local generator; preprocessing is automatic | Video generation models |
| Evaluate saved outputs | A completed run; VLM scoring additionally needs local credentials | Evaluation and VLM example |
| Select official DVD, a Dream.exe LoRA, or another model | Install only assets not already required, then select the model explicitly | Model choices and custom backends |
| Understand or modify the implementation | No additional data | Code structure and reading paths |
Each file answers one question. Detailed keys and ownership rules live in the configuration reference.
| File | Question it answers |
|---|---|
workspace.json |
Where are benchmark inputs, published results, providers, checkpoints, work, and outputs? |
run.json |
Which cases, input videos, stages, seed, and destination should this experiment run? |
credentials.local.json |
Which private endpoint and API key should VLM evaluation use? |
models.json |
Which caller-provided model implementations are available? |
runtime.json |
Which optional exec implementations should replace the defaults for this run? |
The bundled case uses its shipped workspace and run spec unchanged. Downloaded benchmark protocol files are immutable reproduction inputs, not user configuration.
@article{zhao2026dreamexe,
title = {Dream.exe: Can Video Generation Models Dream Executable Robot Manipulation?},
author = {Zhao, Rui and Yang, Kaiming and Zhu, Jifeng and Chen, Siyang and Wang, Ziqi and Wu, Weijia and Lin, Kevin Qinghong and Wang, Heng and Shou, Mike Zheng},
journal = {arXiv preprint arXiv:2606.04811},
year = {2026}
}Dream.exe is released under the Apache 2.0 License. Benchmark assets, external providers, and model checkpoints may have separate terms; review THIRD_PARTY.md before redistribution.
We thank the RoboCasa / RoboCasa365 team for releasing robot demonstration datasets, kitchen environments, and simulation assets that form the foundation of the Dream.exe benchmark.