Calliope

Calliope is a local-first story-to-video studio. You type a story idea; Calliope drafts a storyline with beats, characters, and locations, writes a screenplay-faithful per-scene script, then breaks each scene into shot clips and generates one video per clip by driving your own ComfyUI install. When the clips are done, one click stitches them into a finished film with crossfades and matched loudness (ffmpeg). Everything runs on your machine: projects live in SQLite, media lives in folders, and no cloud service is involved beyond the LLM endpoint you point it at.

Demo

From Script to Video Locally Using Calliope 1.5 https://youtu.be/JcGqt3DYn6c

Install — from source (npm + Python)

Prerequisites

• Python 3.11+
• Node.js 18+ (npm)
• A running ComfyUI install, with the models your workflows need already set up
• An OpenAI-compatible LLM endpoint — local (LM Studio, Ollama, etc.) or hosted
• ffmpeg on PATH — needed for film export

1. Backend (FastAPI)

cd calliope-backend
python -m venv .venv
.venv\Scripts\pip install -e ".[dev]"
.venv\Scripts\python -m calliope.main --host 127.0.0.1 --port 8247

Optionally copy calliope_config.example.json to calliope_config.json and edit it before starting (LLM endpoint, ComfyUI URL). You can also configure everything later in the app's Settings page. Never commit calliope_config.json — it stores your API key.

2. Frontend (SvelteKit)

cd calliope-web
npm install
npm run dev

Open http://127.0.0.1:5173. The dev server proxies /api to the backend on 127.0.0.1:8247.

First run

Open the app, go to Settings, and set:

1. LLM — one or more OpenAI-compatible endpoints (base URL, model name, API key). Save several, then pick which one is Active.
2. ComfyUI — the base URL of your running ComfyUI (e.g. http://127.0.0.1:8188)
3. Agent (optional) — assign a specific LLM per agent role (see below)

Leave Dry-run off — it is meant for testing and produces placeholder results instead of real generations.

Queue settings

In Settings → Queue you can tune how the worker talks to ComfyUI:

• Concurrency — how many jobs run at once.
• Poll interval — how often Calliope checks ComfyUI for a finished job.
• Poll timeout (seconds) — how long Calliope keeps waiting on ComfyUI for a single job before failing it. Default is 1800 (30 minutes). Long video generations can easily exceed 10 minutes, so raise this for heavy workflows — or set it to 0 to wait indefinitely (until the job finishes or you cancel it).
• Max retries — automatic retries before a job is marked failed.

Agent settings

In Settings → Agent:

• Model per agent — assign any saved LLM to each role: Main agent, Planner, Story agent, Script agent, Assets agent, Video agent. Blank means the Active LLM from Settings → LLM applies, so a single-endpoint setup needs no configuration here. A common setup is a strong cloud model for the main agent and planner, and a fast local model for the sub-agents. The Video agent's assignment also drives the MiniMax H3 prompt rewrite.
• System-prompt rules (hardening) — operator rules appended to every agent system prompt. Leave blank to disable.
• Agent shell (run_command) — off by default. When enabled, the agent may run simple commands (ffprobe, scripts you approve) inside the sandbox described below; anything non-read-only asks you first via a confirmation card.

Agent workspace

Settings → System Paths → Agent workspace (default DATA_DIR/workspace) is a scratch folder on disk the AI agent may work in. Every chat session gets its own subfolder (workspace/sessions/<session id>), shared between the main agent and its sub-agents. The agent uses four file tools — list_files, read_file, write_file, delete_file — to draft scripts, beat lists, shot plans, and video prompts as text files while it works, show you the drafts, and then commit the approved content into the project (scenes, clips, beats) with its normal editing tools. The files are the working medium; the database stays the store. Paths cannot escape the session folder, sensitive files (config, database, keys) are denied by name, and the optional shell, when enabled, is sandboxed to the same workspace.

Using the app

The app walks a project through four stages — Story, Assets, Script, Video:

• Story: describe your idea and Draft Storyline — this opens a project-linked chat in the Agents view with the prompt pre-filled, and the agent writes beats, characters, locations, and misc. items. Edit anything by hand before moving on.
• Assets: each character, location, and item has its own Image prompt. Pick a workflow and shared settings (width/height/etc.) at the top, then click Generate per entity to produce reference images on your ComfyUI. Regenerate any single entity without touching the others.
• Script: Regenerate Script also opens a project-linked Agents chat (pre-filled) to rewrite the per-scene script. Scenes preserve the full screenplay: complete action prose and verbatim dialogue. Then Break into shots splits a scene into its shot clips — an LLM coverage pass allocates every dialogue line and action beat across clips of ~5–10 seconds (a 2-minute dialogue scene becomes a dozen clips, not one). Scenes link back to the characters and locations from the Story stage.
• Video: each clip gets rendered by a Generate pass that queues a job on ComfyUI with the right prompt and reference images (plus optional video/audio file refs, if the workflow declares an (Input:video) node — see Video inputs are workflow-owned).
• Film view: once clips are rendered, Export film stitches them with ffmpeg: clips are normalized to 1080p at the majority frame rate of the clips themselves (24 fps clips export at 24 fps; mixed-rate projects conform to whichever rate most clips use), joined with 0.5s crossfades, and loudness-normalized into one final file.
• When everything is done the project is automatically marked Completed.

Playground is a free-form generation page outside the project pipeline: run any imported workflow with arbitrary inputs, upload your own files (image / video / audio) as inputs, and optionally attach a result to a project as an asset.

Build Scene is a browser-based shot composer for blocking out cinematic shots before generating AI images or video. One composition model (scene_json) drives both preview and export, and the Three.js viewport is the single surface: it previews the scene, captures stills, and exports video (one deterministic MediaRecorder pass of the object keyframes + camera track, then normalized server-side to H.264 MP4 via ffmpeg so the Playground "From Build Scene" picker always receives a real MP4). Camera framing is the user's job — keyed from the timeline — while the agent animates objects through the shot_* keyframe tools. There are no camera-beat tools and no ComfyUI in this surface; agent gates (Brief before mutate, Cut before export) keep the agent honest. Scenes are managed like AI Canvas chats: a left rail holds compositions bound to agent sessions, and the agent panel drives the same scene through built-in shot_* tools (no MCP).

Agents is a chat-driven way to run the same pipeline: talk to a production agent that operates Calliope through tools (create project, draft story, write script, queue asset/video renders, watch jobs). Every chat session is bound to at most one project — start a Sandbox chat with no project and the agent materializes one via create_project, linking the session automatically; or link a session to an existing project and ask for edits. Complex builds are decomposed by a planner into sub-agents (story → script → assets → video). Everything the agent does goes through the same database and render queue the project UI reads — nothing bypasses the normal pipeline. When the agent waits on renders (wait_for_jobs), it uses the same Poll timeout as the queue worker (default 30 minutes).

ComfyUI workflows (important)

Calliope does not hardcode Comfy node IDs. It discovers editable nodes from role tags in the node titles of an API Format workflow JSON. The example_ComfyUI_workflows/ folder in this repo contains ready-to-import examples (MiniMax H3 reference-to-video, krea2 text-to-image, character sheet).

1. Tag your nodes in ComfyUI

Rename the input/output nodes so their titles carry a role tag:

Display Name (Input:role)
Display Name (Output:role)

The display name can be anything, in any language. The :role part is the contract. Examples:

Main Prompt (Input:prompt)
Neg (Input:negative)
W (Input:width)
H (Input:height)
Char Ref (Input:character)
Env Ref (Input:location)
Result (Output:image)
Clip (Output:video)

2. Canonical roles

Input roles:

Role	Aliases	Filled by
prompt	positive	Entity Image prompt (Assets), scene/job prompt
negative	neg	Negative prompt when provided
width	w	Shared form / defaults
height	h	Shared form / defaults
character	char, portrait, sheet, face, ref	Character reference path
location	loc, environment, env, background, scene	Location reference path
image	img	Generic image input (ordered ref slot — see below)
video	vid	Video file input (LoadVideo)
audio	sound, sfx	Audio file input (LoadAudio)
seed	—	Shared form
duration	dur, length, seconds	Scene duration (video jobs)

Output roles:

Role	Aliases
image	img
video	vid

Unknown roles still show up in the dynamic form; they just get no special auto-fill. Plain (Input) / (Output) without a role still works through a deprecated label fallback, so old workflows keep working — but tag new workflows with explicit roles.

Prompt profiles

Each workflow has a Prompt format setting (default Plain prose). When set to MiniMax H3 reference (6-section), Calliope compiles each scene prompt into MiniMax H3's full-reference format (subject_definitions → summary → retention_analysis → detailed_description → overall_soundscape → non_diegetic_music, with <Subject N> labels and <d>[Language] …</d> dialogue). The compile is instant and deterministic — no LLM wait; the optional LLM rewrite runs only when you press Regenerate in the prompt preview. It is auto-suggested on import when the workflow contains a MiniMaxH3* node.

For multi-reference workflows, generic (Input:image) inputs are filled in node-id order — characters in scene order, then the location — and that order defines the <Subject N> numbering in the prompt. Keep your ref node ids in the order you want subjects numbered. A (Input:duration) node receives the scene's duration in seconds.

Using the H3 profile from the scene form (Generate clip): the form's auto-fill vs. override rule is simple — anything you type or pick wins over the automatic value.

• Text Prompt — leave it empty. An empty field gets the deterministic six-section H3 prompt built from the scene's action, dialogue, characters, and location. If you type anything, your text is sent verbatim — the compile never replaces it (so you can write directives like <video 1> — extend and continue: yourself).
• Ref 1 / Ref 2 — leave them on "Choose asset…" to auto-fill from the scene's characters (in scene order) then the location, with <Subject N> numbering matched to those slots. Picking an asset manually overrides just that slot (and you take over subject numbering for it).
• Duration auto-fills from the scene's estimated duration; edit it only when you want a different clip length.

Video inputs are workflow-owned (no auto-chaining)

Whether a render consumes a reference video is decided entirely by the ComfyUI workflow — a node tagged (Input:video) (a LoadVideo node). Calliope adds no per-scene "chain" flag and no clip-source picker: if your workflow has a video input, it appears in the Video-stage form like any other media slot, and you supply the video yourself — upload a file, or pick a rendered clip from the project via the asset picker. Leave it empty and the job fails in ComfyUI with its own error; Calliope never silently invents an input for you.

The workflow pattern (per kat3ri/ComfyUI-MiniMax-H3-Extend) is a LoadVideo (Input:video) node feeding the MiniMax H3 extend patched nodes (MiniMaxH3EncodeAVPatched → MiniMaxH3VideoExtendPatched) with the (Output:video) node at the end. Recommended starting settings from that repo: context_frames 2, ref_spacing 1–2, ref_decay 0.3, ref_ramp 3–4 (5–6 if the prior clip had heavy motion).

Review the prompt before you generate

Generate no longer fires blind. Hitting Generate first opens a prompt preview: the exact text that will land on the workflow's (Input:prompt) node — your typed form text or saved draft if there is one, otherwise the instant MiniMax H3 six-section compile (or the prose clip prompt: scene heading, the clip's shot description, and only the dialogue lines that clip performs). Resolving is instant; the LLM is never in the default path.

• Edit it inline — typos, camera notes, pacing, anything. The edited text is what gets sent.
• Regenerate explicitly asks the LLM for a fresh six-section rewrite (this is the only path that waits on a model — and it can also ground on the project's continuity ledger).
• Save draft keeps it on the clip; future generates (single or Generate all) reuse the draft. A hint appears when the draft predates changes to the scene or clip.
• Cancel aborts with nothing enqueued.

After a render, View prompt & inputs opens the scene's render history: every job as a chip, the payload each one actually sent to ComfyUI, and Copy settings to form to pull a past job's input values back into the live form.

Your video-stage setup (workflow choice, input values) auto-saves and comes back after a reload or app restart. Generate all honors every saved setup and draft — the toast reports how many drafts were used.

Better ComfyUI errors

When ComfyUI rejects a workflow, the job error now names the actual cause and node — e.g. ComfyUI rejected the workflow (400): prompt_outputs_failed_validation; node 12: Invalid audio file: "voice.m4a" — instead of a bare status code. Audio reference inputs upload to ComfyUI's flat input directory and work with both stock LoadAudio and VHS's VHS_LoadAudio.

3. Export the workflow

In ComfyUI, use Save (API Format) — not the regular UI workflow graph format. Calliope only understands API Format JSON.

4. Import into Calliope

Settings → Workflows → import the JSON → analyze → check the preview shows the expected role next to each input → save → enable the workflow where you want to use it (Assets, Playground, per scene).

5. Troubleshooting

If ComfyUI "doesn't know what to generate" or jobs come back empty:

• The workflow title must be literally (Input:prompt) — not only (Input) — for prompts to land reliably.
• Check the job payload: input_values for that node must be non-empty (blanks are stripped before submission).
• Make sure Dry-run is off in Settings (default is off).
• An unreachable ComfyUI fails the job honestly — Calliope never silently fakes images. Check the base URL and that ComfyUI is running.

6. HTTP only

Calliope talks to ComfyUI purely over its HTTP API: it uploads reference files with POST /upload/image, patches the workflow JSON and queues it via POST /prompt, polls /history/{prompt_id}, and downloads the results. It never reads or writes ComfyUI's local input/ / output/ folders — any folder paths stay configured on the ComfyUI side, not in Calliope.

calliope-cli — writing project content from a tool

The web UI is the only way to run renders, and it stays that way. But writing text — beats, cast, scenes, shot clips, continuity requirements — does not need a browser, and a 34k-character novel is a bad fit for a textarea. So there is a second door in, for scripts and AI agents:

calliope-cli.bat project create --title "灯笼" --idea-file novel.txt --json
calliope-cli.bat plan next --project 7 --json

32 commands across 10 groups (project story cast script clips context shots plan log schema). Run calliope-cli.bat --help, or schema show story append for the exact payload shape of any write.

Three things it will not do, with no flag to change that:

• Never delete. No delete verb exists; the web UI is the only path that removes a project, and it also removes that project's audit log.
• Never call a model. It writes text you give it. It does not summarise, expand, or "improve" anything.
• Never touch images or video. Those columns are read-only.

Because you edit in the UI while a tool works, every write takes --expect-hash and compares it inside the write transaction:

calliope-cli.bat project hash --project 7 --json
calliope-cli.bat story append --project 7 --file beats.json --expect-hash 4f2a9c1e8b7d6053

Refusing because the content moved is exit 3, distinct from exit 2 (your payload is wrong) — re-read and re-apply rather than retrying the same file. Every write is recorded in cli_audit with per-row before-images, readable via log list / log show.

An opencode skill lives at calliope-backend/skills_opencode/calliope/SKILL.md.

Multi-line text must go through --idea-file (or --file - for stdin). cmd.exe terminates a command line at a raw newline, so the argument after it is silently dropped and the command still exits 0.

License

This project is licensed under the MIT License

Repo layout

calliope-backend/            FastAPI backend (Python)
  src/calliope/authoring/    the only place SQL lives; shared with the web UI
  src/calliope/cli/          calliope-cli — no SQL, no shell, no model calls
  skills_opencode/           agent-facing skill for the CLI
calliope-web/                SvelteKit frontend
calliope-cli.bat             CLI entry point (Windows)
example_ComfyUI_workflows/   ready-to-import API-format workflow JSONs
docs/wiki/                   design notes (wiki source): ComfyUI HTTP vs MCP, multi-ref workflows