/open-video

open-video — autonomous director skill

v0.0.1 default for high-quality H3 clips is skill/h3-video
(Ollama for H3 + harness). Use this skill when the user wants multi-shot / long-film director behavior (plan → judge → stitch) beyond a single strong clip.

1. What open-video does

open-video is the autonomous director layer on top of open video models. It turns a concept into a finished film by running the loop no open engine ships natively:

plan → craft → validate → generate → judge → refine → stitch → deliver

• Model-agnostic core (core/): planner (coherence bible), crafter, validator, judge-loop, stitcher, selector, ref-pack builder.
• Pluggable backends (backends/<model>/): baseline = MiniMax H3 (#1 open model, Arena parity with closed, native stereo audio). Future: Wan2.2 (physics), LTX-2.3 (speed).
• Engine adapter (engines/comfyui/): drives ComfyUI via its HTTP API. open-video is the brain; ComfyUI is the hands.

It is NOT a video engine (ComfyUI is the engine) and NOT a model (H3/Wan/LTX are backends). It is the agent brain ComfyUI lacks: judge→refine + multi-shot stitch + coherence planning as they land.

v0.0.1 honesty: prefer skill/h3-video for reliable high-quality single clips. Multi-minute film is the flagship design. The judge is REAL when env-wired: set OPEN_VIDEO_VLM_URL + OPEN_VIDEO_VLM_MODEL (+ OPEN_VIDEO_VLM_KEY) to any OpenAI-compatible vision endpoint and every shot is scored + diagnosed automatically; with the env unset it is an honest PASS stub — then manual frame review is mandatory. Single open models still cap ~15s/shot; longer output needs multi-shot orchestration (partial).

2. Agentic procedure — run these steps in order

Step 1 — Understand the request. Classify the job: single shot vs multi-shot film; target duration; did the user supply reference image(s) / video / audio; desired aspect; quality bar. If the request is ambiguous and the target is a film >30s, ask ONE focused question (subject + mood + length). Do not generate a long film on guesswork.

Step 2 — Pick the mode. Mode is auto-derived from inputs (see backends/h3/backend.py and scripts/validate_prompt.py detect_mode):

• T2V / T2VA — text only, no image. Native audio + video.
• I2V / I2VA — one image (first frame). Instruction line: "For the target video, at 0.00 seconds into the target video, <Picture 1> (from [Shot 1]) is fully referenced."
• FL2VA — two images (first + last frame). Continuous interpolation. This is the multi-shot chain mode: previous shot's last frame → next shot's first frame for continuous handoff.
• L2VA — one image referenced at the final timestamp.
• R2V / Ref2VA — reference video/audio for identity / style / motion / voice. Tag refs <Picture 1>, <Video 1>, <Audio 1> and explicitly assign each a role.

Step 3 — Craft the 3-field prompt (per backends/h3/PROMPT_GRAMMAR.md). Exact structure:

[<instruction line>     # ONLY for I2V/FL2VA/L2VA — first line, then a blank line]

integrated_multimodal_description: [Shot 1] <style first>, <composition/subjects/scene>. <camera type + amplitude + speed>. [Shot 2] At 00:0X.XXX, the camera cuts to <next beat>.
overall_soundscape: <1–4 sentences: ambient / physical / non-verbal human sound>
non_diegetic_music: <1–3 sentences: instrumentation / tempo / rhythm / dynamics — NO mood words>

Hard rules: state style first in Shot 1 (Cinematic / live-action / 2D-animated / 3D CG / claymation / watercolor / vintage film); don't timestamp Shot 1; later shots use strictly increasing cut times within the duration; keep identity/wardrobe/color/objects/spatial relations consistent across shots; camera motion = natural prose combining type + amplitude + speed (Push In / Pull Out / Pan / Truck / Tilt / Arc / Tracking / Static / POV / Roll — omit amplitude/speed when medium/normal); dialogue as <d>[lang] verbatim words</d> with stable (S1)/(S2) speaker IDs (first appearance gives age/gender/on-screen/pitch/timbre/rate/accent); on-screen text in English double quotes, verbatim; every detail must be visible or audible — no abstract mood/emotion words; prefer camera motion over a cut for mere distance/angle changes.

Step 4 — Validate before generate (core/validator.py, hard gate — never skip). Checks: all 3 required fields present; duration within 4–15s; mode matches the instruction line; image/ref counts match the mode; cut times strictly increasing and within duration; <d>[lang]…</d> well-formed. Fix every issue before spending GPU. A shot that fails validation will fail the judge.

Step 5 — Generate via ComfyUI (backends/h3/backend.py + engines/comfyui/adapter.py). Confirm the server is up first (§3). H3 defaults: 1344×768, 20 steps, res_multistep / simple scheduler, shift_video=12.0 / shift_audio=3.0, INT8 ConvRot quants, engine flags --lowvram --use-sage-attention. length snaps to the 17k+5 grid.

Step 6 — Judge the output (core/judge.py). Activate the real judge with env: OPEN_VIDEO_VLM_URL + OPEN_VIDEO_VLM_MODEL (+ OPEN_VIDEO_VLM_KEY) — the pipeline then extracts frames and assesses vs prompt intent + quality bar automatically (QualityJudge.from_env() is the entry point; explicit QualityJudge(vision_fn=…) also works). Verdict: PASS / REFINE / FAIL, with score + issues in the receipt (run --json exposes them). With the env unset the judge auto-PASSes — then you must manually review frames.

Step 7 — Refine if REFINE or FAIL. Diagnose the specific issue, apply a targeted fix (prompt tweak / +steps / different mode / ref-pack for identity lock / different seed), regenerate. Strategy is refine-primary, not best-of-N: H3 raw quality is already at parity — the loop fixes adherence, length, and consistency, which is where the gap actually lives. Best-of-N is an optional escape hatch, not the default.

Step 8 — Stitch multi-shot (core/pipeline.py LongFilmPipeline.make_film). Each subsequent shot's first_frame = the previous shot's last frame (ffmpeg-extracted at -sseof -0.1); a t2v shot is auto-upgraded to i2v when a handoff frame exists. Then ffmpeg concat (-f concat -c copy)

• cross-shot audio continuity (music theme / dialogue language / ambient crossfade). Optional 2K upscale via API as a final step.

Step 9 — Deliver. One coherent film + per-shot receipts (prompt, seed, settings, judge verdict, extracted frames). Persist receipts under artifacts/verify/.

3. Key commands

Server — start ComfyUI first (the engine), on http://127.0.0.1:8188:

cd /path/to/ComfyUI && python main.py --lowvram --use-sage-attention   # H3 default flags

Health check: curl -s http://127.0.0.1:8188/system_stats (returns JSON when up). In Python: ComfyUIAdapter(server="http://127.0.0.1:8188").health().

Generate one shot — open-video Python API (the contract; lives in backends/ + engines/):

from open_video.backends.h3.backend import H3Backend
from open_video.core.backend import ShotRequest
from open_video.engines.comfyui.adapter import ComfyUIAdapter

engine = ComfyUIAdapter(server="http://127.0.0.1:8188")
backend = H3Backend()
req = ShotRequest(prompt=<3-field prompt string>, mode="t2v",
                  width=1344, height=768, duration_s=10.0, seed=0)
result = backend.generate(req, engine=engine)   # → ShotResult(ok, video_path, receipt)

• User-facing CLI (shipped): open-video "<concept>" --duration 10 --model h3 --json (durations above ~15s trigger the multi-shot v0 template path — single-shot ≤15s uses your prompt verbatim and is the reliable v0.0.1 route).
• Proven baseline scripts (same workflows, ported from the early lab):
  • Single shot: scripts/h3_agent.py --request "<simple NL>" --duration 5 --width 1344 --height 768 (use --prompt "<full 3-field prompt>" for best quality; --first-frame/--last-frame for I2V/FL2VA).
  • Validate: scripts/validate_prompt.py (exit 0 = clean, 1 = issues).

Multishot / long film — core/pipeline.py LongFilmPipeline.make_film(plan, out_path): plan is a list[Shot(scene_id, prompt, mode, duration_s, seed, …)]. The pipeline generates each shot → judges → extracts last frame → chains (FL2VA handoff) → stitches → writes output/film.mp4 and returns (film_path, plan_with_receipts).

• Proven baseline: scripts/h3_multishot.py --plan library/plans/multishot_demo.json --out output/long_demo.mp4 (a ready example plan ships at library/plans/multishot_demo.json; plan JSON = {"shots": [{"prompt_file": "...", "duration": 10, "first_frame": null}, …]}).

4. Constraints (H3 baseline — hard, evidence-based)

• Duration: 4–15s per shot (backends/h3/backend.py constraints() → duration_range_s: (4, 15)). No single shot >15s. For longer content, use multishot — that is the entire point of the stitcher.
• Frame grid: num_frames snaps to 17k+5 @ 24fps (video-VAE temporal constraint; _snap_17k5). Valid lengths: 5, 22, 39, 56, … Do not pass arbitrary frame counts.
• Resolution: local ceiling = 768 short edge (native canvas 768×1344), multiples of 32 (resolution_multiple: 32). 2K = API upscale only — never attempt 2K locally.
• Refs: ≤9 images, ≤3 videos, ≤3 audios, ≤12 total (max_refs).
• Quant — use INT8 ConvRot (minimax_h3_fl2va_pruned_int8_convrot.safetensors, ~21 GB, proven on RTX 5090). Avoid NVFP4 on RTX 5090 — ComfyUI issue #14157 bug (recorded in backends/h3/backend.py default_settings()["known_issues"] and docs/h3_ecosystem.md). Other quants: NF4 (~8 GB, lowest VRAM), W4 ConvRot (~10 GB, stock-compatible), BF16 (~62 GB, multi-GPU only).
• Audio: 32 kHz stereo native; CFG-distilled (no negative prompt, no guidance scale).
• Known issues to guard against (docs/h3_ecosystem.md "Known issues"): wide-shot face corruption (Comfy-Org #30), 2K upscale fails in ref2va (#19), ref2va persistent noise (HF #50), AMD/Apple Silicon partial support (#17/#24/#33), prompt metadata embedded in output files (#13).

5. Docs — read these for depth (do not guess from memory)

• README.md — what/why, three interfaces (App / CLI / Skill), the thesis.
• ARCHITECTURE.md — core / backends / engines / library layers; the quality loop; the long-film pipeline diagram.
• PLAN.md — phased roadmap, open decisions, the make-or-break success metric.
• backends/h3/PROMPT_GRAMMAR.md — the official H3 3-field prompt guide (condensed). Read before crafting any H3 prompt.
• docs/h3_ecosystem.md — quants, multishot tools, speed nodes, known issues, build-on targets.
• templates/model_backend.py — the plugin template (§6).
• CONTRIBUTING.md — plugin points + good-first-issues.

6. Adding a model backend (Wan2.2, LTX, …)

The core never changes — add a plugin. Copy templates/model_backend.py → backends/<model>/backend.py and implement the ModelBackend ABC (core/backend.py):

1. capabilities (Capabilities(...)) — which modes (t2v/i2v/flf2v/r2v), native_audio, max_duration_s, max_short_edge_px, strengths (the selector keys on these).
2. prompt_guide() + craft_prompt(intent, mode) — your model's prompt grammar.
3. constraints() — duration range, frame grid (or None), max_refs, resolution multiple.
4. generate(req, engine) + _build_workflow(req) — build the engine workflow (e.g. ComfyUI JSON loaded from workflows/), run via the engine adapter, return ShotResult(ok, video_path, receipt).
5. default_settings() — steps / sampler / scheduler / quant (evidence-based via bench/).
6. duration_to_length() / resolution_for() — model-specific frame/fps and resolution-grid math.

Then add backends/<model>/workflows/ (engine JSON), PROMPT_GRAMMAR.md, and __init__.py. See backends/h3/backend.py for a complete working example — the H3 backend is the reference implementation. PR it; CONTRIBUTING.md lists plugin points as good-first-issues.