Give a frame-sampling model its sense of motion back.
A multimodal model "watches" a video by looking at frames sampled out of it — usually about one per second. It sees the stills, but not the motion between them. Anything fast that happens between two samples simply doesn't exist for it. Phinomena recovers that lost motion with optical flow and renders it back into images the model can already see.
The name is phi + phenomena. The phi phenomenon (Wertheimer, 1912) is the formal name for the apparent motion your brain constructs from a sequence of still images — the effect that makes film out of a filmstrip. That constructed sense of motion is exactly the faculty a frame-sampling model is missing. It gets the stills; it never gets the phi. Phinomena hands it back.
what the model normally gets what Phinomena adds
(1 fps stills — motion falls in the gaps) (motion recovered + rendered)
t=2s ┌─────────┐ t=3s ┌─────────┐ ┌───────────────────────────┐
│ • │ │ • │ │ ███▶ a hot streak across │
│ │ │ │ ──▶ │ the frame: the fast │
│ (dot │ │ (dot │ │ event, made visible │
│ parked)│ │ parked)│ │ ░ slow blob = background │
└─────────┘ └─────────┘ └───────────────────────────┘
the dot darted across and back at t≈2.5s peak speed the stills imply:
between these two stills — invisible here 0.05 widths/s
peak speed actually present:
3.42 widths/s (63× more)
Those numbers are the real output of the bundled demo (see The demo).
Optical flow answers "where did each pixel go between these two frames?" as a
field of (dx, dy) vectors. That field is the motion the stills couldn't
show. Phinomena uses Farnebäck dense optical flow (cv2.calcOpticalFlowFarneback)
— "dense" meaning a vector for every pixel, so the whole moving thing shows
up, not just a handful of tracked corners.
To make a flow field viewable, it is rendered as an HSV image: the hue
of each pixel encodes the direction it moved, and the brightness encodes
how fast. A color key (legend_flow_wheel.png) ships with every run so the
mapping is self-documenting.
There's a chicken-and-egg trap. If you only look at 1 fps stills, you not only miss a fast event — you have no idea it happened, so you don't know to look closer. Self-directed "zoom into the interesting moment" is impossible if your search signal is as blind as your final view.
Phinomena breaks the loop by separating detection from rendering:
- Detection is cheap and dense. Frames are downscaled hard (default 160 px wide) and a motion score is computed for (nearly) every source frame. These frames are tiny, so running the detector on all of them costs little — and because it's dense in time, it sees the fast event the coarse stills skip.
- Rendering is expensive and sparse. Full-resolution Farnebäck flow and HSV rendering run only inside the time windows detection flags.
So the search signal has high temporal resolution (it can find the spike) while the costly visualization is reserved for where it matters. That's the whole trick that makes self-directed sampling actually work.
The obvious motion statistic — mean flow magnitude over the frame — is a trap: it's area-weighted, so a small fast object barely nudges the average while a big slow blob dominates. Phinomena's detector instead scores each step by the fastest moving region (the max of a lightly-blurred magnitude field). A small object screaming across the frame is exactly what that surfaces.
For the cross-rate blind-spot comparison, raw magnitudes also aren't comparable: a 1 fps frame pair spans a whole second, a 30 fps pair a thirtieth, and resolutions differ. So speed is normalized to frame-widths per second:
speed_wps = (peak_pixels_per_frame / frame_width) * fps
Now the coarse pass and the dense pass are on the same footing, and
miss_ratio = dense_speed / coarse_speed is a meaningful number.
video
│
├─▶ probe ffprobe: duration, fps, dimensions
│
├─▶ extract coarse evenly spaced stills (default 1 fps) ── the model's normal view
│
├─▶ extract detect dense, downscaled frames (≤30 fps, 160 px)
│ │
│ └─▶ motion timeline per-step "fastest region" score over the whole video
│ │
│ ├─▶ blind-spot demo coarse-implied speed vs dense speed → miss_ratio
│ │
│ └─▶ find windows peaks → padded, merged [start,end] windows
│ │
│ └─▶ per window:
│ extract full-res frames (default 30 fps)
│ Farnebäck flow on each consecutive pair
│ ├─ flow/*.png per-pair HSV (direction+speed)
│ └─ trail.png accumulated "long-exposure of motion"
│
├─▶ (optional) audio WAV + RMS loudness envelope (+ optional transcript)
│
└─▶ manifest.json + SUMMARY.md + legend_flow_wheel.png
Requires Python 3.10+ and ffmpeg/ffprobe on the PATH.
pip install -r requirements.txt # numpy + opencv-python-headless
# ffmpeg must be installed separately, e.g.:
# Debian/Ubuntu: apt-get install ffmpeg
# macOS: brew install ffmpegSpeech-to-text is optional and off by default. Enable it with any Whisper backend (see Audio):
pip install faster-whisper# 1. make the demo clip (a dot that darts between 1 fps samples)
python demo/make_demo_clip.py demo/demo.mp4
# 2. analyze it
python -m phinomena analyze demo/demo.mp4 --out demo/out --clean
# 3. read demo/out/SUMMARY.md, then open:
# demo/out/windows/w0/trail.png the whole event in one glance
# demo/out/windows/w0/flow/*.png per-frame direction + speed
# demo/out/legend_flow_wheel.png the color keyRun it on your own video:
python -m phinomena analyze path/to/video.mp4 --out runs/myvideo --audiopython -m phinomena analyze VIDEO --out DIR [options]
| Flag | Default | Meaning |
|---|---|---|
--out DIR |
(required) | Output directory. |
--coarse-fps F |
1.0 |
Rate of the baseline stills — what the model sees without Phinomena. |
--detect-fps F |
min(native, 30) |
Rate of the cheap dense detection pass. Higher catches briefer events. |
--detect-width PX |
160 |
Width detection frames are downscaled to. Smaller = cheaper. |
--dense-fps F |
30.0 |
Rate of the full-res frames rendered inside a window. |
--render-width PX |
640 |
Cap on window frame width (clamped to source width). |
--detector {flow,framediff} |
flow |
flow = motion-specific (ignores brightness shifts); framediff = cheapest. |
--peak-k K |
3.0 |
Peak threshold = mean + K*std of the timeline. Lower = more windows. |
--top-k N |
6 |
Keep at most N peaks before merging. |
--window-radius S |
0.40 |
Seconds of padding on each side of a peak. |
--max-windows M |
none | Hard cap on rendered windows. |
--mag-scale V |
auto | px/frame mapped to full brightness in flow images. Auto = robust per-window. |
--audio |
off | Extract WAV + RMS loudness envelope. |
--transcribe |
off | Also attempt speech-to-text (needs a Whisper backend). |
--fast-seek |
off | Faster window extraction; snaps window start to the nearest keyframe. |
--no-stamp |
(stamps on) | Don't burn timestamps onto rendered images. |
--keep-detect |
off | Keep the tiny detection frames for inspection. |
--clean |
off | Delete the output dir first if it exists. |
out/
├── manifest.json the agent's index into everything (see references/manifest_schema.md)
├── SUMMARY.md human-readable glance
├── legend_flow_wheel.png color key: hue = direction, brightness = speed
├── coarse/ c_000001.png … the baseline stills
└── windows/
└── w0/
├── trail.png headline: the whole event as one long-exposure-of-motion image
├── window.json this window's per-pair stats
├── frames/ f_000001.png … full-res source frames
└── flow/ flow_000001.png … per consecutive pair: HSV direction+speed
With --audio, an audio/ directory holds audio.wav and the envelope/transcript.
SUMMARY.md/blind_spot_demo— where and how much motion the coarse stills miss, as amiss_ratio.motion_timeline— the dense per-step motion score; its peaks are the timestamps worth looking at.windows/wN/trail.png— one glance per event. A bright streak is the fast thing's path; brightness ramps with speed (the TURBO colormap: dark = still, cyan = slow, red = fast).windows/wN/flow/*.png— frame-by-frame. Hue = direction (cross-check againstlegend_flow_wheel.png), brightness = speed (scaled by the window'smag_scale, recorded in the manifest so it's quantitative).
Catching briefer events → raise --detect-fps (e.g. to the native frame
rate). The detection pass is cheap, so this is usually the first knob to turn.
Too many / too few windows → adjust --peak-k. It's the number of standard
deviations above the timeline's mean a peak must clear. 3.0 is conservative;
drop toward 1.5 to surface subtler motion, raise it to isolate only the
dramatic events. --top-k and --max-windows cap the count directly.
Fast motion looks smeared or noisy in the flow images → Farnebäck needs a
larger analysis window to track big displacements. The defaults (in
phinomena/flow.py, DEFAULT_PARAMS) are tuned for moderate motion:
| Param | Default | Raise it to… | Lower it to… |
|---|---|---|---|
winsize |
15 | track faster motion (more robust, blurrier) | keep crisp edges |
levels |
3 | capture larger displacements | save compute |
poly_n / poly_sigma |
5 / 1.2 | smooth, robust fields (try 7 / 1.5) | sharp detail |
Brightness not telling speeds apart → set --mag-scale explicitly to the
px/frame value you want mapped to full brightness, instead of the auto per-window
scale. Using the same value across runs makes brightness comparable between them.
Watching usually means hearing too, so audio is first-class — split into two layers by how heavy their dependencies are:
- Model-free, always on with
--audio. ffmpeg extracts a mono 16 kHz WAV and NumPy computes an RMS loudness envelope. Loud transients (a clap, a crash, a slam) appear as peaks and can corroborate — or independently flag — interesting moments, with no model download. - Optional transcript with
--transcribe. Iffaster-whisper(preferred) orwhisperis installed, a timestamped transcript is produced and interleaved by time. If neither is present, the run records that audio was captured but not transcribed and continues. Install one to switch it on.
demo/make_demo_clip.py synthesizes a 6-second clip built to expose the blind
spot precisely:
- A big disk drifts slowly left-to-right the whole time — ordinary, low-speed background motion.
- A small bright dot is parked at the left edge at every whole second (t = 0, 1, 2, …). Only between t = 2.40 s and 2.70 s does it rocket across the frame and back, then park again.
So a 1 fps sampler (t = 2 s, 3 s, …) sees the dot in the same spot both times and never witnesses the dart. Running Phinomena on it produces:
miss_ratio: 63× (peak speed actually present vs. coarse-implied)
dense peak at: t≈2.42s (the dart — found despite coarse blindness)
motion events: 1 window, 2.02–3.08s
trail.png shows a hot red track of the dart across the top of the frame and a
cool cyan blob (the slow disk) below — the fast event and the slow background,
cleanly separated. A nice incidental: the slow disk renders as a ring, because
optical flow can't see motion inside a flat-colored region (the aperture
problem) — only its textured edges move. The demo is a faithful, reproducible
illustration of both the problem and the fix.
- It's inference from stills, not true vision. Recovered motion is only as
good as the sampling. Anything faster than the detection rate can still slip
through; raise
--detect-fpsto narrow the gap, never to zero. - Optical flow has failure modes. Flat untextured regions (the aperture problem), large occlusions, motion blur, hard cuts, and extreme displacements all degrade Farnebäck flow. Hue at a hard scene cut is meaningless — lean on the trail and the detection peaks there, not per-pixel direction.
- Small fast objects produce a two-blob flow signature (where the object was
and where it arrived). The accumulated
trail.pngis the cleaner read for trajectory; individual flow frames show the artifact. - Brightness is relative to a scale. Always read speed against the window's
mag_scale(in the manifest), not as an absolute. --fast-seektrades accuracy for speed. It snaps a window's start to the preceding keyframe, so reported window timestamps can be off by up to one GOP. The default accurate path decodes from the top using ffmpeg'strimfilter.- Detector choice matters.
framediffis cheapest but conflates lighting changes with motion; preferflowwhen global brightness varies.
The intended loop, also encoded in SKILL.md:
- Run
analyzeon the video. - Read
manifest.json: checkblind_spot_demo(did the coarse view miss something?) andmotion_timelinepeaks (when?). - For each window, look at
trail.pngfirst (the glance), thenflow/*.pngfor direction and speed detail, cross-referencinglegend_flow_wheel.png. - If a window needs finer detail, re-run that span at a higher
--dense-fps— the model directing its own attention, which is the point.
| File | Responsibility |
|---|---|
phinomena/__init__.py |
Version + the single source of truth for defaults. |
phinomena/probe.py |
ffprobe wrapper → duration, fps, dimensions, frame count. |
phinomena/extract.py |
ffmpeg frame extraction (coarse / dense-detect / accurate windowed). |
phinomena/detect.py |
Dense motion timeline + peak/window detection (the bootstrap fix). |
phinomena/flow.py |
Farnebäck flow, flow statistics, the localized motion score. |
phinomena/render.py |
Flow → HSV image, motion-trail composite, legend wheel, timestamp stamping. |
phinomena/audio.py |
WAV extraction, RMS envelope, optional transcript. |
phinomena/__main__.py |
Pipeline orchestration, blind-spot metric, manifest + SUMMARY. |
demo/make_demo_clip.py |
Synthesizes the blind-spot demonstration clip. |
references/manifest_schema.md |
Field-by-field manifest contract. |
Built as a worked illustration of a single idea: a model can't ingest the raw thing, so convert it into a form it's already fluent in — here, motion into images — and let it steer its own sampling. Use it, fork it, point it at better optical-flow backends. MIT-spirited; no warranty.