Goal: Understand how data flows from gallery upload → drag/drop → timeline → engine → resolver before you implement Scene, Renderer, Pool, or DOM lifecycle (study-plan afternoon session).
Rule: Every
console.logand temporary debug UI you add during this walkthrough is scaffolding — remove it before you commit. Same for the debug<pre>showingSceneJSON.
| Works today | Missing (the gap) |
|---|---|
Upload files in AssetPanel |
No <Preview> / pixel output |
| Drag asset onto timeline | No DomRenderer implementation |
Clips stored in TimelineEngine |
Renderer is only an interface |
| Playhead moves, frame counter updates | Nothing consumes Scene at 60 Hz |
resolveTimeline(frame, project) → Scene |
useResolvedScene exists but nothing renders it |
The afternoon session fills the right column. This document is how you earn the left column first.
cd video-editor
npm run devOpens apps/playground. Confirm the layout:
- Left:
AssetPanel(upload + thumbnails) - Right:
Timeline(ruler, tracks, playhead) - Toolbar: tracks, clips, Play/Pause, Undo/Redo, zoom, frame counter
Manual smoke test:
- Upload a video via AssetPanel (file input or drop on panel).
- Click + Video Track.
- Drag the thumbnail onto the track lane.
- Press Play.
Observe: playhead moves, frame counter updates, no video pixels anywhere. That's expected — you're about to learn why.
Optional: ?lab in the URL loads MediaLimitsLab for browser media experiments (apps/playground/src/App.tsx).
Read (in order):
| File | What to find |
|---|---|
packages/editor/src/editor/AssetPanel/AssetPanel.tsx |
<input type="file">, drop zone, onDragStart |
packages/editor/src/core/media/importFiles.ts |
importFiles(File[]) — probe metadata, thumbnails |
packages/editor/src/core/media/store.ts |
useMediaLibraryStore — assets, order |
packages/editor/src/core/media/types.ts |
MediaAsset, MEDIA_DRAG_MIME, DragMediaPayload |
Key shape — memorize MediaAsset:
interface MediaAsset {
id: string
kind: 'video' | 'audio' | 'image'
name: string
src: string // blob URL after import
durationSec: number
width?, height?, sourceFps?, thumbnailUrl?, byteSize, addedAt
}Console logs to add:
// AssetPanel.tsx — when files are picked or dropped
console.log('[upload] files', files.length, [...files].map(f => f.name))
// importFiles.ts — start
console.log('[importFiles] start', files.length)
// importFiles.ts — before return of each asset
console.log('[importFiles] asset', { id, kind, durationSec, width, height })
// store.ts — after add
console.log('[mediaStore] count', Object.keys(getState().assets).length)Checkpoint: When I drop a file, what exactly lands in the media store? You should be able to describe MediaAsset without opening the file.
Read:
| File | What to find |
|---|---|
AssetPanel.tsx |
MEDIA_DRAG_MIME, JSON.stringify({ kind: 'media-asset', assetId }) |
packages/editor/src/timeline/useTimelineDrop.ts |
dragover, drop, frame math, engine.addClip |
MIME constant: application/x-elah-media (core/media/types.ts).
Payload:
{ kind: 'media-asset', assetId: string }On drop, useTimelineDrop resolves assetId → MediaAsset, computes dropFrame from pointer position + scroll + zoom, then calls engine.addClip with assetId, src, startFrame, durationFrames, etc.
Console logs:
// AssetPanel.tsx — onDragStart
console.log('[drag:start]', payload)
// useTimelineDrop.ts — drop (before addClip)
console.log('[drop]', {
assetId, asset: asset?.name,
dropFrame, durationFrames, trackId, trackKind
})Checkpoint: What must cross the AssetPanel → Timeline boundary, and why each field?
durationSec → durationFrames; src → clip can play later; assetId → dedupe source metadata.
Read:
| File | What to find |
|---|---|
packages/editor/src/core/editor/TimelineEngine.ts |
addClip, commit, getProject, events |
packages/editor/src/core/visitor/add.ts |
Immer draft: insert clip, sort, no overlap |
packages/editor/src/core/stores/tracks.store.ts |
sync(project) after engine.on('change') |
packages/editor/src/editor/EditorProvider.tsx |
engine.on('change') → syncTracks |
Flow:
drop → engine.addClip(options)
→ commit() — Immer produce, history push
→ emit('change')
→ EditorProvider syncTracks()
→ useTracksStore.sync()
→ React re-render (ClipBlock, TrackRow, …)
Console logs:
// TimelineEngine.addClip or commit
console.log('[engine] addClip', { trackId, type, startFrame, durationFrames, assetId })
// commit — after produce
console.log('[engine] commit', { version, clipCount })
// tracks.store sync
console.log('[tracksStore] sync', project.version)Checkpoint: Where does mutation become visible to React? Count the layers: engine → event → store → selector → component.
Architecture rule (Ring 0 → Ring 1): Components never write Project directly. Everything goes through TimelineEngine.commit(). See video-editor/ARCHITECTURE.md §2.
Read:
| File | What to find |
|---|---|
packages/editor/src/core/playback/PlaybackEngine.ts |
anchor-and-integrate, getFrameAt(), rAF sampler |
packages/editor/src/core/stores/playback.store.ts |
togglePlayPause, setCurrentFrame, currentFrameEpoch |
packages/editor/src/editor/EditorProvider.tsx |
Engine ↔ store bridge (both directions) |
Bridge lives in EditorProvider.tsx, not Timeline.tsx:
- Engine → Store (
playback.subscribe): updatescurrentFrameonly when integer frame changed; syncsisPlaying. - Store → Engine (
usePlaybackStore.subscribe): play/pause/seek/rate/loop from UI back into engine.
Echo guard (prevents infinite loop):
// Store → Engine: only seek when epoch changed AND frames differ
if (state.currentFrameEpoch !== prev.currentFrameEpoch &&
state.currentFrame !== playback.currentFrame) {
playback.seek(state.currentFrame)
}Ruler scrub: Ruler.onSeek → setCurrentFrame → store epoch bump → engine seek.
Console logs:
// PlaybackEngine — transport only (not every RAF tick)
console.log('[clock] play')
console.log('[clock] pause')
console.log('[clock] seek', frame, 'epoch', this._epoch)
// Throttled during play (e.g. every 30 frames)
console.log('[clock] tick', snapshot.currentFrame)
// EditorProvider — bridge
console.log('[bridge] engine→store', snapshot.currentFrame)
console.log('[bridge] store→engine seek', state.currentFrame)Checkpoint: Click the ruler — why doesn't play/pause loop forever? Point to the epoch + equality guards in EditorProvider.tsx.
Checkpoint: What does the clock emit that nothing consumes for pixels? currentFrame + resolveTimeline → Scene with no renderer.
Deep dive (optional): 01-playback-clock-architecture.md, docs/backlog/update-clock-fucntion.md.
Read:
| File | What to find |
|---|---|
packages/editor/src/core/resolver/scene.ts |
Scene, ActiveVideoClip, ActiveAudioClip, … |
packages/editor/src/core/resolver/resolveTimeline.ts |
rules: inclusion, source mapping, mute/solo, zIndex |
packages/editor/src/core/resolver/resolveTimeline.test.ts |
executable spec — read the cases |
packages/editor/src/editor/useResolvedScene.ts |
useMemo + resolveTimeline(frame, project) |
Scene at frame F tells you:
- Which clips are active (half-open:
startFrame <= F < startFrame + duration) - Each clip's
sourceFrameinside the media file zIndex,opacity, volume (for audio)
It does NOT: touch DOM, seek <video>, or know about React.
Console log:
// resolveTimeline.ts — end of function
console.log('[resolve]', frame, {
v: scene.videos.length,
a: scene.audios.length,
t: scene.texts.length,
i: scene.images.length,
})Temporary debug UI (playground only):
In apps/playground/src/App.tsx:
import { useResolvedScene } from '@elah/editor'
// inside App:
const scene = useResolvedScene()
// render somewhere visible:
<pre style={{ fontSize: 10, maxHeight: 120, overflow: 'auto' }}>
{JSON.stringify(scene, null, 2)}
</pre>Press Play and watch scene.videos[0].sourceFrame increment. This is the renderer's input contract.
Checkpoint: At frame 142, what does the Scene say? Example: "Show clip c1 at sourceFrame 12, zIndex 1000."
Read:
| File | What to find |
|---|---|
packages/editor/src/core/renderer/types.ts |
Renderer { mount, render, dispose } — interface only |
packages/editor/src/index.ts |
exports useResolvedScene, no Preview, no DomRenderer |
Planned contract:
interface Renderer {
mount(container: HTMLElement): void
render(scene: Scene): void
dispose(): void
}Checkpoint: If I had a DomRenderer today, who calls render(scene) 60×/sec?
| Path | Pros | Cons |
|---|---|---|
A. <Preview> uses useResolvedScene() in useEffect |
Easy to wire in React | React may coalesce updates; not frame-accurate |
B. Renderer subscribes to playback.subscribe(), calls resolveTimeline itself |
Frame-accurate, bypasses React | More wiring in Preview mount |
ARCHITECTURE.md §6 recommends B for the DomRenderer MVP. Start with A if you need to see pixels fast; migrate to B when you notice stutter.
Before writing renderer code, document one path on paper or in a section below.
Template:
[upload video.mp4]
AssetPanel.onDrop / file input
→ importFiles([file])
→ MediaAsset { id, src: blob:…, durationSec, kind: 'video' }
→ useMediaLibraryStore
[drag asset onto track t1 @ ~300px, zoom=10]
AssetPanel onDragStart → MEDIA_DRAG_MIME payload
→ useTimelineDrop.onDrop
→ dropFrame = …
→ engine.addClip({ trackId, assetId, src, startFrame, durationFrames, … })
→ commit → emit('change')
→ tracks.store.sync → ClipBlock appears
[press Play]
toolbar togglePlayPause
→ playback.store → EditorProvider → playback.play()
→ rAF → notify({ currentFrame })
→ playback.store.setCurrentFrame
→ useResolvedScene → resolveTimeline(frame, project)
→ Scene { videos: [{ id, sourceFrame, zIndex, … }] }
→ (nothing renders pixels — GAP)
Done when: you can draw this without opening the codebase.
Create video-editor/docs/notes/renderer-design.md (or a section in this file) and answer:
Scene field |
DOM action |
|---|---|
videos[] |
<video> per active clip, currentTime = sourceFrame / fps |
audios[] |
<audio> or Web Audio graph (later) |
texts[] |
<div> with content, positioned with transform |
images[] |
<img src=…> |
Renderer reads only Scene. Never Project, Clip, or Track. See 05-dom-renderer-design.md and 07-video-element-pooling.md.
When a clip leaves scene.videos (ended or seeked away):
- Destroy
<video>? No — return to pool (decode cache is expensive). - Same clip returns later? Reuse pooled element by
clip.id.
Read 07-video-element-pooling.md before implementing VideoPool.ts.
Each render(scene):
- Diff previous scene clip ids vs current.
- Appeared → acquire from pool, mount, seek.
- Persisted → update
currentTime/ opacity only. - Disappeared → return to pool (don't destroy unless pool cap exceeded).
Same idea as React reconciliation, but you own it to avoid 60 Hz React re-renders.
Choose A or B from Step 6. Write down which you'll build first and what triggers render(scene).
Only after Steps 0–8:
| Order | File | Role |
|---|---|---|
| 1 | core/renderer/DomRenderer.ts |
Implements Renderer, no React |
| 2 | core/renderer/VideoPool.ts |
Reuse <video> elements |
| 3 | editor/Preview/Preview.tsx |
Mounts renderer, feeds scenes |
| 4 | apps/playground/src/App.tsx |
<Preview> sibling of AssetPanel + Timeline |
Read first: 05-dom-renderer-design.md, 07-video-element-pooling.md, video-editor/ARCHITECTURE.md §§5–7.
Do not start Step 9 until Step 7 diagram and Step 8 answers exist.
apps/playground/src/App.tsx ← dev shell, toolbar, layout
editor/
EditorProvider.tsx ← engines + store bridges
AssetPanel/AssetPanel.tsx ← upload, drag source
useResolvedScene.ts ← Scene for current frame
timeline/
Timeline.tsx ← ruler, tracks, playhead
useTimelineDrop.ts ← drop → addClip
Playhead.tsx, Ruler.tsx, TrackRow.tsx, ClipBlock.tsx
core/
media/importFiles.ts, store.ts, types.ts
editor/TimelineEngine.ts
playback/PlaybackEngine.ts
resolver/resolveTimeline.ts, scene.ts
renderer/types.ts ← interface only (empty seat)
stores/playback.store.ts, tracks.store.ts
| study-plan.md afternoon focus | Covered in step |
|---|---|
| Scene | Step 5 |
| Renderer | Steps 6, 8 |
| Pool | Step 8.2 → implement Step 9.2 |
| DOM lifecycle | Step 8.3 → implement Step 9.1 |
- Ran playground; uploaded, dragged, played; confirmed no Preview pixels
- Traced upload →
MediaAssetin store - Traced drag MIME →
useTimelineDrop→addClip - Traced
commit→tracks.store.sync→ UI update - Found engine↔store bridge in
EditorProvider.tsx; explained echo guard - Watched live
SceneJSON while playing (temporary debug UI) - Located empty
Rendererinterface - Drew full lifecycle diagram (Step 7)
- Answered four design questions (Step 8)
- Removed all temporary
console.logs and debug UI
When all boxes are checked, you're ready for the afternoon implementation session.