Skip to content

Keep a screen track and an input track from the Mac app behind the Rich recordings lab - #9298

Open
3mdistal wants to merge 9 commits into
mainfrom
clips/rich-recording-desktop-tracks
Open

3mdistal wants to merge 9 commits into
mainfrom
clips/rich-recording-desktop-tracks

Conversation

@3mdistal

@3mdistal 3mdistal commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Why

#9070 gave Clips somewhere to keep a recording's parts as separate tracks, so later edits can move the cursor, reframe the camera bubble or rebalance audio instead of working from one flattened video. Nothing records those tracks yet.

This PR makes the Mac app keep the first two, and only while the Rich recordings lab (clips.rich-recordings) is on. The lab is off by default, and with it off the app records exactly as before.

  • A screen track (screen.mp4): the screen without the cursor, at the display's own pixels. The flattened video keeps the cursor and is scaled, so the cursor can't be moved or restyled later.
  • An input track (input.jsonl.gz): where the cursor was, clicks, scrolling and shortcuts, on the recording's clock, so the cursor can be drawn again and clicks and shortcuts shown.

Camera, mic and system sound tracks come in a later PR.

Change

When tracks are kept. The signed-in app passes the lab's value to native_fullscreen_recording_begin as richTracks. Tracks are kept only when the recording runs on the custom ScreenCaptureKit pipeline with live upload, which the existing resilient recording flags choose, and records the whole display or a region. Window recordings keep none yet: a window can move between displays, and its frame includes the camera bubble, so neither its pixels nor input on it map onto one display. Each skip is logged with its reason.

The recording clock. Both tracks use the flattened video's own clock: milliseconds since its first frame, minus time spent paused. They read the pipeline's start and pause offset directly, so a pause or resume moves all three together. Stop records a boundary in host time, and any input event or frame after it is dropped.

The screen track is a second SCStream with showsCursor = false at the display's native size, at most 60 fps, on its own lower-priority queue. A second AVAssetWriter encodes it as H.264 High and writes fragmented MP4 in one-second segments, so the file can upload while it grows. The bitrate is 0.15 bits per pixel at 24 fps, about 21 Mbps on a 3024×1964 display: about 160 MB a minute for a busy screen, less for a still one. A frame whose status macOS leaves unread, as macOS 26.7.3 does for every frame, counts by its picture alone; idle, blank and suspended frames are still skipped.

The input track comes from a listen-only CGEventTap on its own thread. It records cursor moves and drags (at most one per 10 ms), clicks and scrolling, in the captured area's pixels. Keys are kept only when they can't spell anything:

  • A shortcut with Cmd or Ctrl keeps its key and modifiers, such as cmd+s.
  • A named key (arrows, Return, Escape, Tab, Delete, function keys) is kept, with alt when Option is held.
  • Every other key is only counted, per second, as typing, with no character, key code or modifier.
  • While secure input is on, as in a password field, nothing about keys is kept at all, not even the count. Cursor moves, clicks and scrolling are still kept, since the flattened video already shows the cursor.

A kept key's label is decided when it goes down, and its release repeats that label without reading the key again. Keys held across a pause, secure input, a moment the system switched the tap off, or a full queue are forgotten, so a key pressed before one of those can't be read as kept afterwards. Any such gap marks the track as having lost events. The tap runs only if Clips already has Input Monitoring or Accessibility access, which it checks without prompting. Without either, the recording has only a screen track.

The file is gzip written as one member per second, so each finished second can upload while the recording goes on.

Uploading. Each track uploads its whole chunks to POST /api/uploads/:id/tracks/:track while recording, but only while the flattened video's own upload has at most one chunk waiting, so tracks never hold it back. After Stop, the video finalizes first; then each track sends the rest of its file, input first, and the app calls POST /api/uploads/:id/tracks/finalize with the manifest fields (start time, duration, pauses, capture area, app version, OS). The chunk that ends a track's file adds isFinal=1. Builder.io's storage takes a chunk that doesn't end an upload only in whole 256 KiB pieces, so the track route (#9482) closes the upload with that chunk; a server without that change ignores the flag and closes the upload at finalize. Each track resumes from the chunk index the server expects. A retryable answer, or one whose body can't be read, backs off and tries again a few times. If the server says the tracks have stopped (the size limit, or a redaction burn), both tracks stop and their folder is deleted. If one track's upload fails for good, only that track stops and its file is deleted. The recording itself carries on in every case.

Disk. The tracks live in a folder beside the pending recording, <recording>.tracks, and it is deleted however the recording ends: finalized, discarded, abandoned after a failure, or when the app quits, even while the video or its tracks are still uploading. At startup, Clips deletes tracks folders an earlier run left behind: only directories named exactly clips-pending-recording-<id>-<pid>.tracks from another process, in its own pending uploads folder.

What changed.

  • New, macOS only: native_screen/rich_recording.rs (the session's tracks, clock, start, pause, stop, finish and cleanup), input_track.rs, screen_track.rs, custom_capture/screen_writer.rs and track_upload.rs.
  • native_screen.rs attaches the tracks after the live upload starts and calls them on pause, resume, stop, discard and failure. custom_capture.rs shares the pipeline's clock. lib.rs stops the tracks on quit and runs the startup sweep.
  • flate2 is a new macOS dependency, for the gzip.
  • app.tsx and recorder.ts pass richTracks, with two tests.

No changelog entry: nothing visible changes until the editor uses the tracks.

Verification

Check Result
cargo build on macOS (Apple silicon) passes
49 new Rust tests: privacy rules, secure input and how often it is checked, held keys across gaps, the clock, pauses and late events, Stop's boundary and stopping while paused, upload resume and retries, the last chunk's isFinal, server answers and ones that can't be read, the backlog gate, a file that can't be written, every cleanup path including quitting, the startup sweep's name filter all 157 native_screen tests passed five runs in a row on a headless Mac, before the last three commits. At this head on the rented Mac below, 156 of 160 pass. The 4 that fail are audio sidecar tests this PR doesn't touch, which need ffmpeg, and that Mac has none
Desktop typecheck, the desktop tests and pnpm guards, on Linux pass: 1,069 tests, 108 guards

For the last round of fixes (secure input checks, a stream started while closing, stopping while paused, late events from a pause, a file that can't be written, quitting), each fix was reverted in turn and its test failed.

Real recordings. Builds of this branch recorded six times on rented Mac minis against beta Clips, signed in as a test account with the Rich recordings and Resilient recording labs on. Each recording was full screen with system audio only, over a test page with a running clock, a frame counter, a moving box, a text field, a password field and tiles to click. The first five found three problems, all fixed now:

Recording 6, 3 min 34 s on an Apple M1 with macOS 26.7.3 and all three fixes, passed every check:

Check Recording 6
Both tracks reach Clips and the recording turns rich get-recording-tracks lists both as complete: the screen track (85.7 MB in 22 chunks) and the input track (2.4 KB in one). Both downloads match the Mac's files byte for byte, and no chunk was retried.
The screen track lines up with the flattened video, without a cursor Reading the page clock against each frame's timestamp at 60, 100, 150 and 200 s, the screen track runs 29 to 52 ms ahead, about one frame at 24 fps, with no drift. Its frames show no cursor.
Typing shows only as counts, and nothing during a password field Besides moves and clicks, the track holds End, Cmd+A, Backspace and Enter, and two typing counts, 3 and 11. None of the typed text appears. Six characters typed into the password field left nothing.
No slowdown or dropped frames in the flattened video 5,035 frames over 213.5 s, with a median gap of 41.7 ms and a longest of 66.7 ms. Over 140 s, its timestamps and the page clock agree within 8 ms. The screen track has 11,867 frames, about 56 fps, with a longest gap of 41.7 ms.
The tracks folder is gone afterwards It was deleted as soon as the tracks finalized.

Known limits:

  • Window recordings keep no tracks.
  • The screen track runs about one frame ahead of the flattened video: 29 to 52 ms in recording 6, 44 ms in an earlier one.
  • The screen track has a keyframe about every second, since AVAssetWriter starts one at each segment.
  • The backlog check decides when a track chunk may start, not how bandwidth is shared once it has. A track chunk already sending competes with the video's next one.
  • Secure input is checked every 100 ms. If a shorter secure stretch is missed, a shortcut key held into it could have a later release recorded with the label its press already had. macOS doesn't pass keys typed under secure input to the tap, so no typed character is recorded either way.
  • The screen writer is created while the recording's session lock is held, and the ScreenCaptureKit stop calls have no timeout. Both match the flattened pipeline today.
  • If the app quits before the tracks finish uploading, they never finalize and the recording stays flat.
  • Recordings made while Screen Memory is on keep no tracks yet: they come from Screen Memory's own recorder, which this PR doesn't change.

Review focus

  • Privacy of the input track (input_track.rs): whether any path keeps a typed character's identity, or anything while secure input is on, especially across pauses and gaps.
  • The clock (rich_recording.rs, RecordingClock): whether events and frames land on the flattened video's timeline through pauses and at Stop.
  • Cleanup (rich_recording.rs, remove_once_closed, the startup sweep): whether every ending removes the folder, and whether the sweep could ever match anything but its own leftovers.
  • Effect on the flattened recording: the second stream's queue priority, and the backlog gate in track_upload.rs.

🤖 Generated with Claude Code

3mdistal and others added 7 commits November 17, 2026 00:07
When begin is asked for rich tracks, a recording on the custom capture
pipeline with live upload also keeps input.jsonl.gz in <stem>.tracks/:
pointer moves (at most every 10 ms), clicks, wheel and keys, timed on the
pipeline's own clock with pauses taken out, positioned in the captured
display's native pixels. Typing is only counted; shortcuts and named keys
are kept, and no key is kept while secure input is on. The tap is
listen-only and starts only when Input Monitoring or Accessibility is
already granted.

Each track file is uploaded in order as it grows, then finalized with the
manifest fields once the recording's own upload is finalized. A track
never delays or fails the recording; its folder is deleted only after the
tracks are finalized.
The desktop app passes the Rich recordings lab's value to
native_fullscreen_recording_begin as richTracks, false unless the lab is
on for a signed-in user.
…wards

Option with a letter or digit types a character on macOS, so it now counts
as typing. Only Cmd or Ctrl make a shortcut; Option with a named key stays a
kept key event with alt in its modifiers.

Input positions are pixels of the captured area at the display's native
resolution, from the area's top-left. Regions are cropped to whole points
the way the flattened recording's stream crops them.

A track chunk goes only while the recording's live upload has nothing
waiting beyond the chunk it is on. After Stop, tracks wait for the
recording's own upload to finalize, then send the rest one at a time, input
first, and are given up on only when an upload stops moving.

The tracks folder is deleted however the recording ends: finalized, finalize
failed, tracks stopped by the server, abandoned, discarded, or dropped
without finishing. Nothing retries a track later, so a kept folder would
never be read again. Window recordings keep no tracks yet.
With the Rich recordings lab on, display and region recordings get a second
ScreenCaptureKit stream on the same display and area, at the display's
native pixels (best capture resolution) with the cursor hidden, on its own
queue and encoder. The flattened recording's stream is unchanged.

It writes screen.mp4 as fMP4 H.264 High through the pipeline's segment sink:
one-second fragments, each starting on a keyframe, at most 60 fps, and only
frames where the screen changed, each at its own time. The bit rate is sized
by pixels as the pipeline's is, since AVFoundation takes a quality setting
only for JPEG, HEIC and ProRAW.

Media time is recording time: each frame's capture time less the flattened
recording's first frame and the pipeline's pause offset, as the flattened
writer retimes its own, with the session at zero. Pausing stops the stream
and resuming starts a fresh one. The input track's pixels are this file's.

The track is finalized with its own start and end, initBytes, the codec
string from the avcC profile and level, its size, scale, maxFps and a hidden
cursor. If its stream or writer fails or falls behind, only this track
stops, and it is listed incomplete.
A kept key now takes its label when it goes down and its release repeats
it, so a release never reads a character. Keys held across a pause,
secure input, a disabled tap or a full queue are forgotten, and a fresh
press of a key still held drops the stale entry.

Tracks pause as soon as the capture pipeline soft-pauses, before the
camera-privacy restore that can fail. An upload the server stops halts
both sources and deletes the folder; one that fails for good halts its
own source and deletes its file. Quitting stops a rich recording and
deletes its folder, and startup deletes tracks folders earlier runs left
beside their recordings, matching the recording file name exactly.

Stop freezes a boundary in host time that rejects later input events and
frames, and wakes the tap's run loop so its thread exits at once. A
source that fails to start removes its file, and the folder goes when no
source starts. The screen writer allocates through objc2's Allocated
init inside an exception boundary, and its callbacks are admitted the
way the flattened writer's are.
…s and quitting

The input tap's run loop now waits no longer than the time left until its
next secure input check, so a quiet loop checks every 100 ms instead of
every 250 ms, and a key let go under secure input is forgotten before its
stale label can come back on a later release.

The screen writer keeps a newly started stream only when the track is not
closing as well as on the same generation. A resume that took its
generation after a close had begun no longer leaves a stream running for
the rest of the recording.

The recording clock keeps its pauses in host time. A track stopped while
paused ends where the pause began, so the screen file runs to the
recording's end rather than a frame after its last change, and input
events and frames captured inside any pause get no recording time,
however late their callbacks run. The pipeline's pause offset is used as
before.

A source that cannot write its file now fails its upload instead of
cancelling it, so that track ends like a permanent upload failure: its
capture stops and its file goes, and the other track carries on. The
input writer also stops its tap at once. The screen writer does the same
when its file fails to write or close.

Every tracks folder is listed from attach until it is removed, and
quitting deletes each listed folder without waiting, so a quit while a
stopped recording's video or tracks are still uploading leaves nothing
behind for the next start's sweep.
On a rented Mac with macOS 26.7.3, ScreenCaptureKit's frames came with no
readable status: screencapturekit's frame_status() returned None for every
frame, the flattened recording's own stream included ("status_now=None").
The screen track kept only frames marked Complete or Started, so it
dropped all of them and closed after 0 frames, while replayd showed its
stream delivering about 57 frames a second. A frame with no status now
counts by its picture alone; idle, blank and suspended frames are still
skipped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@3mdistal

Copy link
Copy Markdown
Contributor Author

A real recording on a rented Mac

Setup: a rented Mac mini on macOS 26.7.3, this branch's Clips Alpha debug build, signed in to beta Clips as a test account with the Rich recordings and Resilient recording labs on. Each recording was full screen with system audio only and no camera. It showed a test page with a running clock, a frame counter, a moving box, a text field, a password field and tiles to click. There were four recordings, from 50 seconds to 3 minutes long.

Check Result
1. Both tracks reach Clips and the recording turns rich ✗ Beta refused every track upload with a 503 (below)
2. The screen track lines up with the flattened video and has no cursor ✓ after the fix below, read from the Mac's copy
3. Typing shows only as counts, and nothing is kept during a password field ✓ read from the Mac's copy
4. The flattened video has no slowdown or dropped frames ✓
5. The tracks folder is gone afterwards ✓

No track reached beta, so I read checks 2 and 3 from the track files on the Mac. I hard-linked them as they appeared, before the app deleted them.

Fixed on this branch: the screen track kept no frames (58a3d42)

On macOS 26.7.3, ScreenCaptureKit leaves every frame's status unread: frame_status() returns None. That holds for the flattened recording's own stream too, which logs status_now=None. The screen writer kept only frames marked Complete or Started, so the track closed after 0 frames. Meanwhile replayd showed its stream delivering about 57 frames a second. Now a frame with no status counts by its picture alone, and idle, blank and suspended frames are still skipped. cargo test --lib screen_writer passes, including a new test for this case.

After the fix, the screen track recorded at the display's native 1920×1080: 1,593 frames over 27.9 seconds before beta's 503 ended it, about 57 fps, with a median frame gap of 16.7 ms. Its frames show no cursor, while the flattened video's frames do. I matched the page clock across both videos at three points 3 seconds apart. The screen track runs a steady 44 ms ahead of the flattened video, about one frame at 24 fps, with no drift.

Typing and the password field

The input track held mouse moves, clicks with their button and count, and named keys: arrows, Enter and Backspace, plus Cmd+A as KeyA with meta. Typing appears only as counts: "hello rich tracks" became typing n:17. None of the typed words appear anywhere in the file. From the click into the password field until the next click elsewhere, the track holds nothing. In that span I typed Cmd+A, Backspace, an 11-character fake password and Enter.

Flattened video

The longest recording ran 146.7 seconds. Its flattened video was 1280×720 H.264 with AAC audio: 3,457 frames over 146.5 seconds, or 23.6 fps against a 24 fps target. The median frame gap was 41.7 ms, the longest 66.7 ms, and no gap was over 100 ms. Audio and video end within 0.06 seconds of each other.

Not fixed: beta refuses track uploads

Every track's first chunk got 503 Track storage could not start an upload. The app tried five times over about 7.5 seconds, then gave up and deleted the track file. Once no track was left, it removed the folder. The recordings themselves uploaded and play; they just stay plain.

In tracks/[track].post.ts, storage resolved and its provider is resumable, so the 503 means provider.resumable.startSession threw. In the S3 provider, that's one of two things: no S3 config for the recording's storage mode, or CreateMultipartUpload returning an error. Beta's server log says which, as [track-chunk] <track> upload of <recording id> could not start: <message>. The recordings were made between 21:45 and 22:30 UTC on Nov 18, among them 9f4ae84e-1ce1-42f4-be5b-3e58c99ccf09, 7f58ff92-4181-4e18-a022-60f5263ba8d3 and 9d295c3a-4f33-4b45-80cb-545ade142d0f. That route came in with #9070, so the 503 comes from beta's server, not from this branch.

Also seen

  • The first recording, made on the build before the fix, ended with no tracks at all, and I couldn't tell why. The app's [rich-tracks] lines go only to stderr, which a normal launch doesn't keep. They never reach clips-tray.log, so a user's report would hold no trace of why tracks were skipped or lost.
  • On a 503 that the server marks retryable, the app gives up after about 7.5 seconds and deletes the track. A short storage outage costs the whole track.

@3mdistal

Copy link
Copy Markdown
Contributor Author

The 503 came from Builder.io storage, which refused to start any track's upload. Beta's function log shows the track route asking Builder.io to reserve 21,474,836,480 bytes (the 20 GB budget for all tracks together), and Builder.io answering 413 File is too large. The flattened video asks for 2 GB, which Builder.io takes. #9406 starts each track's upload at no more than the video's size and stops the tracks cleanly at that size. Once it's on beta, I'll record again on the rented Mac for the first check.

Builder.io's storage takes a chunk that doesn't end an upload only in
whole 256 KiB pieces, so a track's short last chunk was refused (the
2 KB input track) or only partly kept, and no track reached beta.

The chunk that ends a closed track file now adds `isFinal=1`, which the
track route uses to close the upload with the track's size. A server
without that change ignores it and closes the upload at finalize.
@3mdistal

Copy link
Copy Markdown
Contributor Author

Recorded again on the rented Mac with #9406 on beta. Its fix works: both tracks' uploads started, and the screen track sent all 44 chunks (about 170 MB). The recording still stayed plain (get-recording-tracks answers no_tracks), for a new reason.

Builder.io's storage is Google Cloud Storage, and it takes a chunk that doesn't end an upload only in whole 256 KiB pieces. The app sends whole 3.75 MiB chunks, which fit, then the rest of the file, which almost never does, and the server relayed every chunk as one that doesn't end the upload:

  • The input track was 2 KB, so its one chunk was refused with 400 ("The number of bytes uploaded is required to be equal or greater than 262144, except for the final request"). The app retried five times, then deleted the track.
  • The screen track's last chunk was 930,396 bytes. It was taken, but finalize then answered 502 six times.

The fix has two halves:

The app is rebuilt from cdeeeaf and installed on the rented Mac. Once the server change is on beta, I'll record again for the first check.

@3mdistal

Copy link
Copy Markdown
Contributor Author

Recorded again on the rented Mac with #9482 on beta. Recording 6 passed all five checks:

  • Both tracks reached Clips and the recording turned rich. get-recording-tracks lists the screen track (85.7 MB in 22 chunks) and the input track (2.4 KB in one) as complete, and both downloads match the Mac's files byte for byte. No chunk was retried.
  • The screen track lines up and has no cursor. It runs 29 to 52 ms ahead of the flattened video, about one frame, with no drift, as in recording 4.
  • Typing shows only as counts. Two typing counts (3 and 11) and four named keys; none of the typed text appears, and six characters typed into the password field left nothing.
  • No slowdown or dropped frames. The flattened video's longest frame gap is 66.7 ms, and its timestamps agree with the page clock within 8 ms over 140 s.
  • The tracks folder was deleted as soon as the tracks finalized.

The PR body's Verification section now has the details, and this edit reruns its checks.

@3mdistal

Copy link
Copy Markdown
Contributor Author

Landing with Alice's approval. On her Task Priorities board, the Motion M4 row asked:

Land the desktop tracks pull request, now that a real recording on beta passed all five checks?

with the suggested answer:

Land it, and delete the rented Mac once it has merged.

She switched the row to Agent with no comment at Nov 20, 9:42 AM ET, which takes that suggestion.

Marking it ready for review now. I'll answer what the review bot raises, re-test against current main, and merge once it's approved and its checks pass.

@3mdistal
3mdistal marked this pull request as ready for review November 20, 2026 14:45

@builder-io-integration builder-io-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Builder reviewed your changes and found 1 potential issue 🟡

Review Details

Code Review Summary

This high-risk PR adds optional Rich recordings capture for the macOS Clips app: a native-resolution cursor-free screen track, a privacy-filtered input event track, synchronized recording/pause timing, and background chunked uploads finalized after the flattened recording. The lab flag defaults off, and the existing recording path remains in place when tracks are disabled or unavailable. The architecture appropriately isolates each track's capture and upload failures from the primary recording, bounds upload pressure, and adds focused tests for clock alignment, key filtering, chunking, retry handling, and cleanup. Four parallel balanced reviews found one confirmed medium-severity reliability issue and one low-severity typing-count accuracy issue; no blocking privacy or capture lifecycle issue was confirmed.

Key Findings

  • 🟡 Chunk response body read failures can discard an already-stored track chunk — a successful server response whose body read fails is converted to an empty string and treated as a permanent rejection, ending upload and triggering local cleanup rather than retrying the idempotent chunk.
  • Autorepeated typing is omitted from counts — keydown autorepeats are dropped before classifying ordinary typing, undercounting held printable keys.

Risk assessment: High, because the change records user input and adds sensitive capture/upload behavior. The approach is otherwise well-isolated and preserves the legacy recording path.

🧪 Browser testing: Will run after this review (PR touches UI code)

match response {
Ok(response) => {
let status = response.status().as_u16();
let body = response.text().await.unwrap_or_default();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Chunk response read failures discard an already-stored track chunk

response.text().await errors are converted to an empty body, then a successful HTTP status is classified as a permanent rejection because {ok:true} is missing. If the server stored the chunk but the response body was interrupted, this ends the track upload and cleanup removes the local file instead of retrying the idempotent chunk; treat body-read failures as transient.

Fix in Builder

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 153b90e: an answer whose body can't be read is now transient, so the chunk goes again, and the track route answers a chunk it already holds with ok: true, duplicate: true. Tracks finalize had the same gap and gets the same rule, since a repeated finalize answers alreadyFinalized. A new test covers both.

A track chunk's answer was read with `unwrap_or_default`, so a body that
broke off read as empty, and a 200 without `ok: true` counted as a
refusal: the track stopped and its file was deleted, though the server
may have stored the chunk. An answer that can't be read is now
transient, and the chunk goes again; the server acknowledges a chunk it
already holds. Tracks finalize gets the same rule, since a finalize
that already went through answers `alreadyFinalized`.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants