Repository navigation
Conversation
When begin is asked for rich tracks, a recording on the custom capture pipeline with live upload also keeps input.jsonl.gz in <stem>.tracks/: pointer moves (at most every 10 ms), clicks, wheel and keys, timed on the pipeline's own clock with pauses taken out, positioned in the captured display's native pixels. Typing is only counted; shortcuts and named keys are kept, and no key is kept while secure input is on. The tap is listen-only and starts only when Input Monitoring or Accessibility is already granted. Each track file is uploaded in order as it grows, then finalized with the manifest fields once the recording's own upload is finalized. A track never delays or fails the recording; its folder is deleted only after the tracks are finalized.
The desktop app passes the Rich recordings lab's value to native_fullscreen_recording_begin as richTracks, false unless the lab is on for a signed-in user.
…wards Option with a letter or digit types a character on macOS, so it now counts as typing. Only Cmd or Ctrl make a shortcut; Option with a named key stays a kept key event with alt in its modifiers. Input positions are pixels of the captured area at the display's native resolution, from the area's top-left. Regions are cropped to whole points the way the flattened recording's stream crops them. A track chunk goes only while the recording's live upload has nothing waiting beyond the chunk it is on. After Stop, tracks wait for the recording's own upload to finalize, then send the rest one at a time, input first, and are given up on only when an upload stops moving. The tracks folder is deleted however the recording ends: finalized, finalize failed, tracks stopped by the server, abandoned, discarded, or dropped without finishing. Nothing retries a track later, so a kept folder would never be read again. Window recordings keep no tracks yet.
With the Rich recordings lab on, display and region recordings get a second ScreenCaptureKit stream on the same display and area, at the display's native pixels (best capture resolution) with the cursor hidden, on its own queue and encoder. The flattened recording's stream is unchanged. It writes screen.mp4 as fMP4 H.264 High through the pipeline's segment sink: one-second fragments, each starting on a keyframe, at most 60 fps, and only frames where the screen changed, each at its own time. The bit rate is sized by pixels as the pipeline's is, since AVFoundation takes a quality setting only for JPEG, HEIC and ProRAW. Media time is recording time: each frame's capture time less the flattened recording's first frame and the pipeline's pause offset, as the flattened writer retimes its own, with the session at zero. Pausing stops the stream and resuming starts a fresh one. The input track's pixels are this file's. The track is finalized with its own start and end, initBytes, the codec string from the avcC profile and level, its size, scale, maxFps and a hidden cursor. If its stream or writer fails or falls behind, only this track stops, and it is listed incomplete.
A kept key now takes its label when it goes down and its release repeats it, so a release never reads a character. Keys held across a pause, secure input, a disabled tap or a full queue are forgotten, and a fresh press of a key still held drops the stale entry. Tracks pause as soon as the capture pipeline soft-pauses, before the camera-privacy restore that can fail. An upload the server stops halts both sources and deletes the folder; one that fails for good halts its own source and deletes its file. Quitting stops a rich recording and deletes its folder, and startup deletes tracks folders earlier runs left beside their recordings, matching the recording file name exactly. Stop freezes a boundary in host time that rejects later input events and frames, and wakes the tap's run loop so its thread exits at once. A source that fails to start removes its file, and the folder goes when no source starts. The screen writer allocates through objc2's Allocated init inside an exception boundary, and its callbacks are admitted the way the flattened writer's are.
…s and quitting The input tap's run loop now waits no longer than the time left until its next secure input check, so a quiet loop checks every 100 ms instead of every 250 ms, and a key let go under secure input is forgotten before its stale label can come back on a later release. The screen writer keeps a newly started stream only when the track is not closing as well as on the same generation. A resume that took its generation after a close had begun no longer leaves a stream running for the rest of the recording. The recording clock keeps its pauses in host time. A track stopped while paused ends where the pause began, so the screen file runs to the recording's end rather than a frame after its last change, and input events and frames captured inside any pause get no recording time, however late their callbacks run. The pipeline's pause offset is used as before. A source that cannot write its file now fails its upload instead of cancelling it, so that track ends like a permanent upload failure: its capture stops and its file goes, and the other track carries on. The input writer also stops its tap at once. The screen writer does the same when its file fails to write or close. Every tracks folder is listed from attach until it is removed, and quitting deletes each listed folder without waiting, so a quit while a stopped recording's video or tracks are still uploading leaves nothing behind for the next start's sweep.
On a rented Mac with macOS 26.7.3, ScreenCaptureKit's frames came with no
readable status: screencapturekit's frame_status() returned None for every
frame, the flattened recording's own stream included ("status_now=None").
The screen track kept only frames marked Complete or Started, so it
dropped all of them and closed after 0 frames, while replayd showed its
stream delivering about 57 frames a second. A frame with no status now
counts by its picture alone; idle, blank and suspended frames are still
skipped.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A real recording on a rented MacSetup: a rented Mac mini on macOS 26.7.3, this branch's Clips Alpha debug build, signed in to beta Clips as a test account with the Rich recordings and Resilient recording labs on. Each recording was full screen with system audio only and no camera. It showed a test page with a running clock, a frame counter, a moving box, a text field, a password field and tiles to click. There were four recordings, from 50 seconds to 3 minutes long.
No track reached beta, so I read checks 2 and 3 from the track files on the Mac. I hard-linked them as they appeared, before the app deleted them. Fixed on this branch: the screen track kept no frames (58a3d42)On macOS 26.7.3, ScreenCaptureKit leaves every frame's status unread: After the fix, the screen track recorded at the display's native 1920×1080: 1,593 frames over 27.9 seconds before beta's 503 ended it, about 57 fps, with a median frame gap of 16.7 ms. Its frames show no cursor, while the flattened video's frames do. I matched the page clock across both videos at three points 3 seconds apart. The screen track runs a steady 44 ms ahead of the flattened video, about one frame at 24 fps, with no drift. Typing and the password fieldThe input track held mouse moves, clicks with their button and count, and named keys: arrows, Enter and Backspace, plus Cmd+A as Flattened videoThe longest recording ran 146.7 seconds. Its flattened video was 1280×720 H.264 with AAC audio: 3,457 frames over 146.5 seconds, or 23.6 fps against a 24 fps target. The median frame gap was 41.7 ms, the longest 66.7 ms, and no gap was over 100 ms. Audio and video end within 0.06 seconds of each other. Not fixed: beta refuses track uploadsEvery track's first chunk got In Also seen
|
|
The 503 came from Builder.io storage, which refused to start any track's upload. Beta's function log shows the track route asking Builder.io to reserve 21,474,836,480 bytes (the 20 GB budget for all tracks together), and Builder.io answering |
Builder.io's storage takes a chunk that doesn't end an upload only in whole 256 KiB pieces, so a track's short last chunk was refused (the 2 KB input track) or only partly kept, and no track reached beta. The chunk that ends a closed track file now adds `isFinal=1`, which the track route uses to close the upload with the track's size. A server without that change ignores it and closes the upload at finalize.
|
Recorded again on the rented Mac with #9406 on beta. Its fix works: both tracks' uploads started, and the screen track sent all 44 chunks (about 170 MB). The recording still stayed plain ( Builder.io's storage is Google Cloud Storage, and it takes a chunk that doesn't end an upload only in whole 256 KiB pieces. The app sends whole 3.75 MiB chunks, which fit, then the rest of the file, which almost never does, and the server relayed every chunk as one that doesn't end the upload:
The fix has two halves:
The app is rebuilt from cdeeeaf and installed on the rented Mac. Once the server change is on beta, I'll record again for the first check. |
|
Recorded again on the rented Mac with #9482 on beta. Recording 6 passed all five checks:
The PR body's Verification section now has the details, and this edit reruns its checks. |
|
Landing with Alice's approval. On her Task Priorities board, the Motion M4 row asked:
with the suggested answer:
She switched the row to Agent with no comment at Nov 20, 9:42 AM ET, which takes that suggestion. Marking it ready for review now. I'll answer what the review bot raises, re-test against current main, and merge once it's approved and its checks pass. |
There was a problem hiding this comment.
Builder reviewed your changes and found 1 potential issue 🟡
Review Details
Code Review Summary
This high-risk PR adds optional Rich recordings capture for the macOS Clips app: a native-resolution cursor-free screen track, a privacy-filtered input event track, synchronized recording/pause timing, and background chunked uploads finalized after the flattened recording. The lab flag defaults off, and the existing recording path remains in place when tracks are disabled or unavailable. The architecture appropriately isolates each track's capture and upload failures from the primary recording, bounds upload pressure, and adds focused tests for clock alignment, key filtering, chunking, retry handling, and cleanup. Four parallel balanced reviews found one confirmed medium-severity reliability issue and one low-severity typing-count accuracy issue; no blocking privacy or capture lifecycle issue was confirmed.
Key Findings
- 🟡 Chunk response body read failures can discard an already-stored track chunk — a successful server response whose body read fails is converted to an empty string and treated as a permanent rejection, ending upload and triggering local cleanup rather than retrying the idempotent chunk.
- Autorepeated typing is omitted from counts — keydown autorepeats are dropped before classifying ordinary typing, undercounting held printable keys.
Risk assessment: High, because the change records user input and adds sensitive capture/upload behavior. The approach is otherwise well-isolated and preserves the legacy recording path.
🧪 Browser testing: Will run after this review (PR touches UI code)
| match response { | ||
| Ok(response) => { | ||
| let status = response.status().as_u16(); | ||
| let body = response.text().await.unwrap_or_default(); |
There was a problem hiding this comment.
🟡 Chunk response read failures discard an already-stored track chunk
response.text().await errors are converted to an empty body, then a successful HTTP status is classified as a permanent rejection because {ok:true} is missing. If the server stored the chunk but the response body was interrupted, this ends the track upload and cleanup removes the local file instead of retrying the idempotent chunk; treat body-read failures as transient.
There was a problem hiding this comment.
Fixed in 153b90e: an answer whose body can't be read is now transient, so the chunk goes again, and the track route answers a chunk it already holds with ok: true, duplicate: true. Tracks finalize had the same gap and gets the same rule, since a repeated finalize answers alreadyFinalized. A new test covers both.
A track chunk's answer was read with `unwrap_or_default`, so a body that broke off read as empty, and a 200 without `ok: true` counted as a refusal: the track stopped and its file was deleted, though the server may have stored the chunk. An answer that can't be read is now transient, and the chunk goes again; the server acknowledges a chunk it already holds. Tracks finalize gets the same rule, since a finalize that already went through answers `alreadyFinalized`.
Why
#9070 gave Clips somewhere to keep a recording's parts as separate tracks, so later edits can move the cursor, reframe the camera bubble or rebalance audio instead of working from one flattened video. Nothing records those tracks yet.
This PR makes the Mac app keep the first two, and only while the Rich recordings lab (
clips.rich-recordings) is on. The lab is off by default, and with it off the app records exactly as before.screen.mp4): the screen without the cursor, at the display's own pixels. The flattened video keeps the cursor and is scaled, so the cursor can't be moved or restyled later.input.jsonl.gz): where the cursor was, clicks, scrolling and shortcuts, on the recording's clock, so the cursor can be drawn again and clicks and shortcuts shown.Camera, mic and system sound tracks come in a later PR.
Change
When tracks are kept. The signed-in app passes the lab's value to
native_fullscreen_recording_beginasrichTracks. Tracks are kept only when the recording runs on the custom ScreenCaptureKit pipeline with live upload, which the existing resilient recording flags choose, and records the whole display or a region. Window recordings keep none yet: a window can move between displays, and its frame includes the camera bubble, so neither its pixels nor input on it map onto one display. Each skip is logged with its reason.The recording clock. Both tracks use the flattened video's own clock: milliseconds since its first frame, minus time spent paused. They read the pipeline's start and pause offset directly, so a pause or resume moves all three together. Stop records a boundary in host time, and any input event or frame after it is dropped.
The screen track is a second
SCStreamwithshowsCursor = falseat the display's native size, at most 60 fps, on its own lower-priority queue. A secondAVAssetWriterencodes it as H.264 High and writes fragmented MP4 in one-second segments, so the file can upload while it grows. The bitrate is 0.15 bits per pixel at 24 fps, about 21 Mbps on a 3024×1964 display: about 160 MB a minute for a busy screen, less for a still one. A frame whose status macOS leaves unread, as macOS 26.7.3 does for every frame, counts by its picture alone; idle, blank and suspended frames are still skipped.The input track comes from a listen-only
CGEventTapon its own thread. It records cursor moves and drags (at most one per 10 ms), clicks and scrolling, in the captured area's pixels. Keys are kept only when they can't spell anything:cmd+s.altwhen Option is held.typing, with no character, key code or modifier.A kept key's label is decided when it goes down, and its release repeats that label without reading the key again. Keys held across a pause, secure input, a moment the system switched the tap off, or a full queue are forgotten, so a key pressed before one of those can't be read as kept afterwards. Any such gap marks the track as having lost events. The tap runs only if Clips already has Input Monitoring or Accessibility access, which it checks without prompting. Without either, the recording has only a screen track.
The file is gzip written as one member per second, so each finished second can upload while the recording goes on.
Uploading. Each track uploads its whole chunks to
POST /api/uploads/:id/tracks/:trackwhile recording, but only while the flattened video's own upload has at most one chunk waiting, so tracks never hold it back. After Stop, the video finalizes first; then each track sends the rest of its file, input first, and the app callsPOST /api/uploads/:id/tracks/finalizewith the manifest fields (start time, duration, pauses, capture area, app version, OS). The chunk that ends a track's file addsisFinal=1. Builder.io's storage takes a chunk that doesn't end an upload only in whole 256 KiB pieces, so the track route (#9482) closes the upload with that chunk; a server without that change ignores the flag and closes the upload at finalize. Each track resumes from the chunk index the server expects. A retryable answer, or one whose body can't be read, backs off and tries again a few times. If the server says the tracks have stopped (the size limit, or a redaction burn), both tracks stop and their folder is deleted. If one track's upload fails for good, only that track stops and its file is deleted. The recording itself carries on in every case.Disk. The tracks live in a folder beside the pending recording,
<recording>.tracks, and it is deleted however the recording ends: finalized, discarded, abandoned after a failure, or when the app quits, even while the video or its tracks are still uploading. At startup, Clips deletes tracks folders an earlier run left behind: only directories named exactlyclips-pending-recording-<id>-<pid>.tracksfrom another process, in its own pending uploads folder.What changed.
native_screen/rich_recording.rs(the session's tracks, clock, start, pause, stop, finish and cleanup),input_track.rs,screen_track.rs,custom_capture/screen_writer.rsandtrack_upload.rs.native_screen.rsattaches the tracks after the live upload starts and calls them on pause, resume, stop, discard and failure.custom_capture.rsshares the pipeline's clock.lib.rsstops the tracks on quit and runs the startup sweep.flate2is a new macOS dependency, for the gzip.app.tsxandrecorder.tspassrichTracks, with two tests.No changelog entry: nothing visible changes until the editor uses the tracks.
Verification
cargo buildon macOS (Apple silicon)isFinal, server answers and ones that can't be read, the backlog gate, a file that can't be written, every cleanup path including quitting, the startup sweep's name filternative_screentests passed five runs in a row on a headless Mac, before the last three commits. At this head on the rented Mac below, 156 of 160 pass. The 4 that fail are audio sidecar tests this PR doesn't touch, which need ffmpeg, and that Mac has nonepnpm guards, on LinuxFor the last round of fixes (secure input checks, a stream started while closing, stopping while paused, late events from a pause, a file that can't be written, quitting), each fix was reverted in turn and its test failed.
Real recordings. Builds of this branch recorded six times on rented Mac minis against beta Clips, signed in as a test account with the Rich recordings and Resilient recording labs on. Each recording was full screen with system audio only, over a test page with a running clock, a frame counter, a moving box, a text field, a password field and tiles to click. The first five found three problems, all fixed now:
Recording 6, 3 min 34 s on an Apple M1 with macOS 26.7.3 and all three fixes, passed every check:
get-recording-trackslists both as complete: the screen track (85.7 MB in 22 chunks) and the input track (2.4 KB in one). Both downloads match the Mac's files byte for byte, and no chunk was retried.typingcounts, 3 and 11. None of the typed text appears. Six characters typed into the password field left nothing.Known limits:
AVAssetWriterstarts one at each segment.Review focus
input_track.rs): whether any path keeps a typed character's identity, or anything while secure input is on, especially across pauses and gaps.rich_recording.rs,RecordingClock): whether events and frames land on the flattened video's timeline through pauses and at Stop.rich_recording.rs,remove_once_closed, the startup sweep): whether every ending removes the folder, and whether the sweep could ever match anything but its own leftovers.track_upload.rs.🤖 Generated with Claude Code