Add iOS screen sharing to Gemini realtime calls#89
Draft
yagudaev wants to merge 1 commit into
Draft
Conversation
Introduces screen capture (in-app RPScreenRecorder + background ReplayKit Broadcast Upload Extension) wired into the realtime relay via a new `sendFrame` path on the useRealtime hook. Frames are throttled to 1 FPS, scaled to 768px max, JPEG-encoded at 0.7, and delivered to Gemini Live via the existing `frame.append` relay message. Broadcast extension and host app communicate through an App Group container + Darwin notifications. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
0696302 to
3b1cc31
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
expo-screen-capturenative module (iOS) using RPScreenRecorder for in-app capture and a ReplayKit Broadcast Upload Extension (BroadcastUpload.appex) for background capture, communicating with the host app through an App Group container + Darwin notifications.sendFrame(base64Jpeg)touseRealtime, forwarding JPEGs to the existing relayframe.appendpath (no relay/server changes needed).with-broadcast-extensionprogrammatically registers the extension target in the Xcode project and adds thegroup.com.yagudaev.voiceclaw.broadcastApp Group to the main app's entitlements.Files of note
mobile/modules/expo-screen-capture/— Swift native module + TS bindingsmobile/broadcast-extension/SampleHandler.swift— ReplayKit sample handler (1 FPS, JPEG → App Group file → Darwin notification)mobile/plugins/with-broadcast-extension.js— Expo config plugin; patches pbxproj with fixed UUIDs for idempotencymobile/lib/use-realtime.ts— newsendFramemethod on the realtime controlsmobile/app/(tabs)/index.tsx— UI toggle, event subscriptions, teardown on call-endTest plan
BroadcastUploadtarget compiles cleanly on iphonesimulator (verified viaxcodebuild -target BroadcastUpload)group.com.yagudaev.voiceclaw.broadcastApp Group needs to be registered in Apple Developer portal and added to the dev provisioning profile before the main app / extension can be signed for device). Full-app simulator build is blocked on a pre-existingswift-numericsmodulemap path-resolution issue unrelated to this change.Follow-ups
🤖 Generated with Claude Code