Skip to content

Add iOS screen sharing to Gemini realtime calls#89

Draft
yagudaev wants to merge 1 commit into
mainfrom
feature/ios-screen-share
Draft

Add iOS screen sharing to Gemini realtime calls#89
yagudaev wants to merge 1 commit into
mainfrom
feature/ios-screen-share

Conversation

@yagudaev

Copy link
Copy Markdown
Owner

Summary

  • New expo-screen-capture native module (iOS) using RPScreenRecorder for in-app capture and a ReplayKit Broadcast Upload Extension (BroadcastUpload.appex) for background capture, communicating with the host app through an App Group container + Darwin notifications.
  • Added sendFrame(base64Jpeg) to useRealtime, forwarding JPEGs to the existing relay frame.append path (no relay/server changes needed).
  • Call UI: new Screen-Share toggle button in realtime calls, gated on Gemini models only. Tap = in-app capture; long-press = presents the system broadcast picker for background capture.
  • Frames throttled to 1 FPS, scaled to 768px max, JPEG quality 0.7 (matches the desktop reference implementation).
  • New config plugin with-broadcast-extension programmatically registers the extension target in the Xcode project and adds the group.com.yagudaev.voiceclaw.broadcast App Group to the main app's entitlements.

Files of note

  • mobile/modules/expo-screen-capture/ — Swift native module + TS bindings
  • mobile/broadcast-extension/SampleHandler.swift — ReplayKit sample handler (1 FPS, JPEG → App Group file → Darwin notification)
  • mobile/plugins/with-broadcast-extension.js — Expo config plugin; patches pbxproj with fixed UUIDs for idempotency
  • mobile/lib/use-realtime.ts — new sendFrame method on the realtime controls
  • mobile/app/(tabs)/index.tsx — UI toggle, event subscriptions, teardown on call-end

Test plan

  • BroadcastUpload target compiles cleanly on iphonesimulator (verified via xcodebuild -target BroadcastUpload)
  • Prebuild runs the new config plugin without error; pbxproj additions are idempotent
  • Pending: on-device test on iPhone 12 Pro — blocked by provisioning (the group.com.yagudaev.voiceclaw.broadcast App Group needs to be registered in Apple Developer portal and added to the dev provisioning profile before the main app / extension can be signed for device). Full-app simulator build is blocked on a pre-existing swift-numerics modulemap path-resolution issue unrelated to this change.
  • Manual: after provisioning is updated, verify tap toggles in-app capture (frames visible in relay logs) and long-press launches the system broadcast picker that keeps streaming when the app backgrounds.

Follow-ups

  • Register App Group in Apple Developer portal / regenerate provisioning profiles.
  • Consider exposing a distinct UI affordance for background-share vs. in-app-share (currently tap/long-press) once UX is validated.
  • No OpenAI path; screen-share button is hidden when the active model isn't a Gemini realtime model (per PRD).

🤖 Generated with Claude Code

Introduces screen capture (in-app RPScreenRecorder + background
ReplayKit Broadcast Upload Extension) wired into the realtime relay
via a new `sendFrame` path on the useRealtime hook. Frames are
throttled to 1 FPS, scaled to 768px max, JPEG-encoded at 0.7, and
delivered to Gemini Live via the existing `frame.append` relay
message. Broadcast extension and host app communicate through an App
Group container + Darwin notifications.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@yagudaev yagudaev force-pushed the feature/ios-screen-share branch from 0696302 to 3b1cc31 Compare April 17, 2026 22:29
@yagudaev yagudaev marked this pull request as draft April 24, 2026 19:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant