Mobile companion app (Android, iOS-ready) with embedded WebUI avatar + local asset cache

[XargonWan]

Summary

Build a native mobile companion app for Synthetic Heart, starting with Android
(iOS-ready by design). The mobile web experience works but is heavy and clunky on phones;
the app should provide a faster, more optimized experience while reusing the existing
classic WebUI for avatar + chat rendering.

The app embeds the classic WebUI (res/synth_webui/ + the home section) in a WebView
for VRM/avatar + chat rendering, bakes in / caches static assets (VRM models, FBX
animations) locally, and adds a new backend mobile interface so Synth can address the
device directly (proactive push, dedicated identity mobile/<device_id>).

The Vue frontend (frontend/) is intentionally out of scope for now — we focus on the
classic WebUI. The web experience is not being dropped; it will be polished separately later.

Goals

• Baked-in / cached assets — static resources (skins, VRM models, FBX animations,
  descriptors) shipped in the APK and/or cached on device.
• More optimized than the mobile website.
• Same look & feel as the classic WebUI.
• Home-screen widget — quick-send text / image / audio to Synth on the fly.
• System assistant — register as the Android digital assistant (Gemini replacement).
• Android Auto — conversational, TTS-first.

Architecture (reuse existing contracts)

• WebSocket /ws carries both chat and avatar state (core/webui.py::websocket_endpoint).
  Server→client: message, message_ack, action_state, tts-play, vrm_animation_v2,
  vrm_face, vrm_model, vrm_expression_set/clear, vrm_preload. Client→server:
  {text, attachments?, is_voice_input?}, {type:touch}, {type:interaction}.
• Karada REST for late-join / asset discovery (core/karada_api.py):
  GET /api/karada/state, /api/karada/animations/manifest, /api/karada/animations/resolve,
  GET /api/skins, POST /api/skins/{name}/activate.
• Uploads: POST /api/chat/attachments (image/file), POST /api/audio/upload (STT).
• New mobile interface (interface/mobile_interface.py, auto-discovered — no core edits):
  display_name="Mobile", actions message_mobile / audio_mobile, send_message() delivering
  via push (FCM for Android, APNs for iOS later / WS), register_interface("mobile", …).
  interface_path = mobile/<device_id> where device_id encodes the platform
  (mobile/android:<id> / mobile/ios:<id>) so a single interface serves both platforms.
  Template: interface/telegram_bot.py.
• Assets to cache: per-skin bundles under skins/<name>/ — model.vrm, persona.json,
  preview.png, animations/*.fbx (+ .fbx.json descriptors). URLs are server-root-absolute
  (/skins/*, /avatars/*) → interceptable by path prefix (ignore the ?t= cache-buster).
  three.js currently loads from a CDN via importmap → needs a local mirror for offline.
• TLS: default deploy is HTTPS/8000 with a self-signed cert → app uses cert pinning.

Plan

Phase 0 — Backend foundations (small, server-side)

• Add a minimal embed view of the classic WebUI: new template
  core/webui_templates/synth_webui_embed.html (only the home section + bootstrap, no top nav)
  and a /embed route in core/webui.py (parallel to index / iframe_host).
• Create interface/mobile_interface.py (actions message_mobile / audio_mobile,
  send_message(), initialize_interface() → register_interface("mobile", …),
  interface_path = mobile/<device_id>). (May be deferred to Phase 2.4 if the MVP stays a
  plain /ws client — but this is the clean path and is the only way to get LLM-addressable
  proactive push + a dedicated identity.)
• Docs: update docs/ + AGENTS.md §6 (new interface + embed route).

Phase 1 — MVP Android app (native shell + WebView)

• Kotlin + Jetpack Compose project (API 26+), modular: :app (UI/OS) + :core
  (networking/config/asset-cache — KMP-ready for future iOS).
• Setup/connection screen: host/port, TLS toggle, self-signed cert trust (pinning),
  persisted in DataStore.
• WebView host loading https://<host>/embed: JS, DOM storage, media autoplay, mic
  permission bridge, optional ?token=.
• Local asset cache + interception: WebViewClient.shouldInterceptRequest for /skins/
  and /avatars/ (match on path, ignore ?t=); first-run download via
  /api/karada/animations/manifest; local mirror of three.js + addons.
• Native↔WebView bridge; verify image/audio upload
  (/api/chat/attachments, /api/audio/upload) works inside the WebView.
• Visual parity: classic WebUI theme (WEBUI_ACCENT_COLOR) + home layout render identically;
  native splash/loading.

Phase 2 — OS integrations (post-MVP, independent)

• Home-screen widget: AppWidgetProvider with quick text send + "send image" / "send
  audio" share-target actions. Delivers via the mobile interface or /ws + /api/chat/attachments.
• System assistant (Gemini replacement): VoiceInteractionService /
  VoiceInteractionSessionService + assist role.
• Android Auto: Car App Library (messaging/assistant), TTS-first conversation template.
• Proactive push: FCM in client + interface send_message() publishing to FCM.

Phase 3 — iOS (future, out of scope now)

• If :core is KMP, reuse networking/asset-cache; WebView = WKWebView with a
  WKURLSchemeHandler for asset interception.

Verification

1. Embed backend: open https://<host>/embed in a browser → shows only avatar + chat (no top
   nav), WS connects, chat + TTS work.
2. Interface (if Phase 0.2): scoped uv run pytest tests/interface/test_mobile_interface.py;
   message_mobile appears in /api/components.
3. Python validation: uv run ruff format ., uv run ruff check --fix .,
   uv run ty check core/webui.py interface/mobile_interface.py.
4. MVP app: build APK; point at server; verify (a) VRM avatar renders, (b) bidirectional chat +
   TTS/lipsync, (c) assets served from local cache on 2nd launch, (d) image/audio upload from WebView.
5. TLS: wss to the self-signed cert with pinning; fail deliberately with a wrong cert to confirm pinning.

Scope

• In scope (MVP): native Android shell, WebView on classic WebUI, embed view, local asset
  cache, visual parity, WS + upload, self-signed cert handling.
• Planned (Phase 2): home widget, system assistant, Android Auto, FCM push + mobile interface.
• Out of scope: native VRM renderer (Kotlin/Filament); porting to the Vue frontend; iOS
  (annotated only); classic WebUI redesign.

Open decisions

1. mobile interface now or later? MVP works as a plain /ws client (zero backend). The
   interface enables proactive push + dedicated identity. Recommended: build it in Phase 0.
2. Shell tech for future iOS. Pure Kotlin (faster now, no iOS reuse) vs KMP :core
   (some setup, reusable on iOS). Recommended: Compose UI + KMP-ready :core.
3. Embed via new template vs /iframe/home. /iframe/home exists but drags the home section
   as-is. Recommended: dedicated /embed template.