Skip to content

Commit 529730e

Browse files
committed
feat(screenpipe): meeting-aware audio session bounds
Capture-node audio was sessionized by blind time rules (60s gap, 30-min hard split), so a 45-minute call became two arbitrary conversations. The collector now supplies real meeting intervals, recorder first: - RecorderMeetingLog mirrors ScreenPipe's persisted meetings table (macOS/Windows watcher) into observation open/close events — retroactive, so meetings recorded while the collector was down still get bounds on backfill. Recent recorder rows suppress the fallback. - MeetingTracker is the Linux fallback (upstream has no Linux sensor): pw-dump running Stream/Input/Audio nodes as the trigger, allowlist classification (native meeting apps; browsers attributed via the active observation's URL), 2-sighting confirm / 30s grace hysteresis. - Forwarded chunks carry meeting_id; group_audio_sessions splits on meeting boundaries, tolerates 5-min in-meeting silences, and replaces the 30-min chop with a 2h safety cap. - --disable-meeting-detector is retired from the recommended record args; clients.migrate_recorder_spec() sweeps it from existing node specs on every fleet update and restarts the recorder once.
1 parent 7c05a9f commit 529730e

13 files changed

Lines changed: 1228 additions & 14 deletions

File tree

‎backends/advanced/src/advanced_omi_backend/routers/modules/device_input_routes.py‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -445,6 +445,7 @@ async def ingest_audio(
445445
device_name: str = Form(...),
446446
direction: Literal["input", "output", "unknown"] = Form(...),
447447
content_hash: str = Form(..., pattern=r"^[0-9a-fA-F]{64}$"),
448+
meeting_id: Optional[str] = Form(None, min_length=1, max_length=200),
448449
source: CaptureSource = Depends(_device_source),
449450
):
450451
existing = await DeviceInputItem.find_one(
@@ -474,6 +475,7 @@ async def ingest_audio(
474475
"device_name": device_name,
475476
"direction": direction,
476477
"duration_seconds": duration_seconds,
478+
**({"meeting_id": meeting_id} if meeting_id else {}),
477479
},
478480
media_data=data,
479481
media_filename=file.filename or "chunk.wav",

‎backends/advanced/src/advanced_omi_backend/services/device_audio_ingest.py‎

Lines changed: 21 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,11 @@
2929
_SESSION_GAP = timedelta(seconds=60)
3030
_CLOSE_DELAY = timedelta(seconds=90)
3131
_MAX_SESSION = timedelta(minutes=30)
32+
# Chunks the collector tagged with the same meeting interval belong to one
33+
# conversation: tolerate longer silences and only split at a safety cap
34+
# instead of the blind 30-minute rule.
35+
_MEETING_SESSION_GAP = timedelta(minutes=5)
36+
_MAX_MEETING_SESSION = timedelta(hours=2)
3237

3338

3439
def _as_utc(value: datetime) -> datetime:
@@ -38,6 +43,11 @@ def _as_utc(value: datetime) -> datetime:
3843
return value.astimezone(timezone.utc)
3944

4045

46+
def _meeting_id(item: DeviceInputItem) -> str | None:
47+
value = item.metadata.get("meeting_id")
48+
return str(value) if value else None
49+
50+
4151
def group_audio_sessions(items: list[DeviceInputItem]) -> list[list[DeviceInputItem]]:
4252
sessions: list[list[DeviceInputItem]] = []
4353
for item in sorted(items, key=lambda row: _as_utc(row.captured_at)):
@@ -48,9 +58,18 @@ def group_audio_sessions(items: list[DeviceInputItem]) -> list[list[DeviceInputI
4858
previous_end = _as_utc(previous.ended_at or previous.captured_at)
4959
session_start = _as_utc(sessions[-1][0].captured_at)
5060
captured_at = _as_utc(item.captured_at)
61+
# A meeting boundary always starts a new session; within one meeting
62+
# the collector's interval is trusted over the blind gap/duration
63+
# rules, up to a safety cap.
64+
same_meeting = _meeting_id(item) is not None and _meeting_id(
65+
item
66+
) == _meeting_id(previous)
67+
gap_limit = _MEETING_SESSION_GAP if same_meeting else _SESSION_GAP
68+
max_session = _MAX_MEETING_SESSION if same_meeting else _MAX_SESSION
5169
if (
52-
captured_at - previous_end > _SESSION_GAP
53-
or captured_at - session_start >= _MAX_SESSION
70+
_meeting_id(item) != _meeting_id(previous)
71+
or captured_at - previous_end > gap_limit
72+
or captured_at - session_start >= max_session
5473
):
5574
sessions.append([item])
5675
else:

‎backends/advanced/tests/test_device_audio_ingest.py‎

Lines changed: 70 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,11 +7,17 @@
77
)
88

99

10-
def item(identifier: str, start: datetime, duration: float = 30):
10+
def item(
11+
identifier: str,
12+
start: datetime,
13+
duration: float = 30,
14+
meeting_id: str | None = None,
15+
):
1116
return SimpleNamespace(
1217
source_item_id=identifier,
1318
captured_at=start,
1419
ended_at=start + timedelta(seconds=duration),
20+
metadata={"meeting_id": meeting_id} if meeting_id else {},
1521
)
1622

1723

@@ -60,6 +66,69 @@ def test_continuous_capture_is_bounded_into_processing_windows():
6066
assert [len(session) for session in sessions] == [30, 2]
6167

6268

69+
def test_meeting_chunks_are_not_split_by_the_thirty_minute_window():
70+
start = datetime(2026, 7, 22, tzinfo=timezone.utc)
71+
rows = [
72+
item(str(index), start + timedelta(minutes=index), meeting_id="meeting:1")
73+
for index in range(45)
74+
]
75+
sessions = group_audio_sessions(rows)
76+
assert [len(session) for session in sessions] == [45]
77+
78+
79+
def test_meeting_boundary_starts_a_new_session():
80+
start = datetime(2026, 7, 22, tzinfo=timezone.utc)
81+
sessions = group_audio_sessions(
82+
[
83+
item("ambient", start),
84+
item("call-1", start + timedelta(seconds=30), meeting_id="meeting:1"),
85+
item("call-2", start + timedelta(seconds=60), meeting_id="meeting:1"),
86+
item("after", start + timedelta(seconds=90)),
87+
]
88+
)
89+
assert [[row.source_item_id for row in session] for session in sessions] == [
90+
["ambient"],
91+
["call-1", "call-2"],
92+
["after"],
93+
]
94+
95+
96+
def test_back_to_back_meetings_stay_separate_conversations():
97+
start = datetime(2026, 7, 22, tzinfo=timezone.utc)
98+
sessions = group_audio_sessions(
99+
[
100+
item("first", start, meeting_id="meeting:1"),
101+
item("second", start + timedelta(seconds=30), meeting_id="meeting:2"),
102+
]
103+
)
104+
assert len(sessions) == 2
105+
106+
107+
def test_meeting_tolerates_longer_silences_than_ambient_capture():
108+
start = datetime(2026, 7, 22, tzinfo=timezone.utc)
109+
sessions = group_audio_sessions(
110+
[
111+
item("one", start, meeting_id="meeting:1"),
112+
item("two", start + timedelta(minutes=4), meeting_id="meeting:1"),
113+
item("three", start + timedelta(minutes=11), meeting_id="meeting:1"),
114+
]
115+
)
116+
assert [[row.source_item_id for row in session] for session in sessions] == [
117+
["one", "two"],
118+
["three"],
119+
]
120+
121+
122+
def test_meetings_still_split_at_the_safety_cap():
123+
start = datetime(2026, 7, 22, tzinfo=timezone.utc)
124+
rows = [
125+
item(str(index), start + timedelta(minutes=4 * index), meeting_id="meeting:1")
126+
for index in range(32)
127+
]
128+
sessions = group_audio_sessions(rows)
129+
assert [len(session) for session in sessions] == [30, 2]
130+
131+
63132
def test_mongo_naive_and_aware_timestamps_group_together():
64133
naive = datetime(2026, 7, 22, 10, 0)
65134
aware = datetime(2026, 7, 22, 10, 0, 30, tzinfo=timezone.utc)

‎clients.py‎

Lines changed: 35 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -482,6 +482,33 @@ def installed_legacy_units() -> list[str]:
482482
# ── update integration ───────────────────────────────────────────────────────
483483

484484

485+
# Flags the recommended recorder invocation no longer carries. The recorder's
486+
# spec argv is per-node state that code updates never regenerate (and the
487+
# tray's capture dialog edits only capture flags), so a retired flag would
488+
# survive every update without this sweep. Currently: the meeting detector
489+
# must stay ON — the collector mirrors the recorder's persisted meetings into
490+
# Chronicle's session bounds (see docs/screenpipe.md).
491+
_RETIRED_RECORDER_FLAGS = ("--disable-meeting-detector",)
492+
493+
494+
def migrate_recorder_spec() -> bool:
495+
"""Drop retired flags from the recorder spec; True if anything changed.
496+
497+
Regenerates the unit/plist and restarts the recorder so the change takes
498+
effect — the one case worth interrupting a live capture session for.
499+
"""
500+
name = "screenpipe"
501+
if not component_installed(name) or not component_spec_path(name).exists():
502+
return False
503+
spec = read_component_spec(name)
504+
argv = [arg for arg in spec["argv"] if arg not in _RETIRED_RECORDER_FLAGS]
505+
if argv == spec["argv"]:
506+
return False
507+
update_component_argv(name, argv)
508+
component_action(name, "restart")
509+
return True
510+
511+
485512
def restart_installed(progress=None) -> list[tuple[str, bool]]:
486513
"""Restart every installed client unit so it picks up updated code.
487514
@@ -492,10 +519,17 @@ def restart_installed(progress=None) -> list[tuple[str, bool]]:
492519
493520
Spec components are skipped: they run a third-party binary this repo does
494521
not ship, so an update gives them nothing and a restart would drop a live
495-
capture session for no reason.
522+
capture session for no reason. The one exception is a retired-flag
523+
migration, which does regenerate and restart the recorder.
496524
"""
497525
progress = progress or (lambda msg: None)
498526
results: list[tuple[str, bool]] = []
527+
try:
528+
if migrate_recorder_spec():
529+
progress("Recorder flags migrated (meeting detector re-enabled)")
530+
results.append(("screenpipe", True))
531+
except Exception:
532+
results.append(("screenpipe", False))
499533
for name in installed_components():
500534
if CLIENT_COMPONENTS[name].get("spec"):
501535
continue

‎docs/screenpipe.md‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -137,6 +137,45 @@ requests zero or one 640px preview; the curator may request one different previe
137137
selected ScreenPipe image is then fetched at a bounded 1280px size. ScreenPipe remains
138138
the high-resolution source and Chronicle never uploads a frame sequence.
139139

140+
## Meeting detection and audio session bounds
141+
142+
Without a meeting signal, the backend can only sessionize forwarded audio by blind
143+
time rules (60 s gap, 30-minute hard split), so a 45-minute call became two arbitrary
144+
conversations. The collector now supplies real meeting intervals from two sources,
145+
recorder first:
146+
147+
1. **The recorder's own meetings table (macOS / Windows — preferred).** ScreenPipe's
148+
meeting watcher (CoreAudio process taps / WASAPI sessions) persists meetings —
149+
with titles — into `db.sqlite`'s `meetings` table. The collector mirrors those
150+
rows into Chronicle events and, because the rows are persisted, this is
151+
**retroactive**: a meeting recorded while the collector was down still gets its
152+
bounds on backfill. The recommended recording flags therefore no longer pass
153+
`--disable-meeting-detector`; use ScreenPipe's `ignored_meeting_apps` for
154+
exclusions. Once the recorder has written a meeting recently, it owns detection
155+
on that node and the collector's own sensor stands down.
156+
2. **The collector's PipeWire tracker (Linux fallback).** Upstream ScreenPipe has no
157+
Linux sensor, so the collector runs the pattern the bot-free note-takers
158+
(Granola, ScreenPipe's watcher) converged on: a process holding a *running*
159+
PipeWire microphone capture stream (`pw-dump`, `media.class =
160+
Stream/Input/Audio`, `state = running` — browsers cork the stream outside calls)
161+
is the trigger; classification is allowlist-only (known meeting apps directly, a
162+
browser attributed through the current observation's `browser_url`, falling back
163+
to `browser-call`), so always-on capturers — ScreenPipe itself, OBS, dictation
164+
tools — can never mint meetings. Hysteresis: two consecutive sightings to open
165+
(pre-join lobbies, mic tests), a 30 s grace before closing at the drop time, and
166+
a stale-sensor close so a dead `pw-dump` cannot pin a meeting open. Live-only:
167+
intervals cover only time the collector was running.
168+
169+
Meeting boundaries ship as ordinary observation open/close events
170+
(`metadata.observation_type = "meeting"`, `detection_source = recorder|pipewire`),
171+
so they land on the timeline and join overlapping conversations through the existing
172+
correlation. Each forwarded audio chunk overlapping a meeting interval — state
173+
persists in `~/.local/state/chronicle-screenpipe/meetings.json` — carries a
174+
`meeting_id`, and the backend's session grouping (`device_audio_ingest.py`) splits
175+
on meeting boundaries, tolerates 5-minute silences inside one meeting, and replaces
176+
the 30-minute split with a 2-hour safety cap. Disable per node at pairing time with
177+
`--no-meeting-detection`.
178+
140179
## Screen context and the `chronicle` fork branch
141180

142181
A completed conversation opens a bounded OCR job over its interval, and the collector
@@ -273,7 +312,9 @@ recorder and Chronicle collector remain independently managed background service
273312
## Implementation map
274313

275314
- `extras/screenpipe-collector/`: collector CLI, observation state machine, checkpoints, and service installation
315+
- `extras/screenpipe-collector/chronicle_screenpipe/meeting.py`: PipeWire meeting detection and audio-chunk tagging
276316
- `backends/advanced/src/advanced_omi_backend/routers/modules/device_input_routes.py`: capture-node ingestion API
317+
- `backends/advanced/src/advanced_omi_backend/services/device_audio_ingest.py`: meeting-aware audio sessionization into conversations
277318
- `backends/advanced/src/advanced_omi_backend/services/observation_curation.py`: sparse preview and vault curation
278319
- `extras/vault-sync/vault_core.py` and `syncthing_manager.py`: vault sync core and Syncthing pairing/control
279320
- `extras/chronicle-tray/chronicle_tray/sections/vault.py`: unified tray UI adapter (imports the vault-sync core in place)

‎extras/screenpipe-collector/README.md‎

Lines changed: 15 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,7 @@ pixels and OCR are retrieved only for bounded jobs requested by Chronicle. See t
2828
--disable-clipboard-capture --capture-scroll true \
2929
--prioritize-input-latency --pause-on-drm-content \
3030
--screenpipe-aec-enabled \
31-
--disable-meeting-detector --disable-telemetry \
31+
--disable-telemetry \
3232
--video-quality balanced --retention-days 90 \
3333
--retention-mode media --api-auth true
3434
```
@@ -46,6 +46,20 @@ pixels and OCR are retrieved only for bounded jobs requested by Chronicle. See t
4646
locally recorded sources are uploaded. The guided setup asks for both the local
4747
capture mode and forwarding mode.
4848

49+
Leave ScreenPipe's meeting detector **enabled** (no
50+
`--disable-meeting-detector`): on macOS and Windows it persists meetings —
51+
with titles — into its own database, and the companion mirrors those rows
52+
into Chronicle and tags forwarded audio chunks with the meeting interval,
53+
so the backend bounds the conversation on the real call instead of fixed
54+
time windows. Because the rows are persisted, bounds survive companion
55+
downtime and backfill retroactively. Use ScreenPipe's
56+
`ignored_meeting_apps` to exclude specific apps. On Linux, where ScreenPipe
57+
has no meeting sensor, the companion detects calls itself from the PipeWire
58+
graph (a known meeting app or browser holding a *running* microphone
59+
stream, browsers attributed through the current observation's URL) —
60+
live-only, so intervals only cover time the companion was running. Disable
61+
all of it at pairing time with `--no-meeting-detection`.
62+
4963
4. Run the companion:
5064

5165
```bash

0 commit comments

Comments
 (0)