-
Notifications
You must be signed in to change notification settings - Fork 5
Add multi-bot voice architecture with audio channels and progressive updates #36
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
odilitime
wants to merge
60
commits into
1.x
Choose a base branch
from
odi-voice
base: 1.x
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from 25 commits
Commits
Show all changes
60 commits
Select commit
Hold shift + click to select a range
ca7451d
fix(plugin-discord): merge multi-bot support with printBanner from up…
odilitime 9e4eb68
fix(plugin-discord): consolidate IDiscordService interface and add co…
odilitime 91168f9
fix(plugin-discord): merge multi-client registry, permission audit ev…
odilitime d002915
fix(plugin-discord): merge audio channel system, use runtime.logger c…
odilitime 5ad4833
fix(plugin-discord): add attachment handling to progressive update path
odilitime ef2c11a
fix(plugin-discord): add component type handling and fix null filtering
odilitime 912bb86
chore(plugin-discord): update attachments module
odilitime 5cd2a1b
chore(plugin-discord): update environment module
odilitime 5a4e3ba
chore(plugin-discord): remove unused UUID import
odilitime 10c39a8
chore(plugin-discord): update tests module
odilitime 676e11a
test(plugin-discord): update MessageManager test mocks
odilitime f99221a
feat(plugin-discord): add ClientRegistry for multi-bot support
odilitime 72032ba
feat(plugin-discord): add IAudioSink and IAudioBroadcast contracts
odilitime 9a18be9
feat(plugin-discord): add DiscordAudioSink implementation
odilitime 1575dab
feat(plugin-discord): add audio channel priority system
odilitime b0bdeb8
feat(plugin-discord): add progressive message updates
odilitime 1fa716a
feat(plugin-discord): add voice connection manager
odilitime 63bbe6a
feat(plugin-discord): add audioState provider
odilitime bc38752
feat(plugin-discord): add setListeningActivity and setVoiceChannelSta…
odilitime 27bc17c
test(plugin-discord): add VoiceManager tests
odilitime 4c22966
test(plugin-discord): add multi-bot audio tests
odilitime 79b5827
test(plugin-discord): add progressive message tests
odilitime 275907c
test(plugin-discord): add token validation tests
odilitime d18416a
docs(plugin-discord): update README with multi-bot and audio features
odilitime 693f58a
docs(plugin-discord): add architecture documentation
odilitime 86c2d1d
token can't be null
odilitime 8fefa21
Unused import logger.
odilitime 3b18151
Unused variable playPromise.
odilitime 96776a3
Use setPresence({ activities: [] }) to clear bot activity instead of …
odilitime f266fab
Incomplete cleanup: missing timer/bridge cleanup.
odilitime a1de596
Bug: Event emission accesses deleted map entry.
odilitime d6f836f
VoiceTarget play and stop methods should handle errors gracefully.
odilitime 5edb78d
Merge branch '1.x' of https://github.com/elizaos-plugins/plugin-disco…
odilitime 2967be9
fix(plugin-discord): resolve merge conflicts and improve logging
odilitime c081fa6
fix(plugin-discord): fix audio sink reconnection and voice settings p…
odilitime f8eab34
fix(plugin-discord): multiple bug fixes and improvements
odilitime 2d2983b
fix(plugin-discord): pre-populate activeConnections with existing voi…
odilitime cb8fab8
fix(plugin-discord): fix test imports and remove external plugin depe…
odilitime 308fa93
fix(plugin-discord): add LLM response validation in setListeningActivity
odilitime 743af18
improve backwards support
odilitime feb536b
fix: resolve merge conflicts and add new providers
odilitime 8ddc016
fix: resolve merge conflicts and add new providers
odilitime e8e1374
fix: resolve merge conflict in service.ts - add generateInviteUrl import
odilitime 058453c
fix: forbidden emoji fallback and multi-token config check
odilitime 87e358f
fix: improve emoji fallback and standardize tests on bun:test
odilitime 0c1ef0f
fix: remove unused generateInviteUrl import from service.ts
odilitime 3396087
fix: filter base64 images from memories to prevent context bloat
odilitime a4c2ed4
fix: add voiceManager null guards in auto-join logic
odilitime 9c5f613
fix(plugin-discord): reduce log spam and add DISCORD_SERVICE_NAME con…
odilitime d1c15fb
fix(plugin-discord): track auto-join timeout and use correct voice ma…
odilitime 7a7ba8c
feat(plugin-discord): use character name in logs instead of agentId
odilitime ae4d8e1
fix(plugin-discord): use UUID for Memory.agentId and fix member fetch…
odilitime 81a6da2
fix(plugin-discord): prevent race condition in progressive message up…
odilitime d163416
fix(plugin-discord): use UUID for Memory.agentId in voice.ts
odilitime 93fb31f
feat(discord): reply context, agent role provider, backwards compat, …
odilitime 4f96e02
fix(discord): setVoiceChannelStatus backwards compat, isDataUrl dedup…
odilitime 760f5ee
fix(discord): add Partials for DM message support
odilitime a0773c9
fix(discord): emoji fallback and client cleanup on failed registration
odilitime be12cf2
fix(discord): await pending logins before destroyAll cleanup
odilitime efb42f9
fix(discord): prevent listener leaks and race conditions in loginBot
odilitime File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,224 @@ | ||
| # Coordination Patterns for Music Playback | ||
|
|
||
| > **Status**: Implemented (Broadcast Architecture) | ||
| > **Last Updated**: December 2025 | ||
|
|
||
| ## Overview | ||
|
|
||
| This document describes the coordination patterns used in the music playback system, based on the broadcast architecture with `IAudioBroadcast` and `IAudioSink` contracts. | ||
|
|
||
| ## Architecture Summary | ||
|
|
||
| ``` | ||
| ┌─────────────────────────────────────────────────────────┐ | ||
| │ plugin-music-player │ | ||
| │ MusicQueue → Broadcast (IAudioBroadcast) → Multiplex │ | ||
| └─────────────────────────────────────────────────────────┘ | ||
| │ | ||
| Events: 'track:started', 'track:finished' | ||
| Method: subscribe() / unsubscribe() | ||
| │ | ||
| ┌─────────────────────────▼───────────────────────────────┐ | ||
| │ plugin-discord │ | ||
| │ DiscordAudioSink (IAudioSink) ← feed(stream) │ | ||
| │ Events: 'statusChange' │ | ||
| └─────────────────────────────────────────────────────────┘ | ||
| ``` | ||
|
|
||
| ## Pattern 1: Contract-Based Decoupling | ||
|
|
||
| ### Problem | ||
| Plugins need to communicate without tight coupling. | ||
|
|
||
| ### Solution | ||
| Define contracts (interfaces) that each plugin owns: | ||
|
|
||
| ```typescript | ||
| // plugin-music-player owns IAudioBroadcast | ||
| interface IAudioBroadcast { | ||
| subscribe(consumerId: string): AudioSubscription; | ||
| unsubscribe(consumerId: string): void; | ||
| feedAudio(stream: Readable, metadata?: AudioBroadcastMetadata): Promise<void>; | ||
| // ... | ||
| } | ||
|
|
||
| // plugin-discord owns IAudioSink | ||
| interface IAudioSink { | ||
| feed(stream: Readable): Promise<void>; | ||
| connect(channelId: string): Promise<void>; | ||
| disconnect(): Promise<void>; | ||
| // ... | ||
| } | ||
| ``` | ||
|
|
||
| ### Benefits | ||
| - Plugins only depend on contracts, not implementations | ||
| - Either plugin can be replaced or upgraded independently | ||
| - Clear ownership boundaries | ||
|
|
||
| ## Pattern 2: Event-Based Coordination | ||
|
|
||
| ### Problem | ||
| External plugins (radio, DJ) need to know when tracks start/finish. | ||
|
|
||
| ### Solution | ||
| Use EventEmitter for state change notifications: | ||
|
|
||
| ```typescript | ||
| // IAudioBroadcast emits events | ||
| broadcast.on('track:started', (metadata) => { | ||
| console.log(`Now playing: ${metadata.title}`); | ||
| }); | ||
|
|
||
| broadcast.on('track:finished', (metadata) => { | ||
| // Trigger next action | ||
| }); | ||
|
|
||
| broadcast.on('silence:started', () => { | ||
| // Queue is empty | ||
| }); | ||
|
|
||
| // IAudioSink emits status changes | ||
| sink.on('statusChange', (status) => { | ||
| // 'connected' | 'disconnected' | 'connecting' | 'error' | ||
| }); | ||
| ``` | ||
|
|
||
| ### Use Cases | ||
| - Radio plugin listening for tracks to announce | ||
| - DJ plugin waiting for track finish to add commentary | ||
| - Web UI updating now-playing display | ||
|
|
||
| ## Pattern 3: Auto-Wiring via Service Discovery | ||
|
|
||
| ### Problem | ||
| Plugins need to connect to each other at runtime. | ||
|
|
||
| ### Solution | ||
| Use `runtime.getService()` for discovery and wire automatically: | ||
|
|
||
| ```typescript | ||
| // In MusicService (plugin-music-player) | ||
| async autoSubscribeDiscord(guildId: string, broadcast: IAudioBroadcast) { | ||
| const discordService = this.runtime.getService('discord'); | ||
| if (!discordService) return; // Graceful degradation | ||
|
|
||
| const sink = discordService.getAudioSink(guildId); | ||
| if (!sink) return; | ||
|
|
||
| // Auto-wire on connection | ||
| sink.on('statusChange', async (status) => { | ||
| if (status === 'connected') { | ||
| const subscription = broadcast.subscribe(`discord-${guildId}`); | ||
| await sink.feed(subscription.stream); | ||
| } | ||
| }); | ||
| } | ||
| ``` | ||
|
|
||
| ### Benefits | ||
| - Zero manual configuration | ||
| - Works when both plugins loaded | ||
| - Gracefully degrades when one is missing | ||
|
|
||
| ## Pattern 4: Reconnection Handling | ||
|
|
||
| ### Problem | ||
| Network hiccups cause Discord disconnections; playback should resume. | ||
|
|
||
| ### Solution | ||
| Use `statusChange` events for automatic recovery: | ||
|
|
||
| ```typescript | ||
| sink.on('statusChange', async (status) => { | ||
| if (status === 'connected') { | ||
| // Re-subscribe to get fresh stream from live point | ||
| const subscription = broadcast.subscribe(`discord-${guildId}`); | ||
| await sink.feed(subscription.stream); | ||
| logger.info('Discord reconnected, re-subscribed to broadcast'); | ||
| } | ||
| }); | ||
| ``` | ||
|
|
||
| ### Key Insight | ||
| Re-subscribing gets the current position in the broadcast, not the beginning. This is because the broadcast is always "live" - like tuning into a radio station. | ||
|
|
||
| ## Pattern 5: Non-Blocking Multiplexing | ||
|
|
||
| ### Problem | ||
| Slow consumers (laggy web clients) could block the main stream. | ||
|
|
||
| ### Solution | ||
| Each consumer gets an independent `PassThrough` stream with backpressure handling: | ||
|
|
||
| ```typescript | ||
| // In StreamMultiplexer | ||
| source.on('data', (chunk) => { | ||
| for (const [id, consumer] of consumers) { | ||
| if (!consumer.write(chunk)) { | ||
| // Consumer buffer full - drop frame for this consumer | ||
| logger.debug(`Backpressure on ${id}, dropping chunk`); | ||
| } | ||
| } | ||
| }); | ||
| ``` | ||
|
|
||
| ### Result | ||
| - Discord playback unaffected by web client performance | ||
| - Each consumer independent | ||
| - No blocking the source stream | ||
|
|
||
| ## Pattern 6: Silence Injection | ||
|
|
||
| ### Problem | ||
| Empty queue causes Discord voice connection timeout. | ||
|
|
||
| ### Solution | ||
| `StreamCore` injects silence frames when no audio is being fed: | ||
|
|
||
| ```typescript | ||
| // In StreamCore | ||
| startSilence() { | ||
| this.silenceInterval = setInterval(() => { | ||
| this.output.write(OPUS_SILENCE_FRAME); // 10ms of silence | ||
| }, 10); | ||
| } | ||
|
|
||
| feed(stream: Readable) { | ||
| this.stopSilence(); // Real audio coming | ||
| stream.pipe(this.output, { end: false }); | ||
| stream.on('end', () => this.startSilence()); | ||
| } | ||
| ``` | ||
|
|
||
| ### Benefits | ||
| - Voice connection stays alive indefinitely | ||
| - Seamless transition when new tracks added | ||
| - No manual connection management needed | ||
|
|
||
| ## Pattern Summary | ||
|
|
||
| | Pattern | Use Case | Key Mechanism | | ||
| |---------|----------|---------------| | ||
| | **Contracts** | Plugin decoupling | IAudioBroadcast / IAudioSink | | ||
| | **Events** | State notifications | EventEmitter | | ||
| | **Auto-Wiring** | Runtime connection | runtime.getService() | | ||
| | **Reconnection** | Resilience | statusChange event | | ||
| | **Multiplexing** | Multiple consumers | PassThrough + backpressure | | ||
| | **Silence** | Connection keep-alive | Interval-based frame injection | | ||
|
|
||
| ## Implementation Locations | ||
|
|
||
| | Pattern | File | | ||
| |---------|------| | ||
| | IAudioBroadcast | `plugin-music-player/src/contracts.ts` | | ||
| | IAudioSink | `plugin-discord/src/contracts.ts` | | ||
| | Auto-wiring | `plugin-music-player/src/service.ts` | | ||
| | Multiplexing | `plugin-music-player/src/core/streamMultiplexer.ts` | | ||
| | Silence injection | `plugin-music-player/src/core/streamCore.ts` | | ||
| | Discord sink | `plugin-discord/src/sinks/discordAudioSink.ts` | | ||
|
|
||
| ## Related Documentation | ||
|
|
||
| - [MUSIC_ARCHITECTURE.md](./MUSIC_ARCHITECTURE.md) - Full architecture overview | ||
| - [plugin-music-player README](../plugin-music-player/README.md) - Usage guide | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.