Fix channel backpressure causing keepalive timeout and stream drops#399
Draft
joshkautz wants to merge 1 commit into
Draft
Fix channel backpressure causing keepalive timeout and stream drops#399joshkautz wants to merge 1 commit into
joshkautz wants to merge 1 commit into
Conversation
When the mpsc message channel reaches capacity, the blocking send().await stalls the entire message processing loop. This prevents keepalive ping responses from being processed, causing the camera to interpret the silence as a disconnection and drop the session. This manifests as: - "Reaching limit of channel" warnings followed by stream death - Streams dying after hours of continuous operation - Users resorting to periodic cron restarts as a workaround Three changes to fix this: 1. Increase channel capacities to reduce how often channels fill: - Outgoing message channel: 100 -> 500 - Poll command channel: 200 -> 1000 - Subscriber channels: 100 -> 500 2. Use non-blocking try_send when channel is at capacity. When the channel is full, dropping a video frame is far better than blocking the message loop and losing the entire camera connection. Normal async send is still used when there is available capacity. 3. Drop audio frames when the audio AppSrc buffer is near capacity (>90% full) instead of pushing them and causing cascading backpressure that stalls the video pipeline. Audio buffer overflow was triggering pipeline flushing, which caused full stream reconnection cycles. Addresses QuantumEntangledAndy#349, QuantumEntangledAndy#346, QuantumEntangledAndy#315, and the channel limit warnings in QuantumEntangledAndy#366.
Author
|
Note: These are tweaks I ended up making for my own use case — a 24/7 ALPR system where streams need to stay up indefinitely and audio isn't critical. If the nature of these changes doesn't align with the trajectory or direction you want to take the project, feel free to close this — I'm simply opening it because they solved real stability issues for me and might help others in similar situations. |
janost
added a commit
to MutuallyAssuredDeployment/neolink
that referenced
this pull request
Mar 18, 2026
Upstream PR QuantumEntangledAndy#399. When mpsc channels reach capacity, the blocking send prevented keepalive processing, causing camera session timeouts. Now uses try_send when at capacity (drops one frame) instead of blocking. Also drops audio frames when AppSrc buffer exceeds 90% to prevent cascading backpressure. Channel capacities increased to reduce drop frequency. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
janost
added a commit
to MutuallyAssuredDeployment/neolink
that referenced
this pull request
Mar 19, 2026
Add a section crediting @wafgo, @Maaggs, @fromagge, @lorek123, and @joshkautz for their unmerged upstream PRs (QuantumEntangledAndy#373, QuantumEntangledAndy#389, QuantumEntangledAndy#394, QuantumEntangledAndy#395, QuantumEntangledAndy#396, QuantumEntangledAndy#398, QuantumEntangledAndy#399, QuantumEntangledAndy#400) that were ported into this fork. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
When the internal mpsc message channels reach capacity,
send().awaitblocks the entire message processing loop inPoller::run(). While blocked:This is the root cause of streams dying after hours of continuous operation, which users work around with periodic cron restarts.
Additionally, when RTSP clients cannot consume frames fast enough, audio buffer overflow triggers
FlowError::Flushing, which cascades into the video pipeline causing a full stream reconnection.Fix
1. Increase channel capacities to reduce how often channels fill under normal load:
2. Non-blocking send when channel is full. When
sender.capacity() == 0, usetry_send()instead of the blockingsend().await. Dropping a single video frame is far better than blocking the message loop and losing the entire camera connection. The normal asyncsendpath is preserved when capacity is available.3. Drop audio frames on buffer overflow. When the audio AppSrc buffer exceeds 90% capacity, audio frames are silently dropped instead of being pushed. This prevents cascading backpressure from stalling the video pipeline. Audio buffer overflow was triggering pipeline flushing, which caused full stream reconnection cycles.
Changes
crates/core/src/bc_protocol/connection/bcconn.rs: Increase channel sizes, replace blocking send with try_send at capacitysrc/rtsp/factory.rs: Add audio frame dropping when AppSrc buffer is near capacityRelationship to Other PRs
The audio frame dropping in this PR is complementary to #394, which adds a config option to disable audio entirely. The two approaches serve different use cases:
Both could be merged. Users who disable audio via #394 would never hit the audio drop path in this PR.
Related Issues