Repository navigation
Fix producer memory leak when unable to dispose - #378
Merged
Merged
Conversation
Keep the failed-close fix on current develop without propagating a connection interface. Exercise a concrete ClientCnx over in-memory pipes, stop every producer timer, and fail pending sends and flushes when close fails. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Contributor
Author
|
@Lanayx is there a specific reasoning to not put ClientCnx behind an interface? |
Member
|
@otwl169 Not really, I just didn't find enough justification of introducing the interface |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
While disposing a producer, the client sends CloseProducer to the broker. If that request fails because the connection drops or the broker does not respond, the dispose throws. When it throws, the batch timer continues posting into the stopped mailbox indefinitely and the producer remains registered on the connection.
With a default batch timer of 1ms, this will mean we add a Tick to the mailbox every 1ms, which AI suggests is 110 MB an hour, but I haven't double checked that.
What changed
I moved ClientCnx behind an interface to allow me to write the unit tests on the ProducerImpl for this behaviour. The only production logic changes to the ProducerImpl are on lines 885 and 886 of that file. The change is that a failed close now removes the producer from the connection and stops the producer, which stops the timer.
Successful close behavior is unchanged, allowing an in-flight batch to flush before the broker acknowledges the close.
What I didn't change
I noticed that the inflight messages don't fail in this path, which makes me think anyone waiting to publish a message will be hanging indefinitely. If that is true, I think it should be fixed as well.
Testing plan
Added test for new behaviour that we stop the timer
Added test for new behaviour that the producer is removed from the connection