LEAK: ByteBuf.release() was not called before it's garbage-collected.

[li-zoe-x]

What version of gRPC-Java are you using?

1.70.0

What is your environment?

linux
jdk 17

What did you expect to see?

no leak

What did you see instead?

LEAK: ByteBuf.release() was not called before it's garbage-collected.

Steps to reproduce the bug

my log:
i.n.u.ResourceLeakDetector : LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records:
Created at:
io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:404)
io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:188)
io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:124)
io.grpc.netty.NettyWritableBufferAllocator.allocate(NettyWritableBufferAllocator.java:51)
io.grpc.internal.MessageFramer.writeKnownLengthUncompressed(MessageFramer.java:226)
io.grpc.internal.MessageFramer.writeUncompressed(MessageFramer.java:172)
io.grpc.internal.MessageFramer.writePayload(MessageFramer.java:143)
io.grpc.internal.AbstractStream.writeMessage(AbstractStream.java:70)
io.grpc.internal.ServerCallImpl.sendMessageInternal(ServerCallImpl.java:168)
io.grpc.internal.ServerCallImpl.sendMessage(ServerCallImpl.java:152)
io.grpc.stub.ServerCalls$ServerCallStreamObserverImpl.onNext(ServerCalls.java:380)

[kannanjgithub]

This is most likely caused by multiple server threads writing to the same StreamObserver without synchronization that causes mismatch with Netty reference counting.

Separate StreamObservers do not need to be synchronized together; incoming and outgoing directions are independent. Since individual StreamObservers are not thread-safe, if multiple threads will be writing to a StreamObserver concurrently, the application must synchronize calls.

[li-zoe-x]

This is most likely caused by multiple server threads writing to the same StreamObserver without synchronization that causes mismatch with Netty reference counting.

Separate StreamObservers do not need to be synchronized together; incoming and outgoing directions are independent. Since individual StreamObservers are not thread-safe, if multiple threads will be writing to a StreamObserver concurrently, the application must synchronize calls.

ths, we are trying this

[fishjam]

I might meet same issue. when I use nativemem in async-profiler to check my app. there is similar leak report.

grpc version

1.22.2

[kannanjgithub]

When a writer of the stream doesn't call onCompleted/onError on the outgoing stream observer to signify the end of the RPC, it will result in the writable buffer leak. If cancellation is initiated by the other side, an onError on the incoming stream observer will be called, which can then call onCompleted on the outbound stream observer.
Channel shutdown without awaitTermination till the streams have been cancelled can also result in both stream leak and byte buffer leak.

(I have struck out my earlier reply since it is likely to result in some other error, not the reported errors)

[kannanjgithub]

I hope your issue is resolved. Feel free to comment to reopen in the future if needed.