Add diagnostic breadcrumbs to investigate intermittent Netty ByteBuf leaks in RNTBD decoders#47272
Add diagnostic breadcrumbs to investigate intermittent Netty ByteBuf leaks in RNTBD decoders#47272FabianMeiswinkel with Copilot wants to merge 68 commits into
Conversation
…mos/encryption/CosmosNettyLeakDetectorFactory.java Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
…to users/fabianm/TestImprovements
…osmosDiagnosticsE2ETest.java Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Initial plan * Improve JavaDoc phrasing in RxDocumentClientImpl Co-authored-by: FabianMeiswinkel <19165014+FabianMeiswinkel@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: FabianMeiswinkel <19165014+FabianMeiswinkel@users.noreply.github.com> Co-authored-by: Fabian Meiswinkel <fabianm@microsoft.com>
…47216) * Initial plan * Move CosmosNettyLeakDetectorFactory.ingestIntoNetty() to class-level static block Co-authored-by: FabianMeiswinkel <19165014+FabianMeiswinkel@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: FabianMeiswinkel <19165014+FabianMeiswinkel@users.noreply.github.com>
…to users/fabianm/NettyBufferLeakFixes
…om/Azure/azure-sdk-for-java into users/fabianm/NettyBufferLeakFixes
…to users/fabianm/NettyBufferLeakFixes
…to users/fabianm/NettyBufferLeakFixes
* Update WebExceptionRetryPolicy.java * Update ThinClientStoreModel.java
…om/Azure/azure-sdk-for-java into users/fabianm/NettyBufferLeakFixes
…Test (#47266) * Initial plan * Improve logging for ByteBufInputStream close failures Change log level from debug to warn and catch Throwable instead of just IOException to make potential ByteBuf leak issues more visible. Co-authored-by: FabianMeiswinkel <19165014+FabianMeiswinkel@users.noreply.github.com> * Fix ByteBuf leak in RetryContextOnDiagnosticTest Changed from Mono.just() to Mono.fromCallable() to defer StoreResponse creation, ensuring ByteBuf lifecycle is properly managed within each subscription rather than eagerly at mock setup time. Co-authored-by: FabianMeiswinkel <19165014+FabianMeiswinkel@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: FabianMeiswinkel <19165014+FabianMeiswinkel@users.noreply.github.com>
…om/Azure/azure-sdk-for-java into users/fabianm/NettyBufferLeakFixes
* Initial plan * Fix memory leak in ServerRntbdRequestDecoder and ServerRntbdContextRequestDecoder When overriding channelRead() in ByteToMessageDecoder and bypassing the parent's decode logic by calling context.fireChannelRead() directly, the ByteBuf reference must be explicitly released to prevent memory leaks. Added ReferenceCountUtil.release() in try-finally blocks to properly manage buffer lifecycle. Co-authored-by: FabianMeiswinkel <19165014+FabianMeiswinkel@users.noreply.github.com> * Replace ReferenceCountUtil.release with safeRelease * Replace ReferenceCountUtil.release with safeRelease --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: FabianMeiswinkel <19165014+FabianMeiswinkel@users.noreply.github.com> Co-authored-by: Fabian Meiswinkel <fabianm@microsoft.com>
…to users/fabianm/NettyBufferLeakFixes
Co-authored-by: FabianMeiswinkel <19165014+FabianMeiswinkel@users.noreply.github.com>
|
Hi @copilot. Thank you for your interest in helping to improve the Azure SDK experience and for your contribution. We've noticed that there hasn't been recent engagement on this pull request. If this is still an active work stream, please let us know by pushing some changes or leaving a comment. Otherwise, we'll close this out in 7 days. |
3 similar comments
|
Hi @copilot. Thank you for your interest in helping to improve the Azure SDK experience and for your contribution. We've noticed that there hasn't been recent engagement on this pull request. If this is still an active work stream, please let us know by pushing some changes or leaving a comment. Otherwise, we'll close this out in 7 days. |
|
Hi @copilot. Thank you for your interest in helping to improve the Azure SDK experience and for your contribution. We've noticed that there hasn't been recent engagement on this pull request. If this is still an active work stream, please let us know by pushing some changes or leaving a comment. Otherwise, we'll close this out in 7 days. |
|
Hi @copilot. Thank you for your interest in helping to improve the Azure SDK experience and for your contribution. We've noticed that there hasn't been recent engagement on this pull request. If this is still an active work stream, please let us know by pushing some changes or leaving a comment. Otherwise, we'll close this out in 7 days. |
Description
Intermittent ByteBuf leaks detected in OrderByDocumentQueryTest trace through
ByteToMessageDecoder.channelRead()in RNTBD protocol handlers. Two decoders overridechannelRead()to conditionally bypass ByteToMessageDecoder's lifecycle management:RntbdRequestDecoderbypasses whenresourceOperationType == 0RntbdContextRequestDecoderbypasses whenresourceOperationType != 0When bypassing via
context.fireChannelRead(message), buffer ownership transfers to downstream handlers without ByteToMessageDecoder's safety net. If downstream fails to release (exceptions, channel closure, timing), buffers leak.Changes
Added diagnostic instrumentation to confirm root cause before implementing fixes:
Buffer Touch Points - Netty's native leak detection labels showing execution path in leak reports:
resourceOperationTypereadTrace Logging - Reference counts, readable bytes, code paths taken (conditional on
logger.isTraceEnabled())Exception Handlers - Added
exceptionCaught()overrides to capture interrupting exceptionsModified Files
RntbdRequestDecoder.java- channelRead() bypass pathRntbdContextRequestDecoder.java- channelRead() bypass pathRntbdResponseDecoder.java- decode() completenessRntbdContextDecoder.java- decode() with exception handlingUsage
Enable with
-Dorg.slf4j.simpleLogger.log.com.azure.cosmos.implementation.directconnectivity.rntbd=TRACELeak reports will show:
Zero production impact - breadcrumbs only activate with TRACE logging.
All SDK Contribution checklist:
General Guidelines and Best Practices
Testing Guidelines
Original prompt
This section details on the original issue you should resolve
<issue_title>[BUG]Investigate possible Netty buffer leak</issue_title>
<issue_description>Describe the bug
Tests in OrderByDocumentQueryTest.java show logs intermittently like below - this is purely based on client implementations - there are two implementations in the sdk/cosmos/azure-cosmos/implementation/directconnectivity and sdk/cosmos/azure-cosmos/implementation/http folders - one using ReactorNettyClient for HTTP and the other using Netty directly with a custom TCP protocol rntbd. Please review the implementations for ByteToMessageDecoder and ChannelHandlers in sdk/cosmos/azure-cosmos for bugs that could explain these intermittent leaks. Most likely due to missing release on discard?
2025-11-13T22:06:20.6753978Z 2025-11-13 22:06:20,657 [reactor-http-epoll-2] ERROR CosmosNettyLeakDetectorFactory - NETTY LEAK (traced) type=ByteBuf records=
2025-11-13T22:06:20.6754181Z
2025-11-13T22:06:20.6754777Z Recent access records:
2025-11-13T22:06:20.6755277Z #1:
2025-11-13T22:06:20.6755856Z io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:300)
2025-11-13T22:06:20.6756462Z io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
2025-11-13T22:06:20.6757072Z io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
2025-11-13T22:06:20.6757662Z io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
2025-11-13T22:06:20.6758257Z io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
2025-11-13T22:06:20.6758815Z io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1519)
2025-11-13T22:06:20.6759392Z io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1390)
2025-11-13T22:06:20.6760098Z io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1430)
2025-11-13T22:06:20.6760714Z io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530)
2025-11-13T22:06:20.6761301Z io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469)
2025-11-13T22:06:20.6761875Z io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
2025-11-13T22:06:20.6762467Z io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
2025-11-13T22:06:20.6763066Z io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
2025-11-13T22:06:20.6763651Z io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
2025-11-13T22:06:20.6764353Z io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1357)
2025-11-13T22:06:20.6764986Z io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
2025-11-13T22:06:20.6765581Z io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
2025-11-13T22:06:20.6766166Z io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:868)
2025-11-13T22:06:20.6766877Z io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799)
2025-11-13T22:06:20.6767463Z io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:501)
2025-11-13T22:06:20.6768038Z io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:399)
2025-11-13T22:06:20.6768631Z io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:998)
2025-11-13T22:06:20.6769240Z io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
2025-11-13T22:06:20.6769984Z io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
2025-11-13T22:06:20.6770555Z java.base/java.lang.Thread.run(Thread.java:1474)
2025-11-13T22:06:20.6771061Z #2:
2025-11-13T22:06:20.6771648Z io.netty.buffer.AdvancedLeakAwareByteBuf.skipBytes(AdvancedLeakAwareByteBuf.java:533)
2025-11-13T22:06:20.6772229Z io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:518)
2025-11-13T22:06:20.6772836Z io.netty.handler.codec.http.HttpClientCodec$Decoder.decode(HttpClientCodec.java:320)
2025-11-13T22:06:20.6773459Z io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530)
2025-11-13T22:06:20.6774045Z io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469)
2025-11-13T22:06:20.6774613Z io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
2025-11-13T22:06:20.6775351Z io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
2025-11-13T22:06:20.677...
💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.