[BUG] Enabling http2 configuration in CosmosAsyncClient client produces occasional connection reset exceptions

[vsarris]

Describe the bug
In my company's project we have added http2 configuration in the gateway mode when setting up CosmosDB async client.
Reasoning was to optimize the http requests with the benefits of http2 protocol.

After this setup we have observed the following in our logs occasionally:
WARN [2025-10-17 08:27:25,259] io.netty.channel.DefaultChannelPipeline: An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception. ! java.net.SocketException: Connection reset

Checking the operations outcomes it seems CosmosDB queries are working as expected(createItem, patchItem, queryItems etc all work fine)

Seems like an internal netty issue ? inside the CosmosDB client execution.

Exception or Stack Trace

WARN  [2025-10-17 08:27:25,259] io.netty.channel.DefaultChannelPipeline: An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
! java.net.SocketException: Connection reset
! at java.base/sun.nio.ch.SocketChannelImpl.throwConnectionReset(SocketChannelImpl.java:401)
! at java.base/sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:434)
! at io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:255)
! at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1132)
! at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:356)
! at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:151)
! at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:796)
! at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:732)
! at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:658)
! at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
! at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:998)
! at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
! at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
! at java.base/java.lang.Thread.run(Thread.java:1583)	

To Reproduce
Using the code snippet below perform various CosmosDB container actions(createItem, patchItem, queryItems)
The Azure Cosmos DB account is accessible only through a private endpoint and doesn't export a public endpoint.

Code Snippet
Example code setting up the java client:

   String connectionUri = "ourConnectionEndpoint";
    CosmosClientBuilder builder = new CosmosClientBuilder()
        .credential(new DefaultAzureCredentialBuilder().build())
        .endpoint(connectionUri)
        .consistencyLevel(ConsistencyLevel.SESSION);

    // Configure gateway mode with HTTP/2
    Http2ConnectionConfig http2Config = new Http2ConnectionConfig()
        .setEnabled(true)
        .setMinConnectionPoolSize(1) //default
        .setMaxConnectionPoolSize(1000) //default
        .setMaxConcurrentStreams(30); //default

    builder.gatewayMode(new GatewayConnectionConfig().setHttp2ConnectionConfig(http2Config));
    CosmosAsyncClient cosmosAsyncClient = builder.buildAsyncClient();
 
    //occasionally calling such code in our business logic 
    CosmosPatchOperations operations = CosmosPatchOperations
        .create()
        .replace("/entity/type", "type3")
        .set("/entity/ids", "id1, id2, id3");
    String myItemId = "itemId";
    CosmosAsyncContainer container = cosmosAsyncClient.getDatabase("db")
        .getContainer("container");
    container
        .patchItem(myItemId,
            new PartitionKey("myPartitionKeyValue"),
            operations,
            MyDBItem.class)
        .doOnSuccess(response ->
            LOG.info("charges {}", response.getRequestCharge()))
        .doOnError(e -> {
          LOG.error("Error patching {}", myItemId, e);
        })
        .map(CosmosItemResponse::getItem);

Expected behavior
CosmosDB java sdk actions shouldn't lead to such exceptions that show that something is not performing as expected in the underlying sockets, raising doubts about the operations succeeding or not.

Or in the http2 configuration a way to handle the underlying netty functionality should have been exposed in case the end user needs to adjust any netty configuration.

Screenshots

Setup (please complete the following information):

• OS: [Linux myhostname 5.15.0-1092-azure 0.3.0 release changes #101-Ubuntu SMP Fri Jun 27 23:08:17 UTC 2025 aarch64 aarch64 aarch64 GNU/Linux]
• IDE: [IntelliJ]
• Library/Libraries: [e.g. com.azure:azure-cosmos:4.74.0 , com.azure:azure-identity:1.18.0]
• Java version: ["21.0.8" 2025-07-15 LTS]
• App Server/Environment: [-]
• Frameworks: [Micronaut]

Additional context

Information Checklist
Kindly make sure that you have added all the following information above and checkoff the required fields otherwise we will treat the issuer as an incomplete report

• Bug Description Added
• Repro Steps Added
• Setup information Added

[github-actions]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @kushagraThapar @pjohari-ms @TheovanKraay.

[FabianMeiswinkel]

@vsarris - just to double check - you mentioned above you would not expect to see such exception - i think what you are observing are "just" WARN-level logs in log4j - there is no exception that woudl be thrown to the application, correct? If so, yes this is a knwon issue in the netty-version currently used in the SDK. I am tracking this and will re-evaluate after we update netty versions - but from my investigations this is not having any negative side-effects (no excepitons, no leaks etc.) - just the annoying noisy log.

[vsarris]

Basically it's a bit of a strange case. A warn reporting exceptions so not sure what we can consider this ;p The root indeed it's a WARN so if it is just about the noisy log without side effects, we can live with that at the moment. Thank you!

[FabianMeiswinkel]

Correct - from what I have observed even during long-running stress runs and perf tests etc. is the noisy log - but there are no side effects.