[BUG] Exception when Cosmos DB HTTP response header is larger than 8192 bytes

[qa-fabio-paulus]

Describe the bug
When the continuation token in the header of the HTTP response from the Cosmos DB is larger than 8192 bytes, the netty that is used by the SDK throws an exception. It seems that this was already fixed for SDK versions <3.0 but the changes were not ported to the SDK 3.x: Azure/azure-cosmosdb-java#24

Exception or Stack Trace

io.netty.handler.codec.TooLongFrameException: HTTP header is larger than 8192 bytes.
	at io.netty.handler.codec.http.HttpObjectDecoder$HeaderParser.newException(HttpObjectDecoder.java:829)
	at io.netty.handler.codec.http.HttpObjectDecoder$HeaderParser.process(HttpObjectDecoder.java:821)
	at io.netty.buffer.AbstractByteBuf.forEachByteAsc0(AbstractByteBuf.java:1306)
	at io.netty.buffer.AbstractByteBuf.forEachByte(AbstractByteBuf.java:1286)
	at io.netty.handler.codec.http.HttpObjectDecoder$HeaderParser.parse(HttpObjectDecoder.java:793)
	at io.netty.handler.codec.http.HttpObjectDecoder.readHeaders(HttpObjectDecoder.java:592)
	at io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:218)
	at io.netty.handler.codec.http.HttpClientCodec$Decoder.decode(HttpClientCodec.java:202)
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:505)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:444)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:283)
	at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1475)
	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1224)
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1271)
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:505)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:444)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:283)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1421)
	... 12 frames truncated

To Reproduce
To reproduce this, you have to make a request that will lead to a large continuation token. I don't know about the internals of the Cosmos DB, but since the state is mapped into the continuation token and larger databases lead to larger states, a large database may be the prerequisite to reproduce this issue. We experience this issue only with multi-partition queries.

Expected behavior
The accepted size of HTTP request/response headers should always be larger than the maximum that can potentially created by the Cosmos DB, whatever this size is. Maybe 32768 bytes as used in the linked issue are sufficient, maybe this is still not enough.

Setup

• OS: Linux Arch with kernel 5.3.7, JRE 11.0.5
• IDE : IntelliJ
• Java SDK: version 3.2.2

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @shurd

[kushagraThapar]

@f-paulus I am investigating this.
Since v3.3.0, we have updated netty and reactor-netty versions to use their latest releases.
While I investigate this on current release, can you please try at your end too if you can reproduce this on v3.3.1 ?

[kirankumarkolli]

FeedOptions.responseContinuationTokenLimitInKb can limit the continuation toke size. As a work-around can you please try setting it <= 8KB?

[qa-fabio-paulus]

@kushagraThapar I can confirm that the same issue persists on v3.3.1.

[qa-fabio-paulus]

@kirankumarkolli Thanks for the hint. I tried it and did set the size of the continuation token to 5kb. With this, I got an empty result instead of the expected set of items. I did a lot of testing and I'm quite sure, that I found another bug. The feed response even did contain a continuation-token together with the empty result list. I filed an issue here: #6115

[apescione]

I'm using Spring Data Cosmos latest (at the moment 2.1.8) and I have the same issue. This version of Spring data uses SDK 3.1.0. Could do you provide also hot fix on this version? When do you think of resolving that? Thank you

[kushagraThapar]

@f-paulus and @apescione - I am prioritizing this in our next sprint, so we are looking this fix to be released somewhere at end of November.
spring-data-cosmosdb release can follow after this gets fixed in Cosmos DB SDK.