Skip to content

Exception in client when cosmosDb response headerSize > 8192 #24

Description

@owennewo

By default the io.netty.handler.codec.http.HttpObjectDecoder that you use has a maxHeaderSize of 8092. Headers in response from cosmosDb REST API (e.g. querying documents) can be far bigger than this. Particularly if the response has a continuation header.

In my case a simple select * from events e where e.type='LOGIN' was enough to hit this limit and therefore get the following error.

2200470 [rxnetty-nio-eventloop-3-2] DEBUG com.microsoft.azure.cosmosdb.rx.internal.query.Paginator  - Querying/Reading class com.microsoft.azure.cosmosdb.Documents
2200999 [rxnetty-nio-eventloop-3-2] DEBUG com.microsoft.azure.cosmosdb.rx.internal.ResourceThrottleRetryPolicy  - Operation will NOT be retried. Current attempt 0, Exception: {} 
io.netty.handler.codec.TooLongFrameException: HTTP header is larger than 8192 bytes.
	at io.netty.handler.codec.http.HttpObjectDecoder$HeaderParser.newException(HttpObjectDecoder.java:839)
	at io.netty.handler.codec.http.HttpObjectDecoder$HeaderParser.process(HttpObjectDecoder.java:831)
	at io.netty.buffer.AbstractByteBuf.forEachByteAsc0(AbstractByteBuf.java:1272)
	at io.netty.buffer.AbstractByteBuf.forEachByte(AbstractByteBuf.java:1252)
	at io.netty.handler.codec.http.HttpObjectDecoder$HeaderParser.parse(HttpObjectDecoder.java:803)
	at io.netty.handler.codec.http.HttpObjectDecoder.readHeaders(HttpObjectDecoder.java:603)
	at io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:227)
	at io.netty.handler.codec.http.HttpClientCodec$Decoder.decode(HttpClientCodec.java:202)
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489)

The cosmosdb code is calling no-arg constructor on HttpClientPipelineConfigurator() which uses MAX_HEADER_SIZE_DEFAULT=8192

I think your RxDocumentClientBuilder class needs to have its CompositeHttpClientBuilder tweaked to somehow put a larger configuration. The following is untested but might be on the right track:

PipelineConfigurator clientPipelineConfigurator = new PipelineConfiguratorComposite<HttpClientResponse<ByteBuf>, HttpClientRequest<ByteBuf>>(new HttpClientPipelineConfigurator<ByteBuf, ByteBuf>(4096, 32768, true ), new HttpObjectAggregationConfigurator());

      CompositeHttpClientBuilder<ByteBuf, ByteBuf> builder = new CompositeHttpClientBuilder<ByteBuf, ByteBuf>()
              .withSslEngineFactory(new DefaultSSLEngineFactory())
              .withMaxConnections(connectionPolicy.getMaxPoolSize())
              .withIdleConnectionsTimeoutMillis(this.connectionPolicy.getIdleConnectionTimeoutInMillis())
              .pipelineConfigurator(clientPipelineConfigurator);

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions