Input HTTP server with sync_response, handling of `Accept-Encoding: gzip` header can give issues

[EmilLaursen]

The HTTP input, with sync_response, interprets a Accept-Encoding: gzip header as follows.

• A Content-Encoding: gzip header is set on the response
• The message is gzipped

See here

Consider this configuration:

http:
  enabled: true
  address: 0.0.0.0:4195
  root_path: /benthos

input:
  http_server:
    path: /lol
    allowed_verbs:
      - POST
    sync_response:
      metadata_headers:
        include_prefixes:
          - "Content-Length"

pipeline:
  threads: 1
  processors:
    - bloblang: |
        root = "bestdata".bytes()
    - bloblang: |
        meta "Content-Length" = content().length()

output:
  sync_response: {}

We set the content-length metadata field and propagate it in the sync_response. Now this will work:

curl -v "http://localhost:4195/lol" --data-binary "data"

But this will fail

curl -v "http://localhost:4195/lol" --data-binary "data" -H "Accept-Encoding: gzip"

because the content-length is no longer correct, due to the gzipHandler.

One way to handle this is to have the gzip handler remove any "Content-Length" header. The golang server will then probably fallback to using Transfer-Encoding: chunked, and things should work.

Another option is to just chalk it up as a user error :)

[Jeffail]

Hey @EmilLaursen, thanks for raising this, I'll take a look as we can at the very least capture the incorrect headers and remove them, I doubt that's going to break other configurations.