Verify that server can send pre-compressed data

[djmitche]

This adds a functional test to verify that the case in #672 -- sending a response containing data that is already compressed -- is supported.

[djmitche]

..now with docs

[asvetlov]

Thanks

[jennolsen84]

is this actually working for you guys? I pulled the latest master, and I can't get it to work :(.

here is the code

import zlib
from aiohttp import web

async def handler(request):
    headers = {'Content-Encoding': 'gzip'}
    deflated_data = zlib.compress(b'mydata')
    return web.Response(body=deflated_data, headers=headers)

app = web.Application()
app.router.add_route('GET', '/', handler)
web.run_app(app)

Then I pointed chrome to http://127.0.0.1:8080/ and that gave me a decode error. similar thing with requests:

DecodeError: ('Received response with content-encoding: gzip, but failed to decode it.', error('Error -3 while decompressing data: incorrect header check',))

[djmitche]

Hm, that test shouldn't pass -- the server sets the content-encoding to gzip, but the go coroutine asserts that it's deflate.

[djmitche]

Ah, indeed, it's not running -- it's missing the run_until_complete. Oops!

[djmitche]

However, even with deflate replaced with zlib, the test still does not pass. What I don't understand is, the compressed data is arriving unchanged, byte-for-byte, at the client:

dustin@personal-devel ~ $ curl -v http://127.0.0.1:8080 | hexdump
* Rebuilt URL to: http://127.0.0.1:8080/
*   Trying 127.0.0.1...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Connected to 127.0.0.1 (127.0.0.1) port 8080 (#0)
> GET / HTTP/1.1
> Host: 127.0.0.1:8080
> User-Agent: curl/7.47.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< CONTENT-ENCODING: gzip
< CONTENT-LENGTH: 14
< DATE: Fri, 10 Jun 2016 12:16:51 GMT
< SERVER: Python/3.5 aiohttp/1.0.3
< 
{ [14 bytes data]
100    14  100    14    0     0   6244      0 --:--:-- --:--:-- --:--:--  7000
* Connection #0 to host 127.0.0.1 left intact
0000000 9c78 adcb 494c 492c 0004 ed08 8102     
000000e

where that 9c 78.. sequence is exactly what zlib.compress(b'mydata') returns. So the only thing I can think is, that's not the right way to perform the compression. And indeed, things work if I change that to

async def handler(request):
    headers = {'Content-Encoding': 'gzip'}
    zlib_mode = 16 + zlib.MAX_WBITS
    zcomp = zlib.compressobj(wbits=zlib_mode)
    deflated_data = zcomp.compress(b'mydata') + zcomp.flush()
    print(repr(deflated_data))
    return web.Response(body=deflated_data, headers=headers)

So perhaps this should be fixed by someone who better understands how zlib works?

[lock]

This thread has been automatically locked since there has not been
any recent activity after it was closed. Please open a new issue for
related bugs.

If you feel like there's important points made in this discussion,
please include those exceprts into that new issue.