gh-43521: Add an opt-in gzip Content-Encoding handler to urllib.request

[soreavis]

This adds urllib.request.HTTPGzipHandler, an opt-in handler that sends Accept-Encoding: gzip and transparently decodes a Content-Encoding: gzip response body. It's the long-standing request in this issue (originally bpo-1508475). Ref #43521.

Design — the thing that kept this open for 20 years wasn't code, it was one unmade design call: an opt-in handler versus default-on decoding down at http.client. This PR is a concrete answer to that call, not a claim on it — the call stays the maintainers'. I went opt-in: the handler is not added to build_opener()'s default handler list, so urlopen() behaves identically for everyone who doesn't ask for it; you get it with build_opener(HTTPGzipHandler). I chose opt-in over on-by-default because the strongest unanswered objection in the thread was Martin Panter's (2014): transparent decoding at this layer would silently decompress a deliberately-gzipped download such as a .tar.gz, and would break partial Range requests. Making the decode the caller's explicit choice sidesteps that. Senthil Kumaran (2012) preferred on-by-default with a flag to switch it off — that's a reasonable alternative; I can flip the default to that if you'd rather — the mechanism is identical either way.

No double-decode: the handler decodes only when it added the Accept-Encoding header itself. If the request already carries a caller-supplied Accept-Encoding, the response is returned untouched — this answers the double-decode objection from Serhiy Storchaka (2012) and Martin Panter (2014), whose own code already decodes gzip from urlopen(). When it does decode, it removes Content-Encoding and Content-Length, which describe the compressed body (Serhiy Storchaka and Senthil Kumaran, 2012, matching RFC guidance).

Streaming: the body is decompressed incrementally with zlib.decompressobj(16 + zlib.MAX_WBITS) as the caller reads, so a read() with no size limit can't bypass the decode (Martin Panter, 2014) and there's no full-buffer decompression bomb — memory is bounded per read. I deliberately did not reuse xmlrpc.client's gzip helper (Martin Panter, 2015): it reads the whole body into a BytesIO and caps it at 20 MiB, which suits XML-RPC payloads but not arbitrary-size urllib downloads. A response made of several concatenated gzip members is decoded in full — not just the first — and trailing bytes that aren't another member are ignored, matching the reader's existing leniency. A truncated gzip stream raises EOFError; a truncated transfer still surfaces http.client.IncompleteRead.

Scope: gzip only. I left deflate out on purpose — its raw-vs-zlib-wrapper ambiguity is a known interop trap, so requesting it is easy but decoding it reliably across servers isn't; it can be a follow-up. zlib is required, and constructing the handler without it raises ImportError, which for an opt-in handler is clearer than a silent no-op.

Tests cover opt-in-ness (the handler is not in build_opener()'s defaults but is accepted explicitly), header injection versus non-override, round-trip decode, chunked/streaming reads, Content-Encoding/Content-Length removal, full pass-through when the caller set their own Accept-Encoding, non-gzip pass-through, concatenated-member decode, trailing-junk tolerance, and the truncated-stream error. test_urllib2, test_urllib and test_urllib2_localnet pass. Docs, a What's New entry and a NEWS blurb are included.

• Issue: transparent gzip compression in urllib #43521

[read-the-docs-community]

Documentation build overview

📚 cpython-previews | 🛠️ Build #33657348 | 📁 Comparing c41270e against main (40f7fbf)

  🔍 Preview build  

3 files changed

± library/urllib.request.html
± whatsnew/3.16.html
± whatsnew/changelog.html