[BUG] Can't access a page that is using zstd

[mhalano]

NOTE: IF YOU'RE UNSURE, PLEASE LOOK AT OR OPEN A DISCUSSION INSTEAD! https://github.com/mborsetti/webchanges/discussions

Describe the bug.
A clear and concise description of what the bug is. If it's related to an error, please also attach the traceback
of the exception obtained by running webchanges with -v or -vv.

2025-03-26 14:50:28,028 cli[123378913575168] INFO: webchanges: 3.29.0 Copyright 2020- Mike Borsetti
2025-03-26 14:50:28,028 cli[123378913575168] INFO: CPython: 3.13.2 ('main', 'Mar 13 2025 14:29:07') GCC 14.2.0
2025-03-26 14:50:28,029 cli[123378913575168] INFO: System: Linux-6.14.0-11-generic-x86_64-with-glibc2.41
/home/mhalano/.local/lib/python3.13/site-packages/webchanges/cli.py:248: RuntimeWarning: Hooks file not imported because /home/mhalano/.config/webchanges/hooks.py is not a file
  warnings.warn(f'Hooks file not imported because {hooks_file} is not a file', RuntimeWarning)
2025-03-26 14:50:28,245 storage[123378913575168] INFO: The configuration file /home/mhalano/.config/webchanges/config.yaml is missing directive(s); the following default values are being used:
database:
  engine: sqlite3
  max_snapshots: 4
differ_defaults:
  _note: Default directives that are applied to individual differs.
  ai_google: {}
  command: {}
  deepdiff: {}
  image: {}
  table: {}
  unified: {}
  wdiff: {}
display:
  empty-diff: false
footnote: null
report:
  browser:
    enabled: false
  discord:
    colored: true
    embed: true
    enabled: false
    max_message_length: null
    subject: '[webchanges] {count} changes{jobs_files}: {jobs}'
    webhook_url: ''
  email:
    sendmail:
      path: sendmail
  gotify:
    enabled: false
    priority: 0
    server_url: ''
    title: ''
    token: ''
  html:
    footer: true
    separate: false
    title: '[webchanges] {count} changes{jobs_files}: {jobs}'
  ifttt:
    enabled: false
    event: ''
    key: ''
  mailgun:
    api_key: ''
    domain: ''
    enabled: false
    from_mail: ''
    from_name: ''
    region: us
    subject: '[webchanges] {count} changes{jobs_files}: {jobs}'
    to: ''
  markdown:
    details: true
    footer: true
    minimal: false
    separate: false
  matrix:
    access_token: ''
    enabled: false
    homeserver: ''
    room_id: ''
  prowl:
    api_key: ''
    application: ''
    enabled: false
    priority: 0
    subject: '[webchanges] {count} changes{jobs_files}: {jobs}'
  pushbullet:
    api_key: ''
    enabled: false
  pushover:
    app: ''
    device: null
    enabled: false
    priority: normal
    sound: spacealarm
    user: ''
  run_command:
    command: ''
    enabled: false
  telegram:
    bot_token: ''
    chat_id: ''
    enabled: false
    silent: false
  text:
    separate: false
  tz: null
  webhook:
    enabled: false
    markdown: false
    max_message_length: null
    rich_text: null
    webhook_url: ''
  xmpp:
    enabled: false
    insecure_password: ''
    recipient: ''
    sender: ''
See documentation at https://webchanges.readthedocs.io/en/stable/configuration.html
2025-03-26 14:50:28,245 storage[123378913575168] INFO: Loaded configuration from /home/mhalano/.config/webchanges/config.yaml
2025-03-26 14:50:28,245 storage[123378913575168] INFO: Using sqlite3 3.46.1 database at /home/mhalano/.local/share/webchanges/snapshots.db
2025-03-26 14:50:28,247 jobs[123378913575168] INFO: Job 1: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,248 jobs[123378913575168] INFO: Job 2: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,248 jobs[123378913575168] INFO: Job 3: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,249 jobs[123378913575168] INFO: Job 4: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,249 jobs[123378913575168] INFO: Job 5: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,250 jobs[123378913575168] INFO: Job 6: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,251 jobs[123378913575168] INFO: Job 7: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,251 jobs[123378913575168] INFO: Job 8: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,252 jobs[123378913575168] INFO: Job 9: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,252 jobs[123378913575168] INFO: Job 10: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,253 jobs[123378913575168] INFO: Job 11: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,254 jobs[123378913575168] INFO: Job 12: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,254 jobs[123378913575168] INFO: Job 13: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,255 jobs[123378913575168] INFO: Job 14: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,256 jobs[123378913575168] INFO: Job 15: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,256 jobs[123378913575168] INFO: Job 16: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,257 jobs[123378913575168] INFO: Job 17: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,257 jobs[123378913575168] INFO: Job 18: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,258 jobs[123378913575168] INFO: Job 19: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,258 jobs[123378913575168] INFO: Job 20: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,259 jobs[123378913575168] INFO: Job 21: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,260 jobs[123378913575168] INFO: Job 22: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,260 jobs[123378913575168] INFO: Job 23: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,262 jobs[123378913575168] INFO: Job 28: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,263 jobs[123378913575168] INFO: Job 29: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,264 jobs[123378913575168] INFO: Job 31: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,264 jobs[123378913575168] INFO: Job 32: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,265 jobs[123378913575168] INFO: Job 33: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,265 jobs[123378913575168] INFO: Job 34: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,266 jobs[123378913575168] INFO: Job 35: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,267 jobs[123378913575168] INFO: Job 36: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,267 jobs[123378913575168] INFO: Job 37: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,268 jobs[123378913575168] INFO: Job 38: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,268 jobs[123378913575168] INFO: Job 39: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,269 jobs[123378913575168] INFO: Job 40: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,269 jobs[123378913575168] INFO: Job 41: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,270 jobs[123378913575168] INFO: Job 42: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,270 jobs[123378913575168] INFO: Job 45: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,271 jobs[123378913575168] INFO: Job 46: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,271 jobs[123378913575168] INFO: Job 47: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,272 jobs[123378913575168] INFO: Job 48: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,272 jobs[123378913575168] INFO: Job 49: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,273 jobs[123378913575168] INFO: Job 50: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,274 jobs[123378913575168] INFO: Job 51: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,274 jobs[123378913575168] INFO: Job 52: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,275 jobs[123378913575168] INFO: Job 53: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,276 jobs[123378913575168] INFO: Job 54: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,277 jobs[123378913575168] INFO: Job 55: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,278 jobs[123378913575168] INFO: Job 56: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,279 jobs[123378913575168] INFO: Job 57: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,280 jobs[123378913575168] INFO: Job 58: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,280 jobs[123378913575168] INFO: Job 59: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,281 jobs[123378913575168] INFO: Job 60: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,281 jobs[123378913575168] INFO: Job 61: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,282 jobs[123378913575168] INFO: Job 62: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,282 jobs[123378913575168] INFO: Job 63: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,283 jobs[123378913575168] INFO: Job 64: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,283 jobs[123378913575168] INFO: Job 65: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,284 jobs[123378913575168] INFO: Job 66: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,284 jobs[123378913575168] INFO: Job 67: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,285 jobs[123378913575168] INFO: Job 68: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,286 jobs[123378913575168] INFO: Job 69: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,286 jobs[123378913575168] INFO: Job 70: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,287 jobs[123378913575168] INFO: Job 71: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,287 jobs[123378913575168] INFO: Job 72: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,288 jobs[123378913575168] INFO: Job 73: Replacing deprecated directive 'filter' with 'filters'
2025-03-26 14:50:28,288 storage[123378913575168] INFO: Loaded 73 jobs from /home/mhalano/.config/webchanges/jobs.yaml
2025-03-26 14:50:28,288 handler[123378913575168] INFO: Job 72: https://help.netflix.com/en/node/30081 started processing (UrlJob)
2025-03-26 14:50:28,288 jobs[123378913575168] INFO: Job 72: Sending GET request to https://help.netflix.com/en/node/30081
2025-03-26 14:50:28,289 jobs[123378913575168] INFO: Job 72: Using the HTTPX HTTP client library with HTTP/2 support
2025-03-26 14:50:29,165 _client[123378913575168] INFO: HTTP Request: GET https://help.netflix.com/en/node/30081 "HTTP/2 200 OK"
2025-03-26 14:50:29,211 handler[123378913575168] INFO: Job 72: Job ended with error; incrementing cumulative error runs to 3
2025-03-26 14:50:29,211 handler[123378913575168] INFO: Job 72: https://help.netflix.com/en/node/30081 ended processing
2025-03-26 14:50:29,211 handler[123378913575168] INFO: Job 72: Got exception while processing job <url url='https://help.netflix.com/en/node/30081' headers={} ignore_cached=True method='GET' ignore_connection_errors=True ignore_http_error_codes='4xx, 5xx' ignore_timeout_errors=True filters=[{'xpath': '/html/body/div[1]/div[4]/div/div[2]/div/div[1]/section/form[6]/div/div/div[2]/table/tbody'}, {'html2text': None}, {'striplines': None}] index_number=72 name="Changes on Netflix's resolution on Linux"
Traceback (most recent call last):
  File "/home/mhalano/.local/lib/python3.13/site-packages/httpx/_decoders.py", line 185, in decode
    output.write(self.decompressor.decompress(data))
                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
zstd.ZstdError: cannot use a decompressobj multiple times

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/mhalano/.local/lib/python3.13/site-packages/webchanges/handler.py", line 231, in process
    data, self.new_etag, mime_type = self.job.retrieve(self, headless)
                                     ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
  File "/home/mhalano/.local/lib/python3.13/site-packages/webchanges/jobs.py", line 1066, in retrieve
    data, etag, mime_type = self._retrieve_httpx(headers=headers, timeout=timeout)
                            ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mhalano/.local/lib/python3.13/site-packages/webchanges/jobs.py", line 773, in _retrieve_httpx
    response = client.request(
        method=self.method,  # type: ignore[arg-type]
    ...<2 lines>...
        params=self.params,
    )
  File "/home/mhalano/.local/lib/python3.13/site-packages/httpx/_client.py", line 825, in request
    return self.send(request, auth=auth, follow_redirects=follow_redirects)
           ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/mhalano/.local/lib/python3.13/site-packages/httpx/_client.py", line 928, in send
    raise exc
  File "/home/mhalano/.local/lib/python3.13/site-packages/httpx/_client.py", line 922, in send
    response.read()
    ~~~~~~~~~~~~~^^
  File "/home/mhalano/.local/lib/python3.13/site-packages/httpx/_models.py", line 881, in read
    self._content = b"".join(self.iter_bytes())
                    ~~~~~~~~^^^^^^^^^^^^^^^^^^^
  File "/home/mhalano/.local/lib/python3.13/site-packages/httpx/_models.py", line 898, in iter_bytes
    decoded = decoder.decode(raw_bytes)
  File "/home/mhalano/.local/lib/python3.13/site-packages/httpx/_decoders.py", line 191, in decode
    raise DecodingError(str(exc)) from exc
httpx.DecodingError: cannot use a decompressobj multiple times
2025-03-26 14:50:29,214 reporters[123378913575168] INFO: Submitting with stdout (<class 'webchanges.reporters.StdoutReporter'>)
2025-03-26 14:50:29,223 command[123378913575168] INFO: Exiting with exit code 0
===========================================================================
01. ERROR: Changes on Netflix's resolution on Linux
===========================================================================

---------------------------------------------------------------------------
ERROR: Changes on Netflix's resolution on Linux (https://help.netflix.com/en/node/30081)
---------------------------------------------------------------------------

cannot use a decompressobj multiple times
---------------------------------------------------------------------------


--
Checked 1 source in 1.1 seconds with webchanges 3.29.0.

Steps to reproduce.
Include relevant snippets of anything required (e.g. jobs and configuration files) for anyone to fully reproduce the
bug.
Try to retrieve the Netflix's URL.

Expected behavior.
A clear and concise description of what you expected to happen.
Should return some values.

Screen scrape/screenshots.
If applicable, add screen scrape or screenshots to help explain the bug; please use -v or -vv to capture
logging.

Version info.
Please run webchanges --detailed-versions and paste the output here.

➜ webchanges --detailed-versions
/home/mhalano/.local/lib/python3.13/site-packages/webchanges/cli.py:248: RuntimeWarning: Hooks file not imported because /home/mhalano/.config/webchanges/hooks.py is not a file
  warnings.warn(f'Hooks file not imported because {hooks_file} is not a file', RuntimeWarning)
Software:
• webchanges: 3.29.0
• CPython: 3.13.2 ('main', 'Mar 13 2025 14:29:07') GCC 14.2.0
• SQLite: 3.46.1

System:
• Platform: Linux-6.14.0-11-generic-x86_64-with-glibc2.41, x86_64
• Processor: x86_64
• CPUs (logical): 16
• Free memory: 43.5G physical plus 0.0B swap.
• Free disk '/': 201.8G (45.4%)
• --max-threads default: 20

Installed PyPi dependencies:
• cssselect: 1.3.0
• h2: 4.2.0
• html2text: 2024.2.26
• httpx: 0.28.1
  - anyio: 4.9.0
  - certifi: 2025.1.31
  - idna: 3.10
• markdown2: 2.5.3
• msgpack: 1.1.0
• platformdirs: 4.3.7
• pyyaml: 6.0.2
• zstandard: 0.23.0

Installed dpkg dependencies:
• jq
   - jq: 1.7.1-3build1
• Pillow
   - libjpeg-dev: 8c-2ubuntu11
   - zlib1g-dev: 1:1.3.dfsg+really1.3.1-1ubuntu1
   - libtiff-dev: 4.5.1+git230720-4ubuntu4
   - libfreetype-dev: 2.13.3+dfsg-1
   - libwebp-dev: 1.5.0-0.1
   - libimagequant-dev: 2.18.0-1build1
   - libraqm-dev: 0.10.2-1
   - libxcb1-dev: 1.17.0-2

Additional context.
Add any other context about the problem.

[mhalano]

I understand the problem came from httpx, but maybe it's related to how the library is being used.

[mborsetti]

Thanks for the report.

Yes, it's definitely from httpx, and appreciate you reporting it. It has been reported upstream here.

In the interim, you can add the directive headers: { accept-encoding: gzip, deflate, br } to the job. This will override the header httpx uses with this one, which omits zstd.

[lovelydinosaur]

Thanks for pointing this out... it'll reproduce with either httpx or requests.

It's a bug lying somewhere in between Python's zstandard implementation and Netflix's server-side zstandard implementation.

[mborsetti]

Hi @mhalano,

I'm no longer seeing this bug. And I have confirmed (with debugging tools) that the page is being served using zstd ('content-encoding', 'zstd').

Please let me know if it's still a bug for you.

[mhalano]

Hi @mborsetti, Yeah, I can confirm that. I disabled the headers I put as you requested, and I got the page content correctly.

Do you know what changed? Maybe httpx version? Maybe something on Netflix side?

[mborsetti]

Do you know what changed? Maybe httpx version? Maybe something on Netflix side?

Don't know, but both httpx and zstandard are the same version (no updates since May), and @tomchristie reported that requests was also affected. So it's probably on Netflix's side, or maybe in some underlying Python library.