Support bzip2 and xz compressed inputs in LLM file analysis - #70302
Support bzip2 and xz compressed inputs in LLM file analysis#70302guan404ming wants to merge 3 commits into
Conversation
fcd96ae to
45bb2b6
Compare
45bb2b6 to
3ed4e00
Compare
| import io | ||
| import json | ||
| import logging | ||
| import lzma |
There was a problem hiding this comment.
_bz2 and _lzma are optional CPython extensions, so on an interpreter built without them (the classic pyenv source build without liblzma-dev) this import now takes down the whole module, and with it LLMFileAnalysisOperator and @task.llm_file_analysis, even for the plain and gzip inputs that worked before this PR. Since detect_file_format already keys off _DECOMPRESSORS membership, registering bzip2/xz under try/except ImportError would degrade those builds to the existing "Compression ... is not supported" error instead (pandas guards these two imports the same way).
There was a problem hiding this comment.
Good catch, fixed. Imports now guarded; missing codecs fall back to the unsupported-compression error.
| path = tmp_path / filename | ||
| path.write_bytes(b"content") | ||
|
|
||
| with pytest.raises(LLMFileAnalysisUnsupportedFormatError, match="not supported for"): |
There was a problem hiding this comment.
This match can't tell the two rejection branches apart: the codec-level message ("Compression 'bzip2' is not supported for file analysis.") contains the same substring, so the new sample.avro.bz2 and sample.png.xz cases would still pass if bzip2/xz support were reverted. Something like match=r"not supported for '\w+' file analysis" pins them to the format-combination branch.
There was a problem hiding this comment.
Fixed, tightened the match to pin the format-combination branch.
| ``.csv.gz``. | ||
| - Gzip is not supported for ``.parquet``, ``.avro``, image, or PDF inputs. | ||
| - ``gzip``, ``bzip2``, and ``xz`` compressed text inputs are supported for | ||
| ``.log``, ``.json``, and ``.csv`` (``.log.gz``, ``.csv.bz2``, ``.json.xz``, ...). |
There was a problem hiding this comment.
The code also accepts compressed .txt here (_COMPRESSION_SUPPORTED_FORMATS includes txt, and test_detect_file_format covers notes.txt.gz), so .txt belongs in this list. The Text-like bullet above is missing .txt too since #70431 didn't touch this page, may as well fix both while you're here.
There was a problem hiding this comment.
Fixed, added .txt to both lists.
2 participants
Why
.log.xzor.csv.bz2were rejected outright.How
Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Opus 4.8) following the guidelines