Skip to content

Fix asset event extra filter matching wrong events for dotted keys on SQLite - #69675

Merged
potiuk merged 4 commits into
apache:mainfrom
steveahnahn:fix-json-extra-filter-dotted-keys
Jul 31, 2026
Merged

Fix asset event extra filter matching wrong events for dotted keys on SQLite#69675
potiuk merged 4 commits into
apache:mainfrom
steveahnahn:fix-json-extra-filter-dotted-keys

Conversation

@steveahnahn

Copy link
Copy Markdown
Contributor

Asset event extra filtering (public API GET /assets/events extra param, and the Execution API asset-event lookups) uses a dialect-aware JSON containment filter. The SQLite fallback built an unquoted json_extract path from the user-supplied key, so a key containing JSON-path metacharacters — e.g. spark.executor.memory — was interpreted as nested-object navigation instead of a literal key. Two silent wrong results follow, diverging from PostgreSQL (@>) and MySQL (JSON_CONTAINS), which both match keys literally:

  • an event with the literal key {"spark.executor.memory": "4g"} is missed (false negative), and
  • an event nesting the same path {"spark": {"executor": {"memory": "4g"}}} is wrongly matched (false positive).

Same filter, different events returned depending on the database backend — with no error. Both key=value parsers pass such keys through verbatim, so this is reachable from both API surfaces.

The fix quotes the key in the JSON path ($."spark.executor.memory"), which SQLite treats literally; json.dumps produces exactly the quoting/escaping the JSON path syntax expects (also covering keys containing quotes or backslashes). PostgreSQL and MySQL code paths are untouched.

Evidence

  • Fail-first on SQLite: the new bracket-key case fails unfixed (returns 0 events instead of 1), and the identity test fails unfixed by returning the nested event instead of the literal-key event. Notably, the dotted-key count assertion alone passes unfixed because the wrong event coincidentally keeps the count at 1 — hence the added identity assertion on source_run_id.
  • Backend differential: the same identity test on unfixed code passes on PostgreSQL and fails on SQLite; with the fix both agree. Verified on a real PostgreSQL backend via breeze: the unfixed code passes all four new tests on PostgreSQL while two of them fail on SQLite; with the fix, all four pass on both backends.
  • Regression: full test_assets.py (151), execution API test_asset_events.py (35), test_sqlalchemy.py (23) all pass.

Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)
    • Claude Code (Fable 5)

Generated-by: Claude Code (Fable 5) following the guidelines

… SQLite

The dialect fallback for the JSON key-value containment filter built an
unquoted json_extract path from the user-supplied key, so a key containing
JSON-path metacharacters (e.g. spark.executor.memory) was interpreted as
nested-object navigation. The filter then silently missed events whose extra
contains the literal dotted key and wrongly matched events nesting the same
path, diverging from the literal-key containment semantics of PostgreSQL and
MySQL. Quote the key in the path so all backends match keys literally.
@potiuk potiuk added the ready for maintainer review Set after triaging when all criteria pass. label Jul 11, 2026

@potiuk potiuk left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks — this is a good find, and the kind of bug that's very hard to notice in the wild.

_default_json_contains built the SQLite path as f"$.{k}", and SQLite's json_extract treats . and [ as path navigation. So a filter for the literal key "spark.executor.memory" produced $.spark.executor.memory, which:

  • missed the event that actually has that literal key, and
  • matched an unrelated event nesting {"spark": {"executor": {"memory": ...}}}

Wrong results in both directions — and silently, since nothing errors. Worse, PostgreSQL's @> and MySQL's JSON_CONTAINS compare literally, so the same filter returned different results depending on the backend. That's the sort of divergence that survives for years because it only reproduces on one deployment type.

json.dumps(k, ensure_ascii=False) is the right fix: SQLite accepts double-quoted key names in JSON paths, and routing through json.dumps gets embedded quotes and backslashes escaped correctly rather than hand-rolling it.

The test is exactly the discriminating one. Creating all three events — the literal dotted key, the nested-object equivalent, and partitions[0] — and asserting the filter matches only the literal is what distinguishes a real fix from one that merely stops erroring. A test with just the dotted key would have passed against several wrong implementations.

Newsfragment is correctly included, and correctly a .bugfix — this changes results users may have unknowingly depended on for SQLite deployments.


Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting

@potiuk
potiuk merged commit 664e2a8 into apache:main Jul 31, 2026
84 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:API Airflow's REST/HTTP API area:providers provider:common-sql ready for maintainer review Set after triaging when all criteria pass.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants