Fix asset event extra filter matching wrong events for dotted keys on SQLite - #69675
Conversation
… SQLite The dialect fallback for the JSON key-value containment filter built an unquoted json_extract path from the user-supplied key, so a key containing JSON-path metacharacters (e.g. spark.executor.memory) was interpreted as nested-object navigation. The filter then silently missed events whose extra contains the literal dotted key and wrongly matched events nesting the same path, diverging from the literal-key containment semantics of PostgreSQL and MySQL. Quote the key in the path so all backends match keys literally.
potiuk
left a comment
There was a problem hiding this comment.
Thanks — this is a good find, and the kind of bug that's very hard to notice in the wild.
_default_json_contains built the SQLite path as f"$.{k}", and SQLite's json_extract treats . and [ as path navigation. So a filter for the literal key "spark.executor.memory" produced $.spark.executor.memory, which:
- missed the event that actually has that literal key, and
- matched an unrelated event nesting
{"spark": {"executor": {"memory": ...}}}
Wrong results in both directions — and silently, since nothing errors. Worse, PostgreSQL's @> and MySQL's JSON_CONTAINS compare literally, so the same filter returned different results depending on the backend. That's the sort of divergence that survives for years because it only reproduces on one deployment type.
json.dumps(k, ensure_ascii=False) is the right fix: SQLite accepts double-quoted key names in JSON paths, and routing through json.dumps gets embedded quotes and backslashes escaped correctly rather than hand-rolling it.
The test is exactly the discriminating one. Creating all three events — the literal dotted key, the nested-object equivalent, and partitions[0] — and asserting the filter matches only the literal is what distinguishes a real fix from one that merely stops erroring. A test with just the dotted key would have passed against several wrong implementations.
Newsfragment is correctly included, and correctly a .bugfix — this changes results users may have unknowingly depended on for SQLite deployments.
Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting
Asset event
extrafiltering (public APIGET /assets/eventsextraparam, and the Execution API asset-event lookups) uses a dialect-aware JSON containment filter. The SQLite fallback built an unquotedjson_extractpath from the user-supplied key, so a key containing JSON-path metacharacters — e.g.spark.executor.memory— was interpreted as nested-object navigation instead of a literal key. Two silent wrong results follow, diverging from PostgreSQL (@>) and MySQL (JSON_CONTAINS), which both match keys literally:{"spark.executor.memory": "4g"}is missed (false negative), and{"spark": {"executor": {"memory": "4g"}}}is wrongly matched (false positive).Same filter, different events returned depending on the database backend — with no error. Both
key=valueparsers pass such keys through verbatim, so this is reachable from both API surfaces.The fix quotes the key in the JSON path (
$."spark.executor.memory"), which SQLite treats literally;json.dumpsproduces exactly the quoting/escaping the JSON path syntax expects (also covering keys containing quotes or backslashes). PostgreSQL and MySQL code paths are untouched.Evidence
source_run_id.test_assets.py(151), execution APItest_asset_events.py(35),test_sqlalchemy.py(23) all pass.Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Fable 5) following the guidelines