Skip to content

Fix asset event extra filter matching wrong events for dotted keys on SQLite#69675

Open
steveahnahn wants to merge 2 commits into
apache:mainfrom
steveahnahn:fix-json-extra-filter-dotted-keys
Open

Fix asset event extra filter matching wrong events for dotted keys on SQLite#69675
steveahnahn wants to merge 2 commits into
apache:mainfrom
steveahnahn:fix-json-extra-filter-dotted-keys

Conversation

@steveahnahn

Copy link
Copy Markdown
Contributor

Asset event extra filtering (public API GET /assets/events extra param, and the Execution API asset-event lookups) uses a dialect-aware JSON containment filter. The SQLite fallback built an unquoted json_extract path from the user-supplied key, so a key containing JSON-path metacharacters — e.g. spark.executor.memory — was interpreted as nested-object navigation instead of a literal key. Two silent wrong results follow, diverging from PostgreSQL (@>) and MySQL (JSON_CONTAINS), which both match keys literally:

  • an event with the literal key {"spark.executor.memory": "4g"} is missed (false negative), and
  • an event nesting the same path {"spark": {"executor": {"memory": "4g"}}} is wrongly matched (false positive).

Same filter, different events returned depending on the database backend — with no error. Both key=value parsers pass such keys through verbatim, so this is reachable from both API surfaces.

The fix quotes the key in the JSON path ($."spark.executor.memory"), which SQLite treats literally; json.dumps produces exactly the quoting/escaping the JSON path syntax expects (also covering keys containing quotes or backslashes). PostgreSQL and MySQL code paths are untouched.

Evidence

  • Fail-first on SQLite: the new bracket-key case fails unfixed (returns 0 events instead of 1), and the identity test fails unfixed by returning the nested event instead of the literal-key event. Notably, the dotted-key count assertion alone passes unfixed because the wrong event coincidentally keeps the count at 1 — hence the added identity assertion on source_run_id.
  • Backend differential: the same identity test on unfixed code passes on PostgreSQL and fails on SQLite; with the fix both agree. Verified on a real PostgreSQL backend via breeze: the unfixed code passes all four new tests on PostgreSQL while two of them fail on SQLite; with the fix, all four pass on both backends.
  • Regression: full test_assets.py (151), execution API test_asset_events.py (35), test_sqlalchemy.py (23) all pass.

Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)
    • Claude Code (Fable 5)

Generated-by: Claude Code (Fable 5) following the guidelines

… SQLite

The dialect fallback for the JSON key-value containment filter built an
unquoted json_extract path from the user-supplied key, so a key containing
JSON-path metacharacters (e.g. spark.executor.memory) was interpreted as
nested-object navigation. The filter then silently missed events whose extra
contains the literal dotted key and wrongly matched events nesting the same
path, diverging from the literal-key containment semantics of PostgreSQL and
MySQL. Quote the key in the path so all backends match keys literally.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant