Fix asset event extra filter matching wrong events for dotted keys on SQLite#69675
Open
steveahnahn wants to merge 2 commits into
Open
Fix asset event extra filter matching wrong events for dotted keys on SQLite#69675steveahnahn wants to merge 2 commits into
steveahnahn wants to merge 2 commits into
Conversation
… SQLite The dialect fallback for the JSON key-value containment filter built an unquoted json_extract path from the user-supplied key, so a key containing JSON-path metacharacters (e.g. spark.executor.memory) was interpreted as nested-object navigation. The filter then silently missed events whose extra contains the literal dotted key and wrongly matched events nesting the same path, diverging from the literal-key containment semantics of PostgreSQL and MySQL. Quote the key in the path so all backends match keys literally.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Asset event
extrafiltering (public APIGET /assets/eventsextraparam, and the Execution API asset-event lookups) uses a dialect-aware JSON containment filter. The SQLite fallback built an unquotedjson_extractpath from the user-supplied key, so a key containing JSON-path metacharacters — e.g.spark.executor.memory— was interpreted as nested-object navigation instead of a literal key. Two silent wrong results follow, diverging from PostgreSQL (@>) and MySQL (JSON_CONTAINS), which both match keys literally:{"spark.executor.memory": "4g"}is missed (false negative), and{"spark": {"executor": {"memory": "4g"}}}is wrongly matched (false positive).Same filter, different events returned depending on the database backend — with no error. Both
key=valueparsers pass such keys through verbatim, so this is reachable from both API surfaces.The fix quotes the key in the JSON path (
$."spark.executor.memory"), which SQLite treats literally;json.dumpsproduces exactly the quoting/escaping the JSON path syntax expects (also covering keys containing quotes or backslashes). PostgreSQL and MySQL code paths are untouched.Evidence
source_run_id.test_assets.py(151), execution APItest_asset_events.py(35),test_sqlalchemy.py(23) all pass.Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Fable 5) following the guidelines