Skip to content

fix(data-quality): tableColumnToMatchSet ordered="false" treated as True (#35616) - #35663

Merged
harshsoni2024 merged 4 commits into
open-metadata:mainfrom
zerafachris:fix/35616-ordered-string-to-bool
Oct 29, 2026
Merged

harshsoni2024 merged 4 commits into
open-metadata:mainfrom
zerafachris:fix/35616-ordered-string-to-bool

Conversation

@zerafachris

@zerafachris zerafachris commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Why

bool("false") returns True in Python — any non-empty string is truthy. tableColumnToMatchSet._run_validation was reading the ordered parameter with get_test_case_param_value(..., bool, default=False), so passing ordered: "false", "False" or "0" all silently enabled ordered matching. The only way to get unordered comparison was to omit the parameter entirely.

What changed

Switched from get_test_case_param_value(..., bool) to the existing get_bool_test_case_param helper (in validations/utils.py) which does str.lower() == "true", so every falsy string variant correctly yields False.

No other files changed.

Tests

Added test_case_column_to_match_set_ordered_false fixture and parametrized test cases in both the SQLite (database) and DataFrame (datalake) test paths:

ordered value before (bug) after (fix)
"false" True (ordered) False (unordered) ✓
"False" True False ✓
"true" True True ✓
omitted False False ✓

The new ordered_false test lists all 11 table columns in shuffled order (different from DDL order) and expects Success, which would have been Failed with the old code because ordered comparison would have seen a mismatched sequence.

Fixes #35616

Prepared with AI assistance (Claude Code, Anthropic), reviewed for correctness before submission.

🤖 Generated with Claude Code

RetriggerView in GreptileConfidence Score: 5/5

The PR appears safe to merge; no actionable issues were found.

Summary

Fixes tableColumnToMatchSet so ordered="false" uses unordered matching.

  • Table-column checks treat ordered: "false" as unordered matching.

No new actionable issues found. Tests were not run.

Reviews (3) · Last reviewed commit: "test(data-quality): use ORM column keys ..." · Reviewed by Greptile


Summary by Gitar

  • Ingestion patch:
    • Converted patch.patch to a list in build_patch before array fields are appended in patch_request.py

This will update automatically on new commits.

… in tableColumnToMatchSet

bool(\"false\") returns True in Python because any non-empty string is truthy.
tableColumnToMatchSet was calling get_test_case_param_value(..., bool, default=False)
for the \"ordered\" parameter, meaning ordered=\"false\" / \"False\" / \"0\" were all
treated as ordered=True. Only omitting the parameter entirely produced unordered
matching.

Switch to get_bool_test_case_param, which already exists in validations/utils.py and
does str.lower() == \"true\", so \"false\"/\"False\"/\"0\" correctly yield False.

Added regression fixtures and test cases in both the SQLite (database) and
DataFrame (datalake) test paths:
- test_case_column_to_match_set_ordered_false: all 11 table columns in shuffled
  order + ordered=\"false\" — expects Success (unordered match) rather than the
  False-before-fix ordered comparison that would produce Failure.

Fixes open-metadata#35616

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@zerafachris
zerafachris requested a review from a team as a code owner October 16, 2026 15:24
@github-actions

Copy link
Copy Markdown
Contributor

❌ PR checklist incomplete

This PR cannot be merged until the following are addressed on its linked issue:

The fields live on the linked issue in the Shipping project (open the issue → right sidebar → Projects). After you set them, re-run this check (or push a commit) — issue/project changes do not re-trigger it automatically.

Maintainers can bypass this check by adding the skip-pr-checks label.

@github-actions

Copy link
Copy Markdown
Contributor

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

@harshsoni2024 harshsoni2024 added Ingestion safe to test Add this label to run secure Github workflows on PRs labels Oct 26, 2026
@github-actions github-actions Bot removed the safe to test Add this label to run secure Github workflows on PRs label Oct 26, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

@harshsoni2024 harshsoni2024 added the safe to test Add this label to run secure Github workflows on PRs label Oct 26, 2026
…se fixture

The SQLAlchemy tableColumnToMatchSet validator reports ORM column keys,
so the users table's "first name" column comes back as "first_name"
and the shared fixture never matched. Give the database test its own
fixture; the datalake test keeps the original one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@zerafachris

Copy link
Copy Markdown
Contributor Author

Pushed fed33d8 to fix the test_case_column_to_match_set_ordered_false failure in the Python unit tests (3.10/3.11/3.12).

The SQLAlchemy tableColumnToMatchSet validator reads the ORM column keys (inspect(table).c.keys()), so on the sqlite users table the first name column comes back as first_name. The datalake validator reads the DataFrame column, which is first name. The fixture both tests shared listed first name, so the database variant could never match. The CI run on the original commit was cancelled, so this only surfaced after the merge from main.

The database test now uses its own fixture, test_case_column_to_match_set_ordered_false_sqa, which lists first_name. The datalake test is unchanged. Locally, test_validations_databases.py and test_validations_datalake.py pass (135 passed). With ordered read the old way, get_test_case_param_value(..., bool), both ordered_false cases fail. ruff 0.15.12 is clean.

(Prepared with AI assistance (Claude Code, Anthropic), reviewed for correctness before submission.)

@github-actions github-actions Bot removed the safe to test Add this label to run secure Github workflows on PRs label Oct 28, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

@harshsoni2024 harshsoni2024 added the safe to test Add this label to run secure Github workflows on PRs label Oct 29, 2026
@github-actions github-actions Bot removed the safe to test Add this label to run secure Github workflows on PRs label Oct 29, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

@gitar-bot

gitar-bot Bot commented Oct 29, 2026

Copy link
Copy Markdown
Code Review ✅ Approved

🟡 Medium risk · Correcting unordered column checks and normalizing patch operations affect validation results and entity updates, which could change detected quality failures or persisted metadata.

Switches the ordered parameter in tableColumnToMatchSet from bool casting to get_bool_test_case_param, so "false" and "False" correctly yield unordered matching. Also normalizes JSON patch operations in build_patch before array fields are appended. No issues found.

Verification

Rules No rules evaluated

Functional validation Not enabled

Set up

Auto-approval Not enabled

Set up

For more verification details, comment gitar display:verbose.

Options

Display: compact → Showing compact verification results.

Comment with these commands to change the behavior for this request:

Compact
gitar display:verbose         

Was this helpful? React with 👍 / 👎 | Powered by Gitar — free for open source

@harshsoni2024 harshsoni2024 added the safe to test Add this label to run secure Github workflows on PRs label Oct 29, 2026
@harshsoni2024
harshsoni2024 added this pull request to the merge queue Oct 29, 2026
@sonarqubecloud

Copy link
Copy Markdown

Merged via the queue into open-metadata:main with commit fd04bd8 Oct 29, 2026
88 of 100 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Ingestion safe to test Add this label to run secure Github workflows on PRs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

tableColumnToMatchSet: ordered: "false" is treated as True, so unordered matching only works when the parameter is left out

2 participants