Skip to content

branch-4.1: [fix](iceberg) Reject invalid name mapping instead of reading NULL - #68003

Draft
hubgeter wants to merge 1 commit into
apache:branch-4.1from
hubgeter:fix/iceberg-invalid-name-mapping-branch-4.1
Draft

hubgeter wants to merge 1 commit into
apache:branch-4.1from
hubgeter:fix/iceberg-invalid-name-mapping-branch-4.1

Conversation

@hubgeter

@hubgeter hubgeter commented Sep 15, 2026

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Problem Summary:
bp #68004

When the Iceberg table property schema.name-mapping.default is present but malformed, the FE caught the parse failure and rebuilt a "current schema" name mapping, which is authoritative. Data files that do not carry Iceberg field ids could then only be resolved by their current column names. After a column rename the old physical column is unmatched, V2 semantics materializes NULL for it, and the query succeeds with historical values lost. If the old name was later reused by another column, the fallback could even bind the wrong physical column.

Iceberg itself does not degrade like this: Spark's BaseReader parses schema.name-mapping.default while constructing the reader and fails the query for a malformed value (IllegalArgumentException / UncheckedIOException), regardless of whether the data files carry field ids.

This PR removes the current-schema fallback and reports the metadata fault instead. IcebergUtils.getNameMapping now throws a UserException that names the table, the property, the root cause and the remediation, so the failure is visible instead of silently returning NULL. Metadata (system) table scans are excluded, because they never resolve physical data columns by name.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hubgeter
hubgeter requested a review from yiguolei as a code owner September 15, 2026 07:00
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

…ding NULL (apache#68003)

### What problem does this PR solve?

Problem Summary:
A table property `schema.name-mapping.default` that is present but malformed was silently
degraded: the FE caught the parse failure and rebuilt a "current schema" mapping, which is
authoritative, so data files without Iceberg field ids could only be resolved by their
current column names. After a column rename the old physical column is then unmatched and
V2 semantics materializes NULL for it - the query succeeds but historical values are lost.
When a name was reused the same fallback could even bind the wrong physical column.

Iceberg readers do not behave that way: Spark's `BaseReader` parses
`schema.name-mapping.default` while constructing the reader and fails the query on a
malformed value (`IllegalArgumentException`/`UncheckedIOException`), regardless of whether
the data files carry field ids.

This PR removes the current-schema fallback and reports the metadata fault instead:
`IcebergUtils.getNameMapping` now throws a `UserException` naming the table, the property,
the root cause and the remediation. Metadata (system) table scans are excluded because they
never resolve physical data columns by name.

### Behavior changes

Before this change:

- A malformed `schema.name-mapping.default` was rewritten into current-schema aliases, so
  ID-less files were read by current column name (renamed columns returned NULL).
- No user visible error was raised; only a FE warning was logged.

After this change:

- A malformed `schema.name-mapping.default` fails the query with an explicit metadata error.
- Queries that only read Iceberg metadata tables (for example `tbl$snapshots`) keep working.

### Check List (For Author)

- Test
    - [x] Unit Test
- Behavior changed:
    - [x] Yes.
- Does this need documentation?
    - [x] No.
@hubgeter
hubgeter force-pushed the fix/iceberg-invalid-name-mapping-branch-4.1 branch from 5db8aa0 to 8dabec7 Compare September 15, 2026 07:00
@hubgeter

Copy link
Copy Markdown
Contributor Author

run buildall

@hubgeter
hubgeter marked this pull request as draft September 15, 2026 10:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants