HIVE-30064: Report accurate affected-row counts for copy-on-write UPDATE and MERGE - #6794
Open
ryukobayashi wants to merge 1 commit into
Open
ryukobayashi wants to merge 1 commit into
ryukobayashi wants to merge 1 commit into
Conversation
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



What changes were proposed in this pull request?
This patch adds matched-row tracking for copy-on-write
UPDATEandMERGEoperations.The CoW rewriters append a trailing
cow_update_matchedboolean marker to their generated projections:truefor rows produced by the matchedUPDATEbranchfalsefor unchanged survivor rowsfalseforMERGEINSERTandDELETEbranchesFileSinkOperatoruses this marker to count only rows that should contribute to the matched-row count while still writing all rows normally.The marker-based logic is applied to:
CopyOnWriteUpdateRewriterCopyOnWriteMergeRewriterFileSinkOperatorRegression coverage was added through Iceberg CoW tests and a helper for retrieving
numModifiedRowsfrom Hive operation status.Why are the changes needed?
Copy-on-write operations rewrite complete data files. Therefore, the rows sent to the
FileSinkOperatorinclude both modified rows and unchanged rows copied from the rewritten files.For
MERGE, a single output also combines multiple logical branches, including matched updates, not-matched inserts, deletes, and unchanged survivor rows.Previously,
FileSinkOperatorcounted all output rows, so Hive could report an incorrectnumModifiedRowsvalue. The reported count could include unchanged rows or rows from otherMERGEbranches.The marker makes the logical operation explicit and avoids relying on the physical Tez plan shape or operator layout to determine the count.
Does this PR introduce any user-facing change?
Yes.
The reported affected-row count for copy-on-write
UPDATEandMERGEoperations is corrected to reflect rows selected by the matchedUPDATEoperation instead of all rows rewritten by the CoW plan.The underlying table data and write behavior are unchanged.
For example, a CoW
MERGEthat updates two rows, inserts two rows, and deletes one row reports the matched update count as2.How was this patch tested?
Added Iceberg CoW regression tests for:
UPDATEwith two matching rowsMERGEwith two matched updates, two inserts, and one deleteThe tests retrieve
numModifiedRowsthrough Hive operation status and verify the expected count.