Skip to content

HIVE-30064: Report accurate affected-row counts for copy-on-write UPDATE and MERGE - #6794

Open
ryukobayashi wants to merge 1 commit into
apache:masterfrom
ryukobayashi:HIVE-30064
Open

ryukobayashi wants to merge 1 commit into
apache:masterfrom
ryukobayashi:HIVE-30064

Conversation

@ryukobayashi

@ryukobayashi ryukobayashi commented Sep 17, 2026

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

This patch adds matched-row tracking for copy-on-write UPDATE and MERGE operations.

The CoW rewriters append a trailing cow_update_matched boolean marker to their generated projections:

  • true for rows produced by the matched UPDATE branch
  • false for unchanged survivor rows
  • false for MERGE INSERT and DELETE branches

FileSinkOperator uses this marker to count only rows that should contribute to the matched-row count while still writing all rows normally.

The marker-based logic is applied to:

  • CopyOnWriteUpdateRewriter
  • CopyOnWriteMergeRewriter
  • FileSinkOperator

Regression coverage was added through Iceberg CoW tests and a helper for retrieving numModifiedRows from Hive operation status.

Why are the changes needed?

Copy-on-write operations rewrite complete data files. Therefore, the rows sent to the FileSinkOperator include both modified rows and unchanged rows copied from the rewritten files.

For MERGE, a single output also combines multiple logical branches, including matched updates, not-matched inserts, deletes, and unchanged survivor rows.

Previously, FileSinkOperator counted all output rows, so Hive could report an incorrect numModifiedRows value. The reported count could include unchanged rows or rows from other MERGE branches.

The marker makes the logical operation explicit and avoids relying on the physical Tez plan shape or operator layout to determine the count.

Does this PR introduce any user-facing change?

Yes.

The reported affected-row count for copy-on-write UPDATE and MERGE operations is corrected to reflect rows selected by the matched UPDATE operation instead of all rows rewritten by the CoW plan.

The underlying table data and write behavior are unchanged.

For example, a CoW MERGE that updates two rows, inserts two rows, and deletes one row reports the matched update count as 2.

How was this patch tested?

Added Iceberg CoW regression tests for:

  • Copy-on-write UPDATE with two matching rows
  • Copy-on-write MERGE with two matched updates, two inserts, and one delete

The tests retrieve numModifiedRows through Hive operation status and verify the expected count.

@sonarqubecloud

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants