Skip to content

[python] Cover native dynamic PK events and compacted first-row scans - #9854

Merged
JingsongLi merged 1 commit into
apache:masterfrom
JingsongLi:codex/native-bucket-first-row-coverage
Sep 16, 2026
Merged

JingsongLi merged 1 commit into
apache:masterfrom
JingsongLi:codex/native-bucket-first-row-coverage

Conversation

@JingsongLi

Copy link
Copy Markdown
Contributor

Purpose

Strengthen native-plan coverage for dynamic primary-key and first-row tables after #9846.

  • Check the actual cross-partition migration events in incremental reads, including old/new partition predicates. Java writes a DELETE in the old partition with the incoming non-partition values; comparing snapshot rows alone did not catch Rust's UPDATE_BEFORE emission.
  • Exercise non-empty first-row reads from compacted runs at different levels with overlapping key ranges and disjoint keys. Check fixed/dynamic bucket modes, skipped pending L0 rows, value filters and historical snapshots. The native path must execute without silently falling back to Python.
  • Update the remaining first-row L0 fallback comment: Rust now supports ordinary first-row overlap merging, but PK-clustering files sorted by non-key columns still need separate read-path coverage.

Depends on apache/paimon-rust#845. Keep this PR in draft until that change reaches Rust main: the native CI job intentionally continues to build Rust main, and the new migration assertion detects its current row-kind bug.

Tests

Built the Python binding from paimon-rust commit c48317e6 and ran all seven native-plan test modules, split_serializer_test.py and global_index_test.py: 196 passed, 276 native plans exercised.

Flake8 passed for all changed Python files. The migration regression fails against the previous Rust main, as expected.

@JingsongLi JingsongLi closed this Sep 15, 2026
@JingsongLi JingsongLi reopened this Sep 15, 2026
@JingsongLi
JingsongLi marked this pull request as ready for review September 15, 2026 14:40

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 6f0dc0c4019686527e2eae7dfe532b24e10aa76a against bbae07d37545cf2cf65b397007fa68c387bb521f, using Java as the semantic baseline. No blocking findings; approving.

  • Confirmed that the only production-file change is a comment: table_scan.py has an identical AST to the base revision. This PR does not change runtime behavior or public APIs.
  • The incremental migration expectations match Java DeleteExistingProcessor: emit DELETE in the old partition using the incoming non-partition values. The new assertions check the physical row kind as well as old/new partition predicates, rather than relying on batch results that can hide an incorrect retraction kind.
  • Checked the synthetic compacted first-row fixtures against Java CoreOptions.batchScanSkipLevel0 and MergeTreeSplitGenerator: overlapping key ranges with disjoint actual keys at L1/L2, fixed/dynamic buckets, skipped pending L0 records, value predicates, and historical snapshots. These test the scan/read paths, not the compactor itself. The new native cases explicitly fail on Python-planner fallback.
  • Confirmed that the Rust #845 dependency is merged. Rebuilt the Python binding from current Rust main 66f655b30a3861a31b5a0abafd25a8633574049c for validation.

Local validation on Python 3.10:

  • Changed test modules: 12 passed.
  • All seven native-plan modules plus split-serializer and global-index tests: 196 passed, 276 native plans exercised.
  • Flake8 with the repository's dev/cfg.ini: passed for all three changed files.
  • Negative control with the pre-#845 Rust binding: the new cross-partition assertion fails specifically because (1, 'a', 'moved', 1) is emitted instead of (1, 'a', 'moved', 3). This confirms that the regression test detects the original bug.

GitHub CI is not fully green at review time: several Python jobs, including Rust Plan, are cancelled, the aggregate CI result is failed, and another test-selection check is queued. The local results above do not claim full CI-matrix coverage.

@JingsongLi
JingsongLi merged commit 2bbdb66 into apache:master Sep 16, 2026
20 of 28 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants