Skip to content

[python] Align native scan planning with Rust - #9825

Open
JingsongLi wants to merge 3 commits into
apache:masterfrom
JingsongLi:codex/align-python-native-planning
Open

JingsongLi wants to merge 3 commits into
apache:masterfrom
JingsongLi:codex/align-python-native-planning

Conversation

@JingsongLi

@JingsongLi JingsongLi commented Sep 14, 2026

Copy link
Copy Markdown
Contributor

Purpose

Use the planning APIs from apache/paimon-rust#831 (merged) to support native incremental and distributed scans in PyPaimon. Timestamp windows combine APPEND deltas into one plan so primary-key versions across commits are merged together; the ending snapshot supplies metadata and deletion vectors even when it contributes no APPEND files.

  • Add native append slices/shards, primary-key bucket shards, and Data Evolution row-position selection. Assign DE positions before explicit/global-index range pruning, and apply limits after distribution.
  • Preserve snapshot IDs for empty plans and explain output, validate branch resolution, and fall back to Python when the installed binding cannot support the scan.
  • Honor index-file-in-data-file-dir for deletion-vector writes, reads, and abort cleanup. Preserve explicit paths and legacy Python index locations, handle partition-directory differences, and retain multiple DV index files in one bucket.
  • Preserve signed-zero ordering when grouping overlapping composite key ranges.

The native planner remains optional and disabled by default. The README documents supported scans, binding requirements, and fallback behavior.

Tests

Passed locally with Python 3.11 and a newly built Rust wheel whose core and Python-binding sources match the merged Rust main:

  • Planner, adapter compatibility, incremental, distribution, key-bound, and deletion-vector path regressions: 161 passed.
  • Reader, parallel reader, and Data Evolution regressions: 174 passed in each of the Python and native planning modes; 9 Lance cases deselected in each mode. The native runs exercised 147 native plans, including ORC reads.
  • Flake8 and Python 3.6 grammar checks on all 24 changed Python files; git diff --check.

The full Python dependency/version matrix and Java integration suite were not run locally.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant