fix: align nested piecewise merge join sorting and comparison

[mikamikasuki]

Which issue does this PR close?

• Closes PiecewiseMergeJoin returns wrong results for List keys with NULL elements and fails on Struct keys #25957.

Rationale for this change

Piecewise merge joins can return incorrect results for nested range keys containing inner NULLs, and fail when sorting Struct or List-of-List keys. Sorting, extreme-key selection, and comparison must use the same ordering.

What changes are included in this PR?

• Sort </<= keys descending with NULLs last, reversing both value and inner-NULL ordering to agree with SQL comparisons.
• Bound classic and left existence scans to non-null keys, preserving trailing NULL rows in outer, anti, and mark results.
• Select nested min/max keys with Arrow's comparator.
• Sort streamed keys with lexsort_to_indices, supporting Struct and List-of-List keys.
• Execute nested range joins with PiecewiseMergeJoinExec.

What is the testing strategy for this PR?

SQL logic tests compare complete result multisets against NestedLoopJoin for List, Struct, and List-of-List keys, all four range operators, classic joins, semi/anti joins, and an EXISTS mark expression. The matrix covers batch sizes 1, 2, and 8192 and partition counts 1 and 4. Plan assertions check that Struct and List-of-List classic joins use PWMJ.

Validation passed: PWMJ SQL logic tests and fuzz tests, the extended workspace suite (including all 527 SQL test files), formatting, Clippy with all targets/features, and the repository lint checks.

Are there any user-facing changes?

Nested range joins produce correct results and support Struct and List-of-List keys with piecewise merge joins enabled. No public API changes.

[jayzhan211]

Thanks @mikamikasuki

[codecov-commenter]

Codecov Report

❌ Patch coverage is 78.57143% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.66%. Comparing base (c3ef346) to head (b897fd9).
⚠️ Report is 11 commits behind head on main.

Files with missing lines	Patch %	Lines
datafusion/core/src/physical_planner.rs	78.57%	1 Missing and 2 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #26043      +/-   ##
==========================================
- Coverage   82.66%   82.66%   -0.01%     
==========================================
  Files        1147     1147              
  Lines      446357   446395      +38     
  Branches   446357   446395      +38     
==========================================
+ Hits       368971   369002      +31     
- Misses      54997    55001       +4     
- Partials    22389    22392       +3     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[2010YOUY01]

Thanks for the fix.

The root cause seem to be sorting and comparison don't agree in PWMJ, is it possible to fix that directly?

If that's tricky, this PR's approach totally makes sense, but we would keep the issue open, and add a note for the remaining fix to the root cause.

[jayzhan211]

@mikamikasuki Would you like to rewrite this PR for the fix? I guess the gaps to the direct fix to the root cause are:

• Sort </<= keys DESC NULLS LAST, and make the classic and left existence streams skip NULL keys at the end instead of assuming they sort first
• Pick the min/max of nested keys with arrow's comparator instead of ScalarValue ordering
• Sort the streamed side with lexsort_to_indices, so Struct and List-of-List keys work

They could be the follow-up if you want, change Close to Part of in the description

[mikamikasuki]

Thanks for pointing out these gaps. I’ve updated this PR to fix the ordering/comparison mismatch directly:

• < and <= keys sort DESC NULLS LAST, and the classic and left existence joins skip trailing NULL keys.
• Nested min/max keys use Arrow’s comparator.
• The streamed side uses lexsort_to_indices, covering Struct and List-of-List keys.
  I’ve also removed the nested-key fallback and added regression coverage against NestedLoopJoin. The local extended workspace tests and lint checks pass.