[GLUTEN-12921][VL] Support full outer BroadcastNestedLoopJoin with guarded rewrite by WangGuangxin · Pull Request #12922 · apache/gluten

WangGuangxin · 2026-08-28T11:53:21Z

What changes are proposed in this pull request?

This PR adds Velox support for BroadcastNestedLoopJoinExec in FULL OUTER JOIN cases by rewriting the Spark physical plan into a composition that Velox already supports, instead of falling back the original
full outer BNLJ.

The rewrite transforms:

BroadcastNestedLoopJoinExec(..., FullOuter, ...)

into:

one outer BNLJ branch that produces matched rows plus unmatched streamed-side rows
one outer BNLJ branch with a synthetic non-null marker column to identify unmatched broadcast-side rows
a Union to combine both branches

This avoids introducing native Velox full outer nested-loop join support while preserving full outer join semantics, including null-sensitive cases.

In addition, this PR adds a conservative size-based guard for the rewrite:

the rewrite is applied only when both logical join sides have known statistics
both sides must be at or below spark.gluten.sql.columnar.broadcastNestedLoopJoin.fullOuterRewriteThreshold

Otherwise, Gluten keeps the original behavior and lets the plan fall back instead of forcing the rewrite on larger joins.

Why are the changes needed?

Velox does not support Spark's FULL OUTER broadcast nested loop join end-to-end today, so Gluten falls back for these queries.

A direct native implementation is not a good short-term option because:

the existing Velox/Gluten BNLJ pipeline is built around a broadcast build side
full outer semantics require emitting unmatched rows from both sides
handling unmatched broadcast-side rows natively would need extra protocol/state beyond the current execution model

A rewrite-based solution is lower risk because it reuses operators already supported by Velox.

However, rewriting every full outer BNLJ into two joins can be more expensive than a single Spark row-based fallback join. To keep the feature safe, this PR only enables the rewrite for small, stats-known joins
and preserves fallback for larger or unknown-size cases.

How was this patch tested?

More UT

Config

A new config is introduced:

spark.gluten.sql.columnar.backend.velox.broadcastNLJ.fullOuterRewriteThreshold

It controls the maximum per-side logical plan size for enabling the full outer BNLJ rewrite.