Skip to content

Run SQL benchmarks with plan v2 (full stack vs develop) #9342

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
joseph-isaacs wants to merge 9 commits into
base: develop
Choose a base branch
from
Draft

Run SQL benchmarks with plan v2 (full stack vs develop) #9342

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
e21444d
Add plan optimizer rules and push expressions
joseph-isaacs Aug 12, 2026
dd45b3e
Add zoned layout scan plan
joseph-isaacs Aug 10, 2026
811c5fa
Add plan-native scan execution
joseph-isaacs Aug 10, 2026
1ded40b
Prune plan scans with zoned statistics
joseph-isaacs Aug 10, 2026
bfc1bf0
Avoid expanding unreferenced struct plan fields
joseph-isaacs Aug 10, 2026
cde2e4f
Run SQL benchmarks with plan v2
joseph-isaacs Aug 11, 2026
375dc4e
Add adaptive plan filter execution
joseph-isaacs Aug 11, 2026
5bb4bd9
Support DataFusion file-scan repartitioning in plan-v2 scans
claude Aug 11, 2026
b472e97
Trace plan-v2 partial file ranges
claude Aug 11, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/workflows/sql-bench-matrix.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -103,6 +103,7 @@ jobs:
env:
VORTEX_EXPERIMENTAL_PATCHED_ARRAY: "1"
FLAT_LAYOUT_INLINE_ARRAY_NODE: "1"
VORTEX_USE_PLAN_V2: "1"
# Makes python output nicer
COLUMNS: 120
strategy:
Expand Down
24 changes: 24 additions & 0 deletions Cargo.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 2 additions & 0 deletions Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ members = [
"vortex-btrblocks",
"vortex-layout",
"vortex-scan",
"vortex-scan-v2",
"vortex-file",
"vortex-ipc",
"vortex",
Expand Down Expand Up @@ -323,6 +324,7 @@ vortex-proto = { version = "0.1.0", path = "./vortex-proto", default-features =
vortex-row = { version = "0.1.0", path = "./vortex-row", default-features = false }
vortex-runend = { version = "0.1.0", path = "./encodings/runend", default-features = false }
vortex-scan = { version = "0.1.0", path = "./vortex-scan", default-features = false }
vortex-scan-v2 = { version = "0.1.0", path = "./vortex-scan-v2", default-features = false }
vortex-sequence = { version = "0.1.0", path = "encodings/sequence", default-features = false }
vortex-session = { version = "0.1.0", path = "./vortex-session", default-features = false }
vortex-sparse = { version = "0.1.0", path = "./encodings/sparse", default-features = false }
Expand Down
13 changes: 9 additions & 4 deletions docs/developer-guide/internals/scan-planning.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,9 +55,14 @@ ID, dtype, row count, and lazy children. Only the unsized tail containing the vt
already serialize their metadata; the ones holding a read context or a bound expression return
`None` until those codecs exist.

## Execution

Each operator executes over a row range and selection mask. `SegmentScan` reads its segment,
structural operators combine their children, and `Eval` applies the remaining derived work.
`vortex-scan-v2` copies the existing scan orchestration around this API, so the original
`LayoutReader` scanner is untouched while the plan-native path is developed.

## Future work

Plans currently stop at construction and optimization. Still to come: a plan registry and foreign
operator placeholder so third-party operators survive a round trip, a serialization envelope, and
an execution stage that walks an optimized plan, reads the referenced segments, and returns the
query result.
Still to come: a plan registry and foreign operator placeholder so third-party operators survive
a round trip, and a serialization envelope.
1 change: 1 addition & 0 deletions vortex-datafusion/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@ tokio-stream = { workspace = true }
tracing = { workspace = true, features = ["std", "attributes"] }
vortex = { workspace = true, features = ["object_store", "tokio", "files"] }
vortex-arrow = { workspace = true }
vortex-scan-v2 = { workspace = true }
vortex-utils = { workspace = true, features = ["dashmap"] }

[dev-dependencies]
Expand Down
Loading
Loading