build: make the datafusion parquet and sql features optional - #588
Merged
gabotechs merged 2 commits intoAug 5, 2026
Merged
Conversation
`datafusion-distributed` turned on datafusion's `parquet` and `sql` features
unconditionally, so every consumer paid for the parquet and sqlparser stacks
whether or not it used them. An embedder whose plans arrive through the codec
rather than `SessionContext::sql`, and which never reads a parquet file, has no
use for either.
Both are now features, and both are in the default set, so nothing changes for
anyone who does not opt out:
parquet = ["datafusion/parquet", "datafusion-proto/parquet"]
sql = ["datafusion/sql"]
`datafusion-proto` moves to `default-features = false` in the workspace
dependency table so that its own default `parquet` feature does not pull the
stack back in; the `parquet` feature above re-enables it.
The only non-test use of either was the gRPC error codec. Upstream already gates
`DataFusionError::ParquetError` and `DataFusionError::SQL` behind the same two
feature names, so the gating here mirrors it.
The proto types stay compiled unconditionally and keep their prost tags; only
the conversions that mention `ParquetError`/`ParserError` are gated. A peer
built with the features can therefore still talk to one built without: the
receiver decodes the message and, when it cannot construct the typed variant,
degrades to `DataFusionError::Internal` rather than failing to decode.
`integration` gains both features, and now names the optional `parquet`
dependency as `dep:parquet` because `parquet` is a real feature from here on.
The crate's own tests drive `SessionContext::sql` and register parquet tables
regardless of the selected features, so dev-dependencies request both; that
keeps `cargo test --no-default-features --lib` compiling without forcing either
on library consumers.
Verified: `cargo clippy --all-targets --features integration -- -D warnings`
clean, `cargo test --no-default-features --lib` passes 278/0, and
`cargo test --features integration` passes. Holding `grpc` constant, building
with `default-features = false` drops `parquet`, `datafusion-datasource-parquet`,
`sqlparser` and `datafusion-sql` from the graph, 20 crates in total.
gabotechs
approved these changes
Aug 4, 2026
The proto types are compiled unconditionally, so the message was already on the wire; the fallback arms just dropped it with `let _ = err`. Give each proto a `Display` that mirrors the upstream one and use it in those arms, so a peer's error reads `ParquetError from peer: Parquet error: <msg>` instead of a bare `ParquetError from peer`. The SQL arm carries the backtrace too. `Display` rather than an inherent method: the modules are not publicly reachable, so a plain `to_message()` tripped `dead_code` in the lib-only build with the features on. Since these mirror an upstream `Display` by hand, they can drift on a dependency bump. The existing roundtrip tests now assert the proto's rendering equals the real error's across every variant, pinning them together in the default-features CI run. Claude-Session: https://claude.ai/code/session_019j6qzKy2rrLCuSgmMB1NmV
Collaborator
|
Thanks @philippemnoel! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
This PR makes
parquetandsqlparseroptional, default-on features, so we can turn them off in @paradedb. Our build times are growing quite unwieldy, and we don't make use of these features in our embedded use case, so we'd like to turn them off. Hopefully this benefits others and doesn't get in the way for Datadog.All tests were run and pass, and we merged this in our fork: paradedb#59.