[VL] Remove redundant string trimming before Velox casts - #12965
Open
rui-mo wants to merge 2 commits into
Open
Conversation
|
Run Gluten Clickhouse CI on x86 |
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
The build helper script now defaults to applying a specific upstream Velox PR patch, which can unintentionally affect reproducibility and CI stability.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
This PR removes Gluten-side string trimming logic that was previously injected before CAST-from-varchar when running on the Velox backend, relying instead on Velox’s Spark-compatible cast trimming behavior.
Changes:
- Removes the Velox-specific CAST trim-node injection path (and its plumbing) from Gluten’s cast conversion/execution APIs.
- Deletes the associated Velox config (
spark.gluten.velox.castFromVarcharAddTrimNode) and removes it from documentation. - Updates Spark 3.5 / 4.0 / 4.1 Gluten UTs to validate whitespace-tolerant casts without enabling the removed config.
File summaries
| File | Description |
|---|---|
| gluten-ut/spark41/src/test/scala/org/apache/spark/sql/GlutenDataFrameSuite.scala | Drops use of the removed Velox trim config while keeping cast-with-whitespace assertions. |
| gluten-ut/spark40/src/test/scala/org/apache/spark/sql/GlutenDataFrameSuite.scala | Same test update for Spark 4.0 profile. |
| gluten-ut/spark35/src/test/scala/org/apache/spark/sql/GlutenDataFrameSuite.scala | Same test update for Spark 3.5 profile. |
| gluten-substrait/src/main/scala/org/apache/gluten/expression/ExpressionConverter.scala | Stops generating a backend-modified Cast for trimming; uses the original Cast child directly. |
| gluten-substrait/src/main/scala/org/apache/gluten/backendsapi/SparkPlanExecApi.scala | Removes the now-unused genCastWithNewChild hook from the backend API. |
| ep/build-velox/src/get-velox.sh | Sets a non-empty default UPSTREAM_VELOX_PR_ID, which forces patch application in builds. |
| docs/velox-configuration.md | Removes documentation for the deleted Velox trimming config. |
| backends-velox/src/main/scala/org/apache/gluten/config/VeloxConfig.scala | Removes the config accessor and registry entry for CAST-from-varchar trim-node injection. |
| backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxSparkPlanExecApi.scala | Deletes Velox’s override that injected StringTrim nodes ahead of casts. |
Review details
- Files reviewed: 9/9 changed files
- Comments generated: 1
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes are proposed in this pull request?
This patch removes the extra string trimming in Gluten before delegating casts to Velox.
Velox already applies Spark-compatible string cast trimming in its Spark cast hooks.
The corresponding Velox behavior is tracked with SPARK-59182, which discusses whether Spark should align the behavior of the two string cast trimming paths.
How was this patch tested?
Unit tests