Skip to content

[SPARK-38743][SQL] Test the error class: MISSING_STATIC_PARTITION_COLUMN - #54692

Open
deepujain wants to merge 1 commit into
apache:masterfrom
deepujain:SPARK-38743-missing-static-partition-column-test
Open

[SPARK-38743][SQL] Test the error class: MISSING_STATIC_PARTITION_COLUMN#54692
deepujain wants to merge 1 commit into
apache:masterfrom
deepujain:SPARK-38743-missing-static-partition-column-test

Conversation

@deepujain

@deepujain deepujain commented Mar 9, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds test coverage and standard error class for the MISSING_STATIC_PARTITION_COLUMN case in QueryCompilationErrors. When an INSERT or DELETE specifies a static partition column name that is not found in the target table's output, Spark now throws an AnalysisException with error class MISSING_STATIC_PARTITION_COLUMN and sqlState 42703, instead of a generic internal error.

Change

  • error-conditions.json: Added MISSING_STATIC_PARTITION_COLUMN with message "Static partition column is not found in the table." and sqlState 42703.
  • QueryCompilationErrors.scala: Updated missingStaticPartitionColumn(staticName) to throw AnalysisException with error class MISSING_STATIC_PARTITION_COLUMN and messageParameters = Map("staticName" -> toSQLId(staticName)) instead of SparkException.internalError.
  • QueryCompilationErrorsSuite.scala: Added test SPARK-38743: MISSING_STATIC_PARTITION_COLUMN that asserts error class, sqlState, and message parameters for the exception from QueryCompilationErrors.missingStaticPartitionColumn("p").

Tests

New test in QueryCompilationErrorsSuite covers the error class, sqlState, and message parameters as required by the JIRA.

Fixes SPARK-38743

JIRA assignee for credit: deepujain

@deepujain
deepujain force-pushed the SPARK-38743-missing-static-partition-column-test branch from ecbe82e to 30e8a2a Compare March 12, 2026 04:58
@deepujain

deepujain commented Mar 12, 2026

Copy link
Copy Markdown
Contributor Author

Rebased on master. Ran QueryCompilationErrorsSuite (SPARK-38743) locally passed. Ready for CI.

@deepujain
deepujain force-pushed the SPARK-38743-missing-static-partition-column-test branch from 30e8a2a to 576a4cf Compare May 27, 2026 20:02
@deepujain

deepujain commented May 27, 2026

Copy link
Copy Markdown
Contributor Author

Rebased on current master. Ran ./build/sbt "sql/testOnly org.apache.spark.sql.errors.QueryCompilationErrorsSuite -- -z SPARK-38743" - passed. Ready for CI.

@deepujain
deepujain force-pushed the SPARK-38743-missing-static-partition-column-test branch from 576a4cf to 8f21bab Compare June 26, 2026 15:53
@deepujain

deepujain commented Jun 26, 2026

Copy link
Copy Markdown
Contributor Author

Rebased this on current master. I reran ./build/sbt "sql/testOnly org.apache.spark.sql.errors.QueryCompilationErrorsSuite -- -z SPARK-38743" locally; it passed. Ready for CI.

@deepujain
deepujain force-pushed the SPARK-38743-missing-static-partition-column-test branch from 8f21bab to f9ae9af Compare July 2, 2026 09:20
@deepujain

deepujain commented Jul 2, 2026

Copy link
Copy Markdown
Contributor Author

Rebased this on current master. I reran ./build/sbt "sql/testOnly org.apache.spark.sql.errors.QueryCompilationErrorsSuite -- -z SPARK-38743" locally; it passed. Ready for CI.

@deepujain
deepujain force-pushed the SPARK-38743-missing-static-partition-column-test branch from f9ae9af to 548c154 Compare July 3, 2026 01:41
@deepujain

deepujain commented Jul 3, 2026

Copy link
Copy Markdown
Contributor Author

Rebased this on current master. I reran ./build/sbt "sql/testOnly org.apache.spark.sql.errors.QueryCompilationErrorsSuite -- -z SPARK-38743" locally; it passed. Ready for CI.

@deepujain
deepujain force-pushed the SPARK-38743-missing-static-partition-column-test branch from 548c154 to 49981e5 Compare July 6, 2026 10:53
@deepujain

deepujain commented Jul 6, 2026

Copy link
Copy Markdown
Contributor Author

Rebased again on current master to refresh the red fork Build check. I reran ./build/sbt "sql/testOnly org.apache.spark.sql.errors.QueryCompilationErrorsSuite -- -z SPARK-38743" locally; it passed. CI has restarted on the new head.

@deepujain
deepujain force-pushed the SPARK-38743-missing-static-partition-column-test branch from 49981e5 to 8177a06 Compare July 7, 2026 06:44
@deepujain

deepujain commented Jul 7, 2026

Copy link
Copy Markdown
Contributor Author

Rebased again on current master to pick up the upstream fix for the ./dev/lint-scala line-length failure. I reran ./build/sbt "sql/testOnly org.apache.spark.sql.errors.QueryCompilationErrorsSuite -- -z SPARK-38743" and ./dev/lint-scala locally; both passed. CI has restarted on the new head.

@deepujain
deepujain force-pushed the SPARK-38743-missing-static-partition-column-test branch from 8177a06 to 0e87656 Compare July 9, 2026 07:16
@deepujain

deepujain commented Jul 9, 2026

Copy link
Copy Markdown
Contributor Author

Rebased on current master again. The previous red Build was in a broad connect/streaming job: python foreachBatch process: process terminates after query is stopped timed out, which is outside this SQL error-class test change. I reran ./build/sbt "sql/testOnly org.apache.spark.sql.errors.QueryCompilationErrorsSuite -- -z SPARK-38743" locally; it passed. CI has restarted on the new head.

@deepujain
deepujain force-pushed the SPARK-38743-missing-static-partition-column-test branch from 0e87656 to d028221 Compare July 9, 2026 09:22
@deepujain

deepujain commented Jul 9, 2026

Copy link
Copy Markdown
Contributor Author

Refreshed the branch after the GitHub runner shut down the sql - slow tests job. This was a no-source-change amend, so the Spark diff is unchanged; earlier today I ran ./build/sbt "sql/testOnly org.apache.spark.sql.errors.QueryCompilationErrorsSuite -- -z SPARK-38743" locally and it passed. CI is starting again on the new head.

@deepujain
deepujain force-pushed the SPARK-38743-missing-static-partition-column-test branch from d028221 to 956faa0 Compare July 9, 2026 23:51
@deepujain

deepujain commented Jul 9, 2026

Copy link
Copy Markdown
Contributor Author

Rebased on current master after the last Build failed in a broad PySpark job: pyspark.tests.test_import_spark saw numpy during import pyspark, which is outside this Scala SQL change. I reran ./build/sbt "sql/testOnly org.apache.spark.sql.errors.QueryCompilationErrorsSuite -- -z SPARK-38743" locally on the rebased head; it passed. CI has restarted on the new head.

@deepujain
deepujain force-pushed the SPARK-38743-missing-static-partition-column-test branch from 956faa0 to f4ba56b Compare July 10, 2026 03:07
@deepujain

deepujain commented Jul 10, 2026

Copy link
Copy Markdown
Contributor Author

Rebased on current master again after master moved to the parser-based SQL splitter change. I reran ./build/sbt "sql/testOnly org.apache.spark.sql.errors.QueryCompilationErrorsSuite -- -z SPARK-38743" locally on the rebased head; it passed. CI has restarted on the new head.

@deepujain
deepujain force-pushed the SPARK-38743-missing-static-partition-column-test branch from f4ba56b to 0b8d7e3 Compare July 11, 2026 03:48
@deepujain

deepujain commented Jul 11, 2026

Copy link
Copy Markdown
Contributor Author

Rebased on current master after the latest Python Arrow change landed. I reran ./build/sbt "sql/testOnly org.apache.spark.sql.errors.QueryCompilationErrorsSuite -- -z SPARK-38743" locally on the rebased head; it passed. Ready for CI.

@deepujain
deepujain force-pushed the SPARK-38743-missing-static-partition-column-test branch from 0b8d7e3 to e3e79e6 Compare July 12, 2026 16:44
@deepujain

deepujain commented Jul 12, 2026

Copy link
Copy Markdown
Contributor Author

Rebased on current master after SPARK-58064 landed. I reran ./build/sbt "sql/testOnly org.apache.spark.sql.errors.QueryCompilationErrorsSuite -- -z SPARK-38743" locally on the rebased head; it passed. Ready for CI.

@deepujain
deepujain force-pushed the SPARK-38743-missing-static-partition-column-test branch from e3e79e6 to fab3fc9 Compare July 13, 2026 00:34
@deepujain

deepujain commented Jul 13, 2026

Copy link
Copy Markdown
Contributor Author

Rebased on current master after SPARK-57884 landed. I reran ./build/sbt "sql/testOnly org.apache.spark.sql.errors.QueryCompilationErrorsSuite -- -z SPARK-38743" locally on the rebased head; it passed. Ready for CI.

@deepujain
deepujain force-pushed the SPARK-38743-missing-static-partition-column-test branch from fab3fc9 to d14b6ec Compare July 14, 2026 09:15
@deepujain

deepujain commented Jul 14, 2026

Copy link
Copy Markdown
Contributor Author

Rebased on current master after SPARK-58104 landed. I reran ./build/sbt "sql/testOnly org.apache.spark.sql.errors.QueryCompilationErrorsSuite -- -z SPARK-38743" locally on the rebased head; it passed. Ready for CI.

@deepujain

deepujain commented Jul 26, 2026

Copy link
Copy Markdown
Contributor Author

Rebased on current master after SPARK-58341 landed. I reran ./build/sbt "sql/testOnly org.apache.spark.sql.errors.QueryCompilationErrorsSuite -- -z SPARK-38743" locally on the rebased head; it passed. Ready for CI.

@deepujain
deepujain force-pushed the SPARK-38743-missing-static-partition-column-test branch from 6d99ad7 to af2d106 Compare July 26, 2026 12:45
@deepujain

deepujain commented Jul 26, 2026

Copy link
Copy Markdown
Contributor Author

Rebased on current master after SPARK-58192 landed. I reran ./build/sbt "sql/testOnly org.apache.spark.sql.errors.QueryCompilationErrorsSuite -- -z SPARK-38743" locally on the rebased head; it passed. Ready for CI.

@deepujain
deepujain force-pushed the SPARK-38743-missing-static-partition-column-test branch from af2d106 to 6499026 Compare July 29, 2026 06:18
@deepujain

deepujain commented Jul 29, 2026

Copy link
Copy Markdown
Contributor Author

Rebased on current master after SPARK-58321 landed. I reran ./build/sbt "sql/testOnly org.apache.spark.sql.errors.QueryCompilationErrorsSuite -- -z SPARK-38743" locally on the rebased head; it passed. Ready for CI.

@deepujain
deepujain force-pushed the SPARK-38743-missing-static-partition-column-test branch from 6499026 to b150de7 Compare August 4, 2026 07:52
@deepujain

deepujain commented Aug 4, 2026

Copy link
Copy Markdown
Contributor Author

Rebased on current master after SPARK-58535 landed. I reran ./build/sbt "sql/testOnly org.apache.spark.sql.errors.QueryCompilationErrorsSuite -- -z SPARK-38743" locally on the rebased head; it passed. Ready for CI.

@deepujain
deepujain force-pushed the SPARK-38743-missing-static-partition-column-test branch from b150de7 to ec9d4b1 Compare August 4, 2026 13:01
@deepujain

deepujain commented Aug 4, 2026

Copy link
Copy Markdown
Contributor Author

Rebased on current master after SPARK-58546 landed. I reran ./build/sbt "sql/testOnly org.apache.spark.sql.errors.QueryCompilationErrorsSuite -- -z SPARK-38743" locally on the rebased head; it passed. Ready for CI.

@deepujain
deepujain force-pushed the SPARK-38743-missing-static-partition-column-test branch from ec9d4b1 to 220f819 Compare August 4, 2026 18:31
@deepujain

deepujain commented Aug 4, 2026

Copy link
Copy Markdown
Contributor Author

Rebased on current master after SPARK-58435 landed. I reran ./build/sbt "sql/testOnly org.apache.spark.sql.errors.QueryCompilationErrorsSuite -- -z SPARK-38743" locally on the rebased head; it passed. Ready for CI.

@deepujain
deepujain force-pushed the SPARK-38743-missing-static-partition-column-test branch from 220f819 to 4287100 Compare August 5, 2026 05:28
@deepujain

deepujain commented Aug 5, 2026

Copy link
Copy Markdown
Contributor Author

Rebased on current master after SPARK-58548 landed. I reran ./build/sbt "sql/testOnly org.apache.spark.sql.errors.QueryCompilationErrorsSuite -- -z SPARK-38743" locally on the rebased head; it passed. Ready for CI.

@deepujain
deepujain force-pushed the SPARK-38743-missing-static-partition-column-test branch from 4287100 to dfd1a9a Compare August 6, 2026 09:02
@deepujain

deepujain commented Aug 6, 2026

Copy link
Copy Markdown
Contributor Author

Rebased on current master after SPARK-58501 landed. I reran ./build/sbt "sql/testOnly org.apache.spark.sql.errors.QueryCompilationErrorsSuite -- -z SPARK-38743" locally on the rebased head; it passed. Ready for CI.

@deepujain
deepujain force-pushed the SPARK-38743-missing-static-partition-column-test branch from dfd1a9a to 3deb8ce Compare August 7, 2026 12:45
@deepujain

deepujain commented Aug 7, 2026

Copy link
Copy Markdown
Contributor Author

Rebased on current master after SPARK-58174 landed. I reran ./build/sbt "sql/testOnly org.apache.spark.sql.errors.QueryCompilationErrorsSuite -- -z SPARK-38743" locally on the rebased head; it passed. Ready for CI.

@deepujain
deepujain force-pushed the SPARK-38743-missing-static-partition-column-test branch from 3deb8ce to 5b61001 Compare August 9, 2026 04:51
@deepujain

deepujain commented Aug 9, 2026

Copy link
Copy Markdown
Contributor Author

Rebased on current master after SPARK-45900 landed. I reran ./build/sbt "sql/testOnly org.apache.spark.sql.errors.QueryCompilationErrorsSuite -- -z SPARK-38743" locally on the rebased head; it passed. Ready for CI.

@deepujain
deepujain force-pushed the SPARK-38743-missing-static-partition-column-test branch from 5b61001 to e89de3e Compare August 9, 2026 14:03
@deepujain

deepujain commented Aug 9, 2026

Copy link
Copy Markdown
Contributor Author

Rebased on current master after SPARK-58549 landed. I reran ./build/sbt "sql/testOnly org.apache.spark.sql.errors.QueryCompilationErrorsSuite -- -z SPARK-38743" locally on the rebased head; it passed. Ready for CI.

@deepujain
deepujain force-pushed the SPARK-38743-missing-static-partition-column-test branch from e89de3e to 79ca061 Compare August 10, 2026 05:12
@deepujain

deepujain commented Aug 10, 2026

Copy link
Copy Markdown
Contributor Author

Rebased on current master after SPARK-58674 landed. I reran ./build/sbt "sql/testOnly org.apache.spark.sql.errors.QueryCompilationErrorsSuite -- -z SPARK-38743" locally on the rebased head; it passed. Ready for CI.

@deepujain
deepujain force-pushed the SPARK-38743-missing-static-partition-column-test branch from 79ca061 to d0c09e2 Compare August 11, 2026 04:50
@deepujain

deepujain commented Aug 11, 2026

Copy link
Copy Markdown
Contributor Author

Rebased on current master after SPARK-58232 landed. I reran ./build/sbt "sql/testOnly org.apache.spark.sql.errors.QueryCompilationErrorsSuite -- -z SPARK-38743" locally on the rebased head; it passed. Ready for CI.

@deepujain
deepujain force-pushed the SPARK-38743-missing-static-partition-column-test branch from d0c09e2 to 9a9a9e4 Compare August 11, 2026 14:44
@deepujain

deepujain commented Aug 11, 2026

Copy link
Copy Markdown
Contributor Author

Rebased on current master to refresh the stale pending Build check. I reran JAVA_HOME=/opt/homebrew/opt/openjdk@17/libexec/openjdk.jdk/Contents/Home PATH="/opt/homebrew/opt/openjdk@17/libexec/openjdk.jdk/Contents/Home/bin:$PATH" SBT_OPTS="-Dspark.driver.bindAddress=127.0.0.1 -Dspark.driver.host=127.0.0.1" ./build/sbt "sql/testOnly org.apache.spark.sql.errors.QueryCompilationErrorsSuite -- -z SPARK-38743" locally; it passed. CI has restarted on the new head.

@deepujain
deepujain force-pushed the SPARK-38743-missing-static-partition-column-test branch from 9a9a9e4 to 2a16240 Compare August 11, 2026 14:52
@deepujain

deepujain commented Aug 11, 2026

Copy link
Copy Markdown
Contributor Author

Spark master moved again during the sweep, so I rebased once more. I reran JAVA_HOME=/opt/homebrew/opt/openjdk@17/libexec/openjdk.jdk/Contents/Home PATH="/opt/homebrew/opt/openjdk@17/libexec/openjdk.jdk/Contents/Home/bin:$PATH" SBT_OPTS="-Dspark.driver.bindAddress=127.0.0.1 -Dspark.driver.host=127.0.0.1" ./build/sbt "sql/testOnly org.apache.spark.sql.errors.QueryCompilationErrorsSuite -- -z SPARK-38743" locally; it passed. CI has restarted on the latest head.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@deepujain