Skip to content

issue: 2607- added support for MapFromEntries#3991

Draft
athlcode wants to merge 6 commits intoapache:mainfrom
athlcode:support/MapFromEntries
Draft

issue: 2607- added support for MapFromEntries#3991
athlcode wants to merge 6 commits intoapache:mainfrom
athlcode:support/MapFromEntries

Conversation

@athlcode
Copy link
Copy Markdown

@athlcode athlcode commented Apr 18, 2026

Which issue does this PR close?

Closes #2706.

Rationale for this change

Comet already delegated MapFromEntries to the upstream datafusion-spark UDF, but the docs still listed it as unsupported and the native path silently collapsed duplicate keys — diverging from Spark's default spark.sql.mapKeyDedupPolicy=EXCEPTION. Paired with the companion datafusion PR (which makes the native UDF raise [DUPLICATED_MAP_KEY] on duplicates).

What changes are included in this PR?

  1. Flip map_from_entries to supported in docs/spark_expressions_support.md.
  2. In maps.scala, add a guard in CometMapFromEntries.getSupportLevel that returns Incompatible when spark.sql.mapKeyDedupPolicy is not EXCEPTION, so LAST_WIN queries fall back to Spark. Binary key/value fallbacks are unchanged.

How are these changes tested?

One new test in CometMapExpressionSuite.scala:

map_from_entries - duplicate keys throw under default EXCEPTION policy — asserts the native path surfaces DUPLICATED_MAP_KEY, confirming error-class parity with Spark.

@coderfender
Copy link
Copy Markdown
Contributor

@athlcode , you might want to wait till the PR on DF's side is merged and released before adding support on comet :
apache/datafusion#21720

@coderfender
Copy link
Copy Markdown
Contributor

Please take a look at : https://datafusion.apache.org/comet/contributor-guide/index.html for help with code formatting, github issue naming conventions and other useful resources.

@athlcode athlcode marked this pull request as draft April 19, 2026 17:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add support for MapFromEntries

2 participants