Substrait producer does not support LogicalPlan::Unnest

Is your feature request related to a problem or challenge?

INTERSECT ALL and EXCEPT ALL are planned with an aggregate that counts the rows on each side, a join, and an unnest of range that repeats each row, as proposed in #25767. The Substrait producer has no case for the Unnest plan node, so exporting one of these plans fails with "Unsupported plan type: Unnest". The two tests at

datafusion/datafusion/substrait/tests/cases/roundtrip_logical_plan.rs

Lines 1806 to 1835 in 1f95f9a

    
           // `INTERSECT ALL` and `EXCEPT ALL` are planned with `unnest(range(..))` to 
        
           // repeat each row, and the producer cannot serialize `Unnest` yet, so these 
        
           // plans cannot be sent to Substrait. Consuming Substrait multiset set 
        
           // operations still works (see `multiset_intersect_all_consume`). 
        
           #[tokio::test] 
        
           async fn self_referential_intersect_all() -> Result<()> { 
        
               assert_unnest_not_supported( 
        
                   "SELECT a FROM data WHERE a > 0 INTERSECT ALL SELECT a FROM data WHERE a < 5", 
        
               ) 
        
               .await 
        
           } 
        
           #[tokio::test] 
        
           async fn self_referential_except_all() -> Result<()> { 
        
               assert_unnest_not_supported( 
        
                   "SELECT a FROM data WHERE a > 0 EXCEPT ALL SELECT a FROM data WHERE a < 5", 
        
               ) 
        
               .await 
        
           } 
        
           async fn assert_unnest_not_supported(sql: &str) -> Result<()> { 
        
               let ctx = create_context().await?; 
        
               let plan = ctx.sql(sql).await?.into_optimized_plan()?; 
        
               let err = to_substrait_plan(&plan, &ctx.state()).unwrap_err(); 
        
               assert!( 
        
                   err.to_string().contains("Unsupported plan type: Unnest"), 
        
                   "{err}" 
        
               ); 
        
               Ok(()) 
        
           }

show the error for both operations, and the producer rejects the node at

datafusion/datafusion/substrait/src/logical_plan/producer/rel/mod.rs

Line 76 in 1f95f9a

LogicalPlan::Unnest(plan) => not_impl_err!("Unsupported plan type: {plan:?}")?,

.

Consuming Substrait multiset set operations still works, so only the export is affected. Before that change these plans were exported as a semi join or an anti join, which returned the wrong number of duplicate rows.

Describe the solution you'd like

Support LogicalPlan::Unnest in the Substrait producer, so that INTERSECT ALL and EXCEPT ALL plans can be exported again with the correct duplicate counts. I do not know yet which Substrait relation should carry it, so I would welcome a pointer from someone who has looked at that.

Describe alternatives you've considered

None so far.

Additional context

The consuming side of multiset set operations is covered by the tests at

datafusion/datafusion/substrait/tests/cases/roundtrip_logical_plan.rs

Lines 1655 to 1688 in 1f95f9a

    
           #[tokio::test] 
        
           async fn multiset_intersect_all_consume() -> Result<()> { 
        
               let proto_plan = 
        
                   read_json("tests/testdata/test_plans/intersect_multiset_all.substrait.json"); 
        
               assert_substrait_sql( 
        
                   proto_plan, 
        
                   "SELECT a FROM data INTERSECT ALL SELECT a FROM data2 INTERSECT ALL SELECT a FROM data2", 
        
               ) 
        
               .await 
        
           } 
        
           #[tokio::test] 
        
           async fn primary_except_consume() -> Result<()> { 
        
               let proto_plan = read_json("tests/testdata/test_plans/minus_primary.substrait.json"); 
        
               assert_substrait_sql( 
        
                   proto_plan, 
        
                   "SELECT a FROM data EXCEPT SELECT a FROM data2 EXCEPT SELECT a FROM data2", 
        
               ) 
        
               .await 
        
           } 
        
           #[tokio::test] 
        
           async fn primary_except_all_consume() -> Result<()> { 
        
               let proto_plan = 
        
                   read_json("tests/testdata/test_plans/minus_primary_all.substrait.json"); 
        
               assert_substrait_sql( 
        
                   proto_plan, 
        
                   "SELECT a FROM data EXCEPT ALL SELECT a FROM data2 EXCEPT ALL SELECT a FROM data2", 
        
               ) 
        
               .await 
        
           }

.

Willingness to contribute

I cannot contribute to this feature at this time

docs: note that INTERSECT ALL and EXCEPT ALL plans cannot be exported…

Navigation Menu

Repository navigation

Substrait producer does not support LogicalPlan::Unnest #26372

Is your feature request related to a problem or challenge?

Describe the solution you'd like

Describe alternatives you've considered

Additional context

Willingness to contribute

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	// `INTERSECT ALL` and `EXCEPT ALL` are planned with `unnest(range(..))` to
	// repeat each row, and the producer cannot serialize `Unnest` yet, so these
	// plans cannot be sent to Substrait. Consuming Substrait multiset set
	// operations still works (see `multiset_intersect_all_consume`).
	#[tokio::test]
	async fn self_referential_intersect_all() -> Result<()> {
	assert_unnest_not_supported(
	"SELECT a FROM data WHERE a > 0 INTERSECT ALL SELECT a FROM data WHERE a < 5",
	)
	.await
	}

	#[tokio::test]
	async fn self_referential_except_all() -> Result<()> {
	assert_unnest_not_supported(
	"SELECT a FROM data WHERE a > 0 EXCEPT ALL SELECT a FROM data WHERE a < 5",
	)
	.await
	}

	async fn assert_unnest_not_supported(sql: &str) -> Result<()> {
	let ctx = create_context().await?;
	let plan = ctx.sql(sql).await?.into_optimized_plan()?;
	let err = to_substrait_plan(&plan, &ctx.state()).unwrap_err();
	assert!(
	err.to_string().contains("Unsupported plan type: Unnest"),
	"{err}"
	);
	Ok(())
	}

	#[tokio::test]
	async fn multiset_intersect_all_consume() -> Result<()> {
	let proto_plan =
	read_json("tests/testdata/test_plans/intersect_multiset_all.substrait.json");

	assert_substrait_sql(
	proto_plan,
	"SELECT a FROM data INTERSECT ALL SELECT a FROM data2 INTERSECT ALL SELECT a FROM data2",
	)
	.await
	}

	#[tokio::test]
	async fn primary_except_consume() -> Result<()> {
	let proto_plan = read_json("tests/testdata/test_plans/minus_primary.substrait.json");

	assert_substrait_sql(
	proto_plan,
	"SELECT a FROM data EXCEPT SELECT a FROM data2 EXCEPT SELECT a FROM data2",
	)
	.await
	}

	#[tokio::test]
	async fn primary_except_all_consume() -> Result<()> {
	let proto_plan =
	read_json("tests/testdata/test_plans/minus_primary_all.substrait.json");

	assert_substrait_sql(
	proto_plan,
	"SELECT a FROM data EXCEPT ALL SELECT a FROM data2 EXCEPT ALL SELECT a FROM data2",
	)
	.await
	}

Substrait producer does not support LogicalPlan::Unnest #26372

Description

Is your feature request related to a problem or challenge?

Describe the solution you'd like

Describe alternatives you've considered

Additional context

Willingness to contribute

Activity

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions