What is the problem the feature request solves?
Comet does not support TimestampType join keys in SMJ. This is because of an upstream restriction:
DataFusion's SMJ comparator at datafusion-physical-plan-53.0.0/src/joins/sort_merge_join/stream.rs:1820 and utils.rs:1900 matches only DataType::Timestamp(time_unit, None). Spark's TimestampType serializes to Timestamp(µs, "UTC"), so it hits the fallthrough not_impl_err!.
Describe the potential solution
No response
Additional context
No response
What is the problem the feature request solves?
Comet does not support TimestampType join keys in SMJ. This is because of an upstream restriction:
DataFusion's SMJ comparator at
datafusion-physical-plan-53.0.0/src/joins/sort_merge_join/stream.rs:1820andutils.rs:1900matches onlyDataType::Timestamp(time_unit, None). Spark's TimestampType serializes to Timestamp(µs, "UTC"), so it hits the fallthroughnot_impl_err!.Describe the potential solution
No response
Additional context
No response