[spark] Add union read for lake-enabled log tables

[fresh-borzoni]

Summary

closes #2983

Adds batch read for lake-enabled log tables. When a table has datalake enabled, reads combine lake storage (Paimon/Iceberg) with Fluss log tail. Lake and log are planned as separate Spark partition, lake tasks read from lake storage without Fluss connections, log tail tasks reuse the existing reader. Falls back to pure log reads when no snapshot exists. Only enabled in FULL startup mode.

Tests cover both Paimon and Iceberg.

Follow-up PRs

• PK table lake reads (sort-merge with lake snapshot)
• Streaming with lake bootstrap
• Filter/partition/limit push-down to lake source
• DV support for Paimon

[Yohahaha]

@fresh-borzoni thank you for the patch, I create an issue to track it #2983.

[Yohahaha]

PK table lake reads (sort-merge with lake snapshot)

I will add spark sql support for union read pk table, #2984

[fresh-borzoni]

Thanks @Yohahaha! Just a heads up, this PR already implements batch union read for log tables, so #2983 should be covered once it's merged.

Regarding PK table union read (#2984),I was planning to follow up with that as noted in the PR description.
Happy to collaborate if you're interested, let me know!

[Yohahaha]

Thanks @Yohahaha! Just a heads up, this PR already implements batch union read for log tables, so #2983 should be covered once it's merged.

yeah, you could add "closes #2983" in the PR description so that the corresponding issue can be properly linked and closed, like other PRs.

when fluss release a new version, RM can easily collect the features of version scope.

[Yohahaha]

Regarding PK table union read (#2984),I was planning to follow up with that as noted in the PR description. Happy to collaborate if you're interested, let me know!

@fresh-borzoni I was planing to implementing it over the next two weeks. Do you already have a draft PR?

[Yohahaha]

left some comments, thank you!

[fresh-borzoni]

@Yohahaha Thank you, feel free to take #2984, I have some starting code, but I'd love to collaborate and appreciate your help here

[fresh-borzoni]

@Yohahaha Ty for the review,
Addressed comments, PTAL 🙏