Skip to content

[core] Fix dedicated-format bundle write path#7598

Open
QuakeWang wants to merge 1 commit intoapache:masterfrom
QuakeWang:fix/dedicated-bundle-path
Open

[core] Fix dedicated-format bundle write path#7598
QuakeWang wants to merge 1 commit intoapache:masterfrom
QuakeWang:fix/dedicated-bundle-path

Conversation

@QuakeWang
Copy link
Copy Markdown

Purpose

Fix the dedicated-format writeBundle path so bundle writes remain correct when data is fanned out to main/blob/vector writers.

This change:

  • materializes only non-replayable bundles in the dedicated fan-out path instead of always materializing them;
  • introduces explicit opt-in bundle capabilities in common (ReplayableBundleRecords and ProjectableBundleRecords) without changing the base BundleRecords contract;
  • allows projection to preserve typed bundle implementations such as ArrowBundleRecords;
  • preserves row-level side effects in bundle writes, including sequence number updates and file-index writing.

Tests

Added or updated tests in paimon-core:

  • ProjectedFileWriterTest
  • BundleAwareRowDataRollingFileWriterTest
  • DedicatedFormatRollingFileWriterTest
  • DedicatedFormatRollingFileWriterVectorTest

These cover:

  • replayable/projectable bundle pass-through and fallback behavior;
  • sequence-number side effects for bundle writes;
  • dedicated-format bundle writes with main-file index side effects;
  • external-storage blob fallback for bundle writes;
  • single-use bundle writes for blob/vector dedicated paths.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant