Skip to content

[core] Add compression extension to text format data file names - #9853

Merged
JingsongLi merged 1 commit into
apache:masterfrom
jackylee-ch:text-compression-suffix
Sep 17, 2026
Merged

JingsongLi merged 1 commit into
apache:masterfrom
jackylee-ch:text-compression-suffix

Conversation

@jackylee-ch

Copy link
Copy Markdown
Contributor

Purpose

A text file is always written through the configured codec (AbstractTextFileWriter:44), while the reader recovers the codec from the file name alone; file.compression is not consulted on read. DataFilePathFactory.isTextFormat listed only json and csv, so file.format=text wrote zstd bytes into data-<uuid>-0.text and read them back as UTF-8 lines — garbage rows, no exception.

Format tables are unaffected (formatTableFileCompression() returns none for text); managed tables hit it with no configuration at all, since file.compression defaults to zstd.

Files already written compressed under the old name stay unreadable; this changes new writes only.

Tests

DataFilePathFactoryTest#testTextFormatCarriesTheCompressionExtension. Reverting the fix:

Expecting actual: ...-0.text
to end with: .text.zst

org.apache.paimon.io.*Test: 113 run, 0 failures.

Written with Claude Code; verification is mine.

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the text-compression suffix handling. Keeping writer selection and reader codec inference consistent prevents compressed text files from being interpreted with the wrong codec. The change and coverage look good to me.

@JingsongLi
JingsongLi merged commit f582e30 into apache:master Sep 17, 2026
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants