feat(writer): add PositionDeleteFileWriter - #2986
Open
laskoviymishka wants to merge 2 commits into
Open
Conversation
laskoviymishka
force-pushed
the
feat/position-delete-writer
branch
from
August 11, 2026 16:26
c242c13 to
054dc33
Compare
Add PositionDeleteFileWriter and its builder under writer/base_writer. It writes a position delete file with the two required columns file_path (string, field id 2147483546) and pos (long, 2147483545) and sets DataContentType::PositionDeletes on close. position_delete_schema() and position_delete_arrow_schema() return the canonical schema, built from the existing metadata_columns field definitions. write() checks that a batch has exactly those two required, non-nullable, correctly typed columns before writing, and rejects a closed writer. close() propagates the partition key and leaves sort_order_id null. The writer does not sort its input; the write() docs note that the caller must supply rows sorted by (file_path, pos) until a later writer enforces it. Setting referenced_data_file and a higher-level DeltaWriter are separate follow-ups. Tested: schema shape, a parquet round trip for single and multiple writes, partition propagation, and the validation cases (wrong column count, wrong or missing field ids, wrong types including LargeUtf8, nullable columns, and writes after close). Refs apache#340.
laskoviymishka
force-pushed
the
feat/position-delete-writer
branch
from
August 11, 2026 19:13
054dc33 to
6f76750
Compare
laskoviymishka
marked this pull request as ready for review
August 12, 2026 11:11
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #340.
What changes are included in this PR?
This adds
PositionDeleteFileWriterunderwriter/base_writer, next to the dataand equality-delete writers. We already had a writer for equality deletes but
nothing that writes position delete files, so I modelled this one on
equality_delete_writer.rs.It writes the two required columns,
file_path(string, field id 2147483546) andpos(long, 2147483545), and sets the content type toPositionDeleteson close.position_delete_schema()/position_delete_arrow_schema()hand back thecanonical schema, built from the field definitions already in
metadata_columnsso the reserved ids aren't duplicated here.
write()checks the batch is exactly those two required, non-null, correctlytyped columns before handing it to the parquet writer, so a bad batch fails with a
clear message instead of a confusing error further down.
close()carries thepartition key onto the data file and leaves
sort_order_idnull.One thing worth calling out: this writer doesn't sort its input. Position deletes
have to be sorted by
file_paththenpos, and for now that's on the caller (thewrite()docs say so). I'd rather land the plain writer first and put sorting ontop of it, so a sorting writer,
referenced_data_file, and a higher-levelDeltaWriter(#2218) are follow-ups instead of being crammed in here.Are these changes tested?
Yes, 12 unit tests: the schema shape, a parquet round trip for single and multiple
writes (write it, read it back, compare), partition and spec-id propagation on
close, and the rejection cases: wrong column count, wrong or missing field ids on
either column, wrong types (including LargeUtf8, which is what a lot of Arrow
producers give you), nullable columns, and writing after close.
cargo test -p iceberg,clippy, andfmtare all clean.AI assistance disclosure
Drafted with help from Claude Code, then reviewed and tested by me before opening.