Skip to content

Project immutable ingestion output before decoding recovery payloads - #332

Open
beinan wants to merge 1 commit into
codex/streaming-ingestionfrom
codex/trace-wal-output-projection
Open

beinan wants to merge 1 commit into
codex/streaming-ingestionfrom
codex/trace-wal-output-projection

Conversation

@beinan

@beinan beinan commented Oct 8, 2026

Copy link
Copy Markdown
Collaborator

Adds OutputRange::read_projected so consumers can omit recovery-only payloads while retaining snapshot, fragment, sequence, physical-byte, decoded-byte and null-record validation. The existing full-record API remains unchanged.

Stacked on #325. Validation: all 69 ingestion tests and Clippy passed; the real failing immutable WAL range was also checked with Rust/Lance 9: all 1,090 rows and nine consumer fields matched the full-read oracle, with retained Arrow output reduced from 279,143,011 to 2,192,709 bytes. A regression covers large omitted metadata, nullable fields, corrupt null records and read limits.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant