Skip to content

Add batched ingestion with local state and replay-safe Lance output - #325

Open
beinan wants to merge 16 commits into
mainfrom
codex/streaming-ingestion
Open

beinan wants to merge 16 commits into
mainfrom
codex/streaming-ingestion

Conversation

@beinan

@beinan beinan commented Oct 6, 2026 •

Copy link
Copy Markdown
Collaborator

Adds reusable batched ingestion primitives with durable receipts, independent WAL consumers and a version-fenced Lance sink. The optional local-state backend keeps binary alignment cells in disposable local storage and commits matching state deltas, exact receipts and typed output in one Lance 2.2 batch with Zstd.

Immutable checkpoints support disk-loss recovery. Bounded binary range descriptors let independent consumers validate the exact run/schema/partition, version watermarks, fragment ancestry and payload budgets. Shared Arrow IPC allocations are counted once so wide schemas do not prematurely split small batches. Application-specific alignment, legacy migration and deployment remain outside this crate.

Validated with real Lance tests for coalescing, retries, stale writers, missing files, lost commit responses, checkpoint recovery and range corruption. This remains experimental; no production throughput claim.

@beinan beinan changed the title Add bounded streaming ingestion with replay-safe Lance sink Add batched ingestion with durable WAL and replay-safe Lance sink Oct 7, 2026
@beinan beinan changed the title Add batched ingestion with durable WAL and replay-safe Lance sink Add batched ingestion with local state and replay-safe Lance output Oct 7, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant