Repository navigation
docs: fold production review into the scheduler design - #337
Merged
beinan merged 1 commit intoOct 8, 2026
Merged
Conversation
Corrections from review of lance-format#333/lance-format#334/lance-format#335: - One write-turn holder per table, not one assignment per table. Preparation units run concurrently with the turn holder and with each other; the scheduler must never re-serialise merge behind compaction or index preparation (lance-format#308, lance-format#327). - Class ordering plus in-class aging still starves commit-ready compaction under a hot merger. Add hard bounds: max_consecutive_turns per kind (applies across classes) and max_turn_wait_secs promoting to class 1. - Demand events become per-shard watermarks (sealed_through_seq), which are idempotent under loss and duplication; stats snapshots may lower a watermark only with a newer observed revision. - Heartbeat bytes_free is a sample. Assignments carry reserved_bytes; the planner computes executor headroom from live assignments, rebuilt on failover; executors still enforce the local memory budget. - Shadow phase keeps every existing loop running; per-table switch to the planner happens only on a homogeneous fleet. - Isolating legacy RPCs bounds blast radius only; stall detection stays on the execution's actual read/encode/commit progress. Factual fixes: generate_id() is UUIDv7 (time-ordered, not random) so the queue is approximately FIFO without priority; failure.rs accepts any line/column digits, so renumbering is safe while path or wrapper changes are not. Refs lance-format#333. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
17 of 18 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #333. Addresses the review comments on #334/#335.
Changes to the design
max_consecutive_turns[kind](applies across classes, incl. class 1) andmax_turn_wait_secs[kind]promoting to class 1. Invariant 5 rewritten.sealed_through_seq), max-merge per shard, sum across shards; snapshot lowers a watermark only with a newer observed revision; new invariant 5a (idempotent folding).bytes_freeheartbeat is a sample, not a reservationreserved_bytes; headroom =bytes_total − Σ reservedover live assignments, rebuilt from etcd on failover; executor still enforces local budget (§4.4).Factual corrections
generate_id()is UUIDv7 (core/src/id.rs:28): the queue is approximately enqueue-time ordered; what it lacks is priority. Backlog §0 and C6 corrected.failure.rs:74-79parses line/column asu32, so renumbering is safe. What breaks it is a path move, crate rename, or a change in how Lance wrapsInvalidInput. C1 corrected.Docs only;
typosclean.🤖 Generated with Claude Code