Repository navigation
docs: scheduler improvement backlog and migration order - #335
Merged
beinan merged 1 commit intoOct 8, 2026
Merged
Conversation
Companion to the unified scheduler design. Lists the architectural, correctness, operability and code-health defects in the current maintenance scheduling layer with file:line references, and the P0-P4 order in which to fix them with mixed-version rules for each phase. Headline numbers: 11 loops, 4 pacing mechanisms, 5 backoff systems, 3 admission surfaces, 27 etcd key families, ~71 env knobs, 6 per-table allowlists; ~35 of the last 50 merged PRs touch this layer. Refs lance-format#333. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
17 of 18 tasks
beinan
added a commit
that referenced
this pull request
Oct 8, 2026
Part of #333. Addresses the review comments on #334/#335. ## Changes to the design | Review point | Change | |---|---| | One assignment per table re-serialises merge and compaction | Invariant 1 is now *one write-turn holder* per table; preparation units run concurrently with the turn holder and each other (§4.3, §4.5, §7). Explicit: never re-serialise merge behind preparation (#308, #327). | | Strict class order + in-class aging still starves commit-ready compaction under a hot merger | Two hard bounds in §4.2: `max_consecutive_turns[kind]` (applies across classes, incl. class 1) and `max_turn_wait_secs[kind]` promoting to class 1. Invariant 5 rewritten. | | Delta demand events race and get overwritten; stale snapshots can hide progress | Per-shard watermarks (`sealed_through_seq`), max-merge per shard, sum across shards; snapshot lowers a watermark only with a newer observed revision; new invariant 5a (idempotent folding). | | `bytes_free` heartbeat is a sample, not a reservation | Assignments carry `reserved_bytes`; headroom = `bytes_total − Σ reserved` over live assignments, rebuilt from etcd on failover; executor still enforces local budget (§4.4). | | Shadow phase would leave a scheduling gap | P1 keeps all loops running; scanner *additionally* writes demand; per-table switch only after the fleet is homogeneous (§8, backlog P1/P2 and mixed-version rules). | | Isolating legacy RPCs is not recovery | Stated explicitly in §4.4 and backlog C3: isolation bounds blast radius; stall detection stays on the execution's actual read/encode/commit progress. | ## Factual corrections - `generate_id()` is UUIDv7 (`core/src/id.rs:28`): the queue is approximately enqueue-time ordered; what it lacks is priority. Backlog §0 and C6 corrected. - `failure.rs:74-79` parses line/column as `u32`, so renumbering is safe. What breaks it is a path move, crate rename, or a change in how Lance wraps `InvalidInput`. C1 corrected. Docs only; `typos` clean. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #333. Companion to #334.
What
Adds
docs/design/scheduler-improvement-backlog.md: the defects in the current maintenance scheduling layer, grouped by severity with file:line references, and the ordered P0–P4 plan to fix them.Highlights:
/merge-progresspushing read consistency onto clients; random-id queue order; backpressure as a cliff at 4096 generations.Validation
Docs only.
typosclean locally.🤖 Generated with Claude Code