Repository navigation
test(master): slow-but-alive preparation outlives the idle timeout - #340
Merged
Merged
Conversation
The preparation watchdog is progress-based: it cancels after MAINTENANCE_IDLE_TIMEOUT_SECS without a completed step, not after a total duration. Nothing tested that boundary directly. If the watchdog ever regressed to wall-clock, or if real index IO stopped being counted as a step, a long BTree/ZoneMap build would be cancelled, classified retryable (lance-format#328), and rebuilt every few seconds forever - the lance-format#327 + lance-format#330 + lance-format#328 interaction noted in the scheduler backlog. One etcd-backed test with a 3 s idle timeout covers the three cases: an 8 s preparation completing a step every second finishes; a silent preparation is cancelled within [IDLE, IDLE + 4) with the exact stall error; and a real index build under the same short timeout completes, proving data/_indices IO counts as progress. Mutation check: with the checkpoints removed from case (a), the test fails with "alive preparation was cancelled" after ~4 s. Refs lance-format#333. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
17 of 18 tasks
beinan
added a commit
that referenced
this pull request
Oct 8, 2026
…t IO (#355) Part of #333. Fixes review finding **6** on #340 (credit: production review): the earlier test used hand-placed `checkpoint()`s, a one-row real build, and never asserted the build outlived the idle timeout. ## Scenarios (real BTree preparation, 64 rows, 3 s idle timeout) | | Storage | Expected | Guard against vacuity | |---|---|---|---| | (a) | every data/index op sleeps 1.2 s | completes | asserts ≥ 3 ops intercepted **and** total time > idle timeout | | (b) | parked after first op | cancelled with exact stall error in `[IDLE, IDLE+4)` | asserts shim saw IO | | (c) | healthy table concurrent with (b) | completes; not cancelled; does not keep (b) alive | — | ## Support code - `MergeWriteScope::with_preparation_authorizer_and_io_shim`: layers a shim **beneath** the progress wrapper, so the watchdog counts the shimmed store's completed IO exactly as real storage. - `preparation_io::test_support::{Throttle, ThrottleWrapper}` (`#[doc(hidden)]`). - Uses Lance's `file-object-store://` scheme: local directory through the ObjectStore API so wrappers apply as on cloud stores (plain `file` paths take the direct IO fast path and bypass them — which is why the first attempt saw 0 ops). ## Mutation check Removing read-chunk + `get_ranges` progress accounting from `ProgressStore` fails (a) with `slow-but-alive real build was cancelled` after ~4 s. Restored. ## Validation New test + `slow_but_alive_…` + `prepared_index_task_…` pass against etcd (12 s); clippy `-D warnings`, fmt, typos clean. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #333 (P0.4, backlog C2).
Problem
watch_maintenance_preparationcancels a compaction/index preparation afterMAINTENANCE_IDLE_TIMEOUT_SECSwithout a completed step. That boundary had no direct test. A regression to wall-clock cancellation, or a change that stopped counting real index IO as a step, would cancel every long index build, classify it retryable (#328) and rebuild it every few seconds — the #327 + #330 + #328 interaction flagged in the backlog.Approach
One etcd-backed test, 3 s idle timeout, three cases:
[IDLE, IDLE+4)with the exact"maintenance preparation made no progress"error.data/and_indices/IO is observed as progress through the preparation store wrapper.Mutation check: removing
checkpoint()from case 1 fails the test withalive preparation was cancelledafter ~4 s.Validation
Test passes locally against real etcd (~12 s). Clippy
-D warnings, fmt, typos clean. Test-only change.🤖 Generated with Claude Code