Repository navigation
fix(merge): classify stale preparations by a stable error code - #338
Merged
beinan merged 1 commit intoOct 8, 2026
Merged
Conversation
The failure classifier recognised stale index/compaction preparations by matching the exact prose and the literal source path `crates/lance-context-core/src/store_base.rs:` inside the string Lance produces for `InvalidInput`. Moving the check to another file, renaming the crate, or a change in how Lance renders the wrapper would silently turn a two-second retry back into a one-hour cooldown with `needs_attention` set - the production failure lance-format#328 fixed. Add `MaintenanceErrorCode` to lance-context-api with one variant, `StalePreparation`, rendered as `[LC_STALE_PREPARATION]`. Every stale-preparation error site in store_base.rs (index ×3, compaction ×2, merge watermark overlap ×1) tags its message. `classify` checks ownership first, then the code, then falls back to the legacy text matcher for failure records written by older binaries; `corrected()` accepts either form. The legacy matcher is now documented as removable once no stored record lacks a code. Alternatives: a typed error enum through lance::Error would be cleaner but errors cross HTTP and etcd as strings; a code inside the string is the smallest change that survives every boundary. The api crate hosts the enum so core, merge and master share it without a cycle. Validation: new unit tests cover code round-trip through wrappers and paths, ownership precedence, and correction of old records carrying a code. merge crate: 11 unit + 10 etcd tests pass; core prepared/stale tests pass; clippy -D warnings, fmt and typos clean. Refs lance-format#333. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
17 of 18 tasks
beinan
added a commit
that referenced
this pull request
Oct 8, 2026
…/403 (#347) Part of #333. Fixes two defects in #338 found in review (credit: production review). ## Problems 1. **Mixed-version regression (P1).** #338 changed the exact string old masters match on. A stale preparation produced by a new executor and classified by an old master → `DataOrConfiguration`, one-hour cooldown — the bug #338 set out to fix. Reproduced with old vs new classifier. 2. **Code outranks a 403 (P2).** `classify` checked the code before HTTP status, so `HTTP 403: … [LC_STALE_PREPARATION]` → `Retryable`. The #338 description claimed a test for this; there was none. `corrected()` also had its own rule instead of sharing `classify`'s. ## Fixes - Emission **off by default**; `MAINTENANCE_ERROR_CODES=1` enables it after every master/executor runs a code-aware reader. `tag()` returns the exact legacy string when off; `tag_forced()` for tests. - Code moved to a **suffix**; the legacy text matcher tolerates the suffix (defence in depth for a reader with only the text path). - `classify` order: ownership → explicit 401/403 → code → text. `corrected()` delegates to `classify` and only softens a stored `DataOrConfiguration`. - `docs/concurrent-indexing.md` documents the rollout order. ## Validation - New: 401/403 wrapping the code in either position → `DataOrConfiguration`, and `corrected()` leaves it; emission off by default; byte-exact legacy string still matched; tagged string recognised by text path alone. - `lance-context-api` 1/1, `lance-context-merge` failure 7/7, `lance-context-core` prepared/stale 9/9; clippy `-D warnings`, fmt, typos clean. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #333 (P0.2, backlog C1).
Problem
merge/failure.rsrecognised stale index/compaction preparations by matching exact prose plus the literal pathcrates/lance-context-core/src/store_base.rs:in the string Lance renders forInvalidInput. Moving the check, renaming the crate, or a change in Lance's wrapper format silently reverts a 2 s retry to a 1 h cooldown withneeds_attention— the production failure #328 fixed. (Line numbers are not the fragility; any digits are accepted.)Approach
lance-context-api:MaintenanceErrorCode::StalePreparation→[LC_STALE_PREPARATION], withtag()/parse(). The api crate hosts it so core, merge and master share it without a dependency cycle.store_base.rs: all six stale-preparation sites tag their message (index ×3, compaction ×2, merge watermark overlap ×1).failure.rs:classifychecks ownership first, then the code, then falls back to the legacy text matcher for records written by old binaries.corrected()accepts either form. The legacy matcher is marked removable once no stored record lacks a code.docs/concurrent-indexing.mdupdated.Alternatives
A typed error enum through
lance::Erroris cleaner, but errors cross worker HTTP and etcd task records as strings; a code inside the string is the smallest change that survives every boundary. Leaving the text matcher as the only path: rejected, it is the bug's root cause.Validation
DataOrConfigurationrecord with a code is corrected without resetting attempts/clock.lance-context-merge: 11 unit + 10 etcd-backed tests pass locally;lance-context-coreprepared/stale tests pass;clippy --all-targets -D warnings,fmt,typosclean.🤖 Generated with Claude Code