Skip to content

An open seals what the engine before it left unsealed, so a moved shard's backlog does not read as lag for 45 s - #145

Closed
sorenbs wants to merge 2 commits into
slatefrom
seal-inherited
Closed

An open seals what the engine before it left unsealed, so a moved shard's backlog does not read as lag for 45 s#145
sorenbs wants to merge 2 commits into
slatefrom
seal-inherited

Conversation

@sorenbs

@sorenbs sorenbs commented Oct 31, 2026

Copy link
Copy Markdown
Member

What

An open that finds pages the engine before it left unsealed now seals them at the sealer's first evaluation, through the deferred flag a budget-capped seal already uses, instead of waiting for the 45 s age trigger.

Why

A shard that moves to a new owner carries its WAL backlog unsealed: a lag move retires the old engine without a seal, and the new owner's open seals what is left. But the open restarts the backlog's age, and a few MB trips no size or WAL trigger, so nothing sealed it for 45 s. All that time the new owner reported the backlog as maintenance without progress (no_progress_secs, the heartbeat's absorb_lag_max_secs):

  • at 30 s or more a peer is no lag-move target, and no shard returns to it;
  • past REBALANCE_LAG_SECS (60 s), when a slow store stretches the seal, it counts as lagging and its shards move again.

In a 16-instance fleet on one object store (Habanero's services bench), this kept 6 of 64 shards moving with no load. A moved-in shard showed 2.5 MB unabsorbed, backlog_started_ms == last_progress_ms, and no_progress_secs past 39.

Tests

  • shard::seal::lifecycle_tests::an_open_seals_the_pages_the_engine_before_it_left_unsealed closes an engine whose pages sit below every trigger, with an age trigger of an hour. The next engine's open must seal them within 5 s. Without the change it fails at 5 s.
  • dst::dst_tests::billing_maintenance::ownership_handoff_moves_backlog_without_aba watches whether the old owner's late cleanup changes the new owner's backlog. It now pauses the new owner's sealer, which would otherwise seal the backlog it inherited. Its assertions are unchanged; it passed 20 runs in a row.
  • scripts/dev/preflight.sh and scripts/gate.sh pass. TLA-016 was re-recorded for the changed inputs. CI runs the mutation owners shard_seal and shard_seal_open.
  • One gate run failed bootstrap::tests::write_tier::an_isolated_append_waits_for_one_wal_write_and_the_rest_of_the_gap below its lower bound (465 ms against 556) under the full suite. It passed 5 of 5 runs alone and on the next full gate. The test opens a fresh shard with nothing unsealed, so this change does not reach it. It looks timing-sensitive under load and is not fixed here.

🤖 Generated with Claude Code

sorenbs and others added 2 commits October 31, 2026 19:14
@sorenbs @claude
An open seals what the engine before it left unsealed, instead of wai…
357584e 
…ting 45 s for the age trigger

A shard that moves to a new owner carries its WAL backlog unsealed: a lag
move retires the old engine without a seal, and the next owner's open
seals what is left. The open restarted the backlog's age, though, and no
size or WAL trigger fires on a few MB, so nothing sealed it until the
45 s age trigger. All that time the new owner reported the backlog as
maintenance without progress: a peer at 30 s or more is no lag-move
target and no shard returns to it, and past REBALANCE_LAG_SECS, when a
slow store stretches the seal, it counts as lagging and its shards move
again. In a 16-instance fleet on one object store this kept 6 of 64
shards moving with no load.

An open that finds unsealed pages now sets the sealer's deferred flag,
so the loop seals at its first evaluation, as after a seal its budget
cut short. A lifecycle test closes an engine with pages below every
trigger and an age trigger of an hour, and the next engine's open seals
them within 5 s; without the change it waits the hour. The handoff test
that watches the old owner's late cleanup pauses the new owner's sealer,
which would otherwise seal the backlog it inherited.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sorenbs @claude
TLA-016 re-recorded for the seal at an open that inherits unsealed pages
d10d4e6 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sorenbs added a commit that referenced this pull request Nov 1, 2026
@sorenbs @claude
A failed inherited seal is retried at its backoff, not left to the ag…
1cfebba 
…e trigger the open restarted

#145 makes an open that finds pages a previous owner left
unsealed seal them at its first evaluation, through the sealer's deferred
flag. The loop clears that flag before the seal runs, and only a budget-cut
seal re-arms it, so an inherited seal that failed once (a store error at
its object PUT, a committer refusal) fell back to the triggers after its
backoff, whose age had restarted at the open: the backlog waited the full
age_seal (45 s) again, exactly on the slow store the change is for.

seal_triggered (src/shard/seal.rs) now re-arms the flag when a Deferred
seal fails with anything but Closed; the backoff check before the flag
keeps it from looping hot.

The inherited-seal tests now assert exact state (Fable 5.1 review of #145):
- an_open_seals_the_pages_the_engine_before_it_left_unsealed: the new
  engine's counters are seals 1, empty seals 0, failures 0, objects 1, and
  every acked record reads back durable;
- a_failed_inherited_seal_is_retried_at_its_backoff (new): the first
  inherited seal fails at its PUT, the next seals within 3 s; counters
  1/0/1/1 and every record reads back;
- an_open_with_nothing_unsealed_seals_nothing (new): an engine that sealed
  everything leaves its successor no seal (counters 0/0/0/0 a second after
  the open), so a cell restart adds no object PUT.

Red, without the seal.rs change:
  thread 'shard::seal::lifecycle_tests::a_failed_inherited_seal_is_retried_at_its_backoff'
  panicked at src/shard/seal/lifecycle_tests.rs:534:9:
  the inherited backlog was not sealed within 3s
  test result: FAILED. 2 passed; 1 failed
#145's own test is red on its unfixed tree too: an open restarts the
backlog's age (3600 s in the test) and nothing else fires, so it panics
"the inherited backlog was not sealed within 5 s".
Green: cargo test --lib shard::seal:: dst_tests::idle_release::
dst_tests::billing_maintenance:: 126 passed.

Ledgers: none changed (lifecycle tests are not DST-inventoried; pins and
the scenario map check clean). Formal: TLA-016 digests seal.rs, so its
receipt is re-recorded in the next commit. Edge change: none on the wire;
the operator surfaces (maintenance view, fleet lag) see an inherited
backlog seal at once (#145's change; recorded with it if the owner wants a
record).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sorenbs added a commit that referenced this pull request Nov 1, 2026
@sorenbs @claude
A scale-validation guide tells an agent on Prisma Compute how to test…
b2bb5d4 
… streams and queues at 1-2 shards per instance and the 100 ms WAL gap, what each costs at Class A/B -50%, and what it earns at Cloudflare's prices

Why: an agent validating at scale on Compute ran the bench with FLEET_MAX=64 (64 shards on one
instance) and WAL_FLUSH_GAP_MS=25, which multiplies WAL PUT cost per instance. The owner's posture
is 1-2 shards per instance, the binary's 100 ms gap, one WAL PUT in flight per shard.

docs/SCALE-VALIDATION.md: the deploy posture (exact env, INITIAL_SHARDS = N or 2N on a fresh
PATH_PREFIX, FLEET_MIN = FLEET_MAX, the settings that void a cost result, how to verify posture from
/v1/debug/{load,timings,store}, /operator/data.json and the boot log); forwarding as PR #143-#145
describe it (queues and live:wait not forwarded); streams tests ST-1..ST-4 (cell anchor PACK-T-1000
and PACK-S-MUX-0200, fleet linearity at 1/8/16 instances, M and L, idle floor) with void and
fail-early rules; queue 1x/10x/100x and hot-queue runs; the cost model (WAL PUTs per busy shard,
$48.60 a shard-month at 10/s and $1.875/M; Class A and B per shard as measured); measured cost against
K2 and Cloudflare Queues revenue; the owner's access-fee / marketing-expense ruling for T and S;
proposed acceptance bars; the report template; open items for the owner. Every figure cites a repo
path or is marked as program notes. docs/README.md: one map entry.

Red/green: none (documentation only; nothing built or run).
Ledgers: none. Formal receipts: none stale. Edge change: none.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sorenbs sorenbs mentioned this pull request Nov 1, 2026
Closed
sorenbs added a commit that referenced this pull request Nov 1, 2026
@sorenbs @claude
A failed inherited seal is retried at its backoff, not left to the ag…
86e2894 
…e trigger the open restarted

#145 makes an open that finds pages a previous owner left
unsealed seal them at its first evaluation, through the sealer's deferred
flag. The loop clears that flag before the seal runs, and only a budget-cut
seal re-arms it, so an inherited seal that failed once (a store error at
its object PUT, a committer refusal) fell back to the triggers after its
backoff, whose age had restarted at the open: the backlog waited the full
age_seal (45 s) again, exactly on the slow store the change is for.

seal_triggered (src/shard/seal.rs) now re-arms the flag when a Deferred
seal fails with anything but Closed; the backoff check before the flag
keeps it from looping hot.

The inherited-seal tests now assert exact state (Fable 5.1 review of #145):
- an_open_seals_the_pages_the_engine_before_it_left_unsealed: the new
  engine's counters are seals 1, empty seals 0, failures 0, objects 1, and
  every acked record reads back durable;
- a_failed_inherited_seal_is_retried_at_its_backoff (new): the first
  inherited seal fails at its PUT, the next seals within 3 s; counters
  1/0/1/1 and every record reads back;
- an_open_with_nothing_unsealed_seals_nothing (new): an engine that sealed
  everything leaves its successor no seal (counters 0/0/0/0 a second after
  the open), so a cell restart adds no object PUT.

Red, without the seal.rs change:
  thread 'shard::seal::lifecycle_tests::a_failed_inherited_seal_is_retried_at_its_backoff'
  panicked at src/shard/seal/lifecycle_tests.rs:534:9:
  the inherited backlog was not sealed within 3s
  test result: FAILED. 2 passed; 1 failed
#145's own test is red on its unfixed tree too: an open restarts the
backlog's age (3600 s in the test) and nothing else fires, so it panics
"the inherited backlog was not sealed within 5 s".
Green: cargo test --lib shard::seal:: dst_tests::idle_release::
dst_tests::billing_maintenance:: 126 passed.

Ledgers: none changed (lifecycle tests are not DST-inventoried; pins and
the scenario map check clean). Formal: TLA-016 digests seal.rs, so its
receipt is re-recorded in the next commit. Edge change: none on the wire;
the operator surfaces (maintenance view, fleet lag) see an inherited
backlog seal at once (#145's change; recorded with it if the owner wants a
record).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sorenbs added a commit that referenced this pull request Nov 1, 2026
@sorenbs @claude
A scale-validation guide tells an agent on Prisma Compute how to test…
ce78412 
… streams and queues at 1-2 shards per instance and the 100 ms WAL gap, what each costs at Class A/B -50%, and what it earns at Cloudflare's prices

Why: an agent validating at scale on Compute ran the bench with FLEET_MAX=64 (64 shards on one
instance) and WAL_FLUSH_GAP_MS=25, which multiplies WAL PUT cost per instance. The owner's posture
is 1-2 shards per instance, the binary's 100 ms gap, one WAL PUT in flight per shard.

docs/SCALE-VALIDATION.md: the deploy posture (exact env, INITIAL_SHARDS = N or 2N on a fresh
PATH_PREFIX, FLEET_MIN = FLEET_MAX, the settings that void a cost result, how to verify posture from
/v1/debug/{load,timings,store}, /operator/data.json and the boot log); forwarding as PR #143-#145
describe it (queues and live:wait not forwarded); streams tests ST-1..ST-4 (cell anchor PACK-T-1000
and PACK-S-MUX-0200, fleet linearity at 1/8/16 instances, M and L, idle floor) with void and
fail-early rules; queue 1x/10x/100x and hot-queue runs; the cost model (WAL PUTs per busy shard,
$48.60 a shard-month at 10/s and $1.875/M; Class A and B per shard as measured); measured cost against
K2 and Cloudflare Queues revenue; the owner's access-fee / marketing-expense ruling for T and S;
proposed acceptance bars; the report template; open items for the owner. Every figure cites a repo
path or is marked as program notes. docs/README.md: one map entry.

Red/green: none (documentation only; nothing built or run).
Ledgers: none. Formal receipts: none stale. Edge change: none.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sorenbs added a commit that referenced this pull request Nov 1, 2026
@sorenbs @claude
The segment cleaner's edge record is #237, so #235 stays the owner-ra…
eb48455 
…tified forwarding record

f84208f wrote the cleaner's record (additive seal.gc counters) as #235, a number the lead's
integration stack (prisma/streams #143-#145) already carries for the forwarding change the owner
ratified, cited across its commits and code docs, with #236 for the count balance. The lead asked
for this record to move.
Change: docs/reviews/2026-09-hardening/edge-changes.md (the heading and status of the cleaner's
record, now #237, with a note of its first number) and its WIRE-MATRIX line. No code, test or
other document in the tree cites the cleaner's #235.
Red/green: none (documentation). Preflight passes; the gate ran on this commit.
Ledgers: none. Formal: none. Edge change #237 (renumbered; awaiting the lead's ratification).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sorenbs added a commit that referenced this pull request Nov 1, 2026
@sorenbs @claude
A failed inherited seal is retried at its backoff, not left to the ag…
48a655f 
…e trigger the open restarted

#145 makes an open that finds pages a previous owner left
unsealed seal them at its first evaluation, through the sealer's deferred
flag. The loop clears that flag before the seal runs, and only a budget-cut
seal re-arms it, so an inherited seal that failed once (a store error at
its object PUT, a committer refusal) fell back to the triggers after its
backoff, whose age had restarted at the open: the backlog waited the full
age_seal (45 s) again, exactly on the slow store the change is for.

seal_triggered (src/shard/seal.rs) now re-arms the flag when a Deferred
seal fails with anything but Closed; the backoff check before the flag
keeps it from looping hot.

The inherited-seal tests now assert exact state (Fable 5.1 review of #145):
- an_open_seals_the_pages_the_engine_before_it_left_unsealed: the new
  engine's counters are seals 1, empty seals 0, failures 0, objects 1, and
  every acked record reads back durable;
- a_failed_inherited_seal_is_retried_at_its_backoff (new): the first
  inherited seal fails at its PUT, the next seals within 3 s; counters
  1/0/1/1 and every record reads back;
- an_open_with_nothing_unsealed_seals_nothing (new): an engine that sealed
  everything leaves its successor no seal (counters 0/0/0/0 a second after
  the open), so a cell restart adds no object PUT.

Red, without the seal.rs change:
  thread 'shard::seal::lifecycle_tests::a_failed_inherited_seal_is_retried_at_its_backoff'
  panicked at src/shard/seal/lifecycle_tests.rs:534:9:
  the inherited backlog was not sealed within 3s
  test result: FAILED. 2 passed; 1 failed
#145's own test is red on its unfixed tree too: an open restarts the
backlog's age (3600 s in the test) and nothing else fires, so it panics
"the inherited backlog was not sealed within 5 s".
Green: cargo test --lib shard::seal:: dst_tests::idle_release::
dst_tests::billing_maintenance:: 126 passed.

Ledgers: none changed (lifecycle tests are not DST-inventoried; pins and
the scenario map check clean). Formal: TLA-016 digests seal.rs, so its
receipt is re-recorded in the next commit. Edge change: none on the wire;
the operator surfaces (maintenance view, fleet lag) see an inherited
backlog seal at once (#145's change; recorded with it if the owner wants a
record).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sorenbs added a commit that referenced this pull request Nov 1, 2026
@sorenbs @claude
A scale-validation guide tells an agent on Prisma Compute how to test…
ad53664 
… streams and queues at 1-2 shards per instance and the 100 ms WAL gap, what each costs at Class A/B -50%, and what it earns at Cloudflare's prices

Why: an agent validating at scale on Compute ran the bench with FLEET_MAX=64 (64 shards on one
instance) and WAL_FLUSH_GAP_MS=25, which multiplies WAL PUT cost per instance. The owner's posture
is 1-2 shards per instance, the binary's 100 ms gap, one WAL PUT in flight per shard.

docs/SCALE-VALIDATION.md: the deploy posture (exact env, INITIAL_SHARDS = N or 2N on a fresh
PATH_PREFIX, FLEET_MIN = FLEET_MAX, the settings that void a cost result, how to verify posture from
/v1/debug/{load,timings,store}, /operator/data.json and the boot log); forwarding as PR #143-#145
describe it (queues and live:wait not forwarded); streams tests ST-1..ST-4 (cell anchor PACK-T-1000
and PACK-S-MUX-0200, fleet linearity at 1/8/16 instances, M and L, idle floor) with void and
fail-early rules; queue 1x/10x/100x and hot-queue runs; the cost model (WAL PUTs per busy shard,
$48.60 a shard-month at 10/s and $1.875/M; Class A and B per shard as measured); measured cost against
K2 and Cloudflare Queues revenue; the owner's access-fee / marketing-expense ruling for T and S;
proposed acceptance bars; the report template; open items for the owner. Every figure cites a repo
path or is marked as program notes. docs/README.md: one map entry.

Red/green: none (documentation only; nothing built or run).
Ledgers: none. Formal receipts: none stale. Edge change: none.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sorenbs sorenbs mentioned this pull request Nov 1, 2026
Closed
@sorenbs

sorenbs commented Nov 1, 2026

Copy link
Copy Markdown
Member Author

Landed on slate as part of the integration stack ending at 5660231 (gated locally; CI round 5 on PR #143's branch passed every mutant shard). #143's forwarding is edge #235, ratified by the owner on 2026-11-01. #144 was reworked as its own task beside fleet::start (edge #236). #145 carries an added retry after a failed inherited seal. The commits are on slate; closing without a GitHub merge, per the repository's slate-direct workflow.

🤖 Generated with Claude Code

@sorenbs sorenbs closed this Nov 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@sorenbs