Skip to content

An instance that owns more than the fair share of shards moves one, per move cooldown, to a member below the share - #144

Closed
sorenbs wants to merge 4 commits into
forward-to-ownerfrom
even-shards
Closed

sorenbs wants to merge 4 commits into
forward-to-ownerfrom
even-shards

Conversation

@sorenbs

@sorenbs sorenbs commented Oct 31, 2026 •

Copy link
Copy Markdown
Member

Stacked on #143. A fleet member that owns more than its fair share of shards moves one, per move cooldown, to a member below the share, until every member is within it.

Why

The ring's rendezvous hash uses FNV-1a over "<shard> streams-<n>". Member names that differ in one digit score in step, so the spread is uneven:

  • 64 shards over 4 members: 17 / 11 / 8 / 28 measured, 17 / 10 / 7 / 30 from the hash.
  • 64 shards over 8 members: one member still held 28 measured, 33 from the hash.

The rebalancer moved shards only for lag, and a healthy fleet has none, so nothing evened out: one member kept 28 shards for 8 minutes. A platform that balances pinned requests evenly is then capped by that member, which carried about 44% of the load.

What changes

src/fleet/count_balance.rs, called from the fleet tick when this instance is not lagging and its move cooldown has passed.

  • When it moves: an instance that owns more than ceil(shards / members) writes the same override, retirement and eager handoff that a lag move uses.
  • What it moves: one of its own shards with no override, preferring one whose rendezvous home is the target, so a count move never undoes a lag move.
  • Where it moves: to a member below the share that is not lagging.
  • No pile-ups: instances past the share take different targets at the same tick, each by its rank by name.
  • No flapping: return-home already admits a return only within the same share, so the two passes don't undo each other.

Measured

On Habanero's services bench, with REBALANCE_MOVE_COOLDOWN_SECS=5:

  • Spread: 32/32 within 50 s, 16 each within 85 s, and 8 each within about 4 minutes.
  • Throughput: 1, 2 and 4 instances then served 3,698, 7,774 and 15,262 pinned appends a second through the edge. That is 105% and 103% of linear.

Return-home and count placements (second commit)

Return-home dropped a count move's override 300 s after the move, which put the shard back on its crowded ring owner. The count pass then moved it again, without end. The Overrides document now keeps the prefixes a count move placed, in counted:

  • return-home leaves those prefixes alone;
  • a later count move may move them again;
  • a return-home drop removes the prefix from the set.

A document written before the field has none, so nothing changes for it. fleet::overrides holds the types now, since fleet.rs had reached its line ceiling. TLA-011 was re-recorded for it.

Records

  • Edge change #236.
  • #235's known window is corrected: a non-owner answers 404 for 4 to 8 s after a create even when the create went straight to the owner.
  • docs/SCALING.md.
  • The fleet_count_balance mutation owner, and re-measured sizes.
  • Exception-growth rows for fleet::start, which the owner approved on 2026-10-31.

Tests

  • Red: with the pass disabled, moves_bring_every_member_within_the_fair_share failed with 2 members: 33.
  • Green: the four fleet::count_balance tests pass, and so does dst_tests::fleet_controller. That suite caught a first version that moved an overridden shard straight back; the no-override rule fixes it.
  • Gate: scripts/gate.sh reached GATEDONE, and the release suite had 2,328 passed and 0 failed.

Breaking change

A healthy fleet now moves shards after a start or a change of membership. A pinned client sees its stream's owner change once for each moved shard, and follows Streams-Forwarded-To or Streams-Replay-To.

🤖 Generated with Claude Code

sorenbs and others added 4 commits October 31, 2026 03:04
… when the fleet has published the owner's URL, instead of answering 409 not_ring_owner

A platform that balances requests across a fleet's instances reaches a non-owner for most streams. Each such
request cost the client a 409 and a replay, and an unpinned retry lands at random again: on Habanero's
services bench, 64 unpinned appends saw 6,383 replays and 4,628 errors in 30 s, at a quarter of the pinned rate.
Asked for by the owner, 2026-10-31.

The change: a layer inside track_inflight, on data-plane routes only. When the handler answers 409 with
Streams-Replay-To and the trusted peer table holds the owner's URL, the request is sent once to the owner,
marked Streams-Forwarded, and the client gets the owner's answer, streamed, with Streams-Forwarded-To. A
request marked Streams-Forwarded is never forwarded again, so a forward is one hop. The body is not buffered
by the layer: the handler reads it through a deferred body, so an unread body streams to the owner, and a body
the handler read whole within 64 KiB is forwarded from the frames it read, which share its bytes. A larger
body, an unknown URL, or a refused connection leaves the 409. A failure after the request was sent answers
503 forward_failed with Retry-After: 1, because the owner may have applied it.
Canonical owner: http::forward::to_owner.

Red: dst::dst_tests::forward_to_owner, before the layer was wired:
  create fwd1: 409 {"error":{"code":"not_ring_owner","message":"shard 00 belongs to inst-b"}}
  (src/dst/tests/forward_to_owner.rs:57)
Green: forward_to_owner (8), seal_fence_relay (10), livefeed_ownership (8).
Gate: GATEDONE. The release suite: 2,324 passed, 0 failed.
Controls: a request already forwarded, no URL, an unreachable owner, and a body past 64 KiB each keep the 409.
seal_fence_relay's redirect test and livefeed_ownership's typed refusal now mark their requests as already
forwarded; a new test pins the forwarded close.
Known window: a non-owner whose create was forwarded caches the stream's absence for the registry's 5 s TTL
and answers 404 there until it passes (recorded in the edge change).

Ledgers: owners.json (forward_to_owner), test inventory, mutation owners (http_forward) and measured sizes
(queue_maintenance 124 -> 125 was already on slate), DST README, WIRE-MATRIX. Formal: none. Edge change #235.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…er move cooldown, to a member below the share

The ring's rendezvous hash (FNV-1a over "<shard> streams-<n>") scores members whose names differ in one
digit in step, so it spreads shards unevenly: 17/11/8/28 of 64 over 4 members on Habanero's services bench,
and one member still held 28 at 8 members after 8 minutes. The rebalancer moved shards only for lag, and a
healthy fleet has none. A platform that balances requests evenly is then capped by the busiest member.

The change: fleet::count_balance. An instance that is not lagging, past its move cooldown, and owns more than
ceil(shards / members) writes the same override a lag move writes for one of its shards (preferring one whose
rendezvous home is the target) to a member below the share that is not lagging. Instances past the share take
different targets at the same tick (each its own rank, by name), so concurrent moves do not overshoot one
member. Only a shard with no override moves, so a count move never undoes a lag move or another count move,
and return-home admits a return only within the same share.
Canonical owner: fleet::count_balance::pick.

Red: fleet::count_balance::tests::moves_bring_every_member_within_the_fair_share with the pass disabled:
  2 members: 33   (src/fleet/count_balance.rs:167)
Green: fleet::count_balance (4 tests). dst_tests::fleet_controller caught a first version moving an overridden
shard straight back; the no-override rule is its fix. On the bench, with REBALANCE_MOVE_COOLDOWN_SECS=5, the shards spread
to 32/32 in 50 s, 16 each in 85 s and 8 each in about 4 minutes; 1, 2 and 4 instances then served 3,698,
7,774 and 15,262 appends a second (105% and 103% of linear). A first version sent concurrent moves to the
same member and kept moving shards for minutes; the rank rule is its fix.
Also corrects #235's known window: a non-owner answers 404 for 4 to 8 s after a create at the owner too.

Gate: GATEDONE. The release suite: 2,328 passed, 0 failed.
Ledgers: mutation owners (fleet_count_balance) and measured sizes (fleet 202 -> 203), SCALING.md; exception
growth rows for fleet::start, approved by the owner on 2026-10-31.
Formal: none. Edge change #236.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ay move it again

Return-home drops an override once it is REBALANCE_RETURN_SECS old and the rendezvous home is within
the fair share. Five minutes after the count pass had spread 64 shards 4 to each of 16 members, it
began handing count placements back to their homes, and the count pass moved others to make up: on
Habanero's services bench a measured run began during those moves, and its creates met
`shard_moving` and two instances opening one shard's log.

`fleet/overrides.json` now lists the prefixes a count move placed in `counted` (absent in older
documents, which read as before). Return-home leaves them; the count pass may move them again, as
well as shards with no override, while a lag move's override still moves only by return-home. The
document's types moved to src/fleet/overrides.rs, which keeps fleet.rs within its line budget.
Canonical owner: fleet::count_balance::movable.

Green: fleet:: (103 tests), including count_balance::a_count_placement_may_move_again_and_a_lag_move_may_not
and overrides::a_document_without_counted_prefixes_reads_and_writes_as_before.
Ledgers: mutation owners (fleet_overrides) and measured sizes; exception-growth rows for fleet::start
refreshed, under the owner's delegation of 2026-10-31 ("Ask fable or decide yourself"). Formal: none.
Edge change #236 (amended).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sorenbs

sorenbs commented Nov 1, 2026

Copy link
Copy Markdown
Member Author

Landed on slate as part of the integration stack ending at 5660231 (gated locally; CI round 5 on PR #143's branch passed every mutant shard). #143's forwarding is edge #235, ratified by the owner on 2026-11-01. #144 was reworked as its own task beside fleet::start (edge #236). #145 carries an added retry after a failed inherited seal. The commits are on slate; closing without a GitHub merge, per the repository's slate-direct workflow.

🤖 Generated with Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant