Repository navigation
Conversation
… when the fleet has published the owner's URL, instead of answering 409 not_ring_owner
A platform that balances requests across a fleet's instances reaches a non-owner for most streams. Each such
request cost the client a 409 and a replay, and an unpinned retry lands at random again: on Habanero's
services bench, 64 unpinned appends saw 6,383 replays and 4,628 errors in 30 s, at a quarter of the pinned rate.
Asked for by the owner, 2026-10-31.
The change: a layer inside track_inflight, on data-plane routes only. When the handler answers 409 with
Streams-Replay-To and the trusted peer table holds the owner's URL, the request is sent once to the owner,
marked Streams-Forwarded, and the client gets the owner's answer, streamed, with Streams-Forwarded-To. A
request marked Streams-Forwarded is never forwarded again, so a forward is one hop. The body is not buffered
by the layer: the handler reads it through a deferred body, so an unread body streams to the owner, and a body
the handler read whole within 64 KiB is forwarded from the frames it read, which share its bytes. A larger
body, an unknown URL, or a refused connection leaves the 409. A failure after the request was sent answers
503 forward_failed with Retry-After: 1, because the owner may have applied it.
Canonical owner: http::forward::to_owner.
Red: dst::dst_tests::forward_to_owner, before the layer was wired:
create fwd1: 409 {"error":{"code":"not_ring_owner","message":"shard 00 belongs to inst-b"}}
(src/dst/tests/forward_to_owner.rs:57)
Green: forward_to_owner (8), seal_fence_relay (10), livefeed_ownership (8).
Gate: GATEDONE. The release suite: 2,324 passed, 0 failed.
Controls: a request already forwarded, no URL, an unreachable owner, and a body past 64 KiB each keep the 409.
seal_fence_relay's redirect test and livefeed_ownership's typed refusal now mark their requests as already
forwarded; a new test pins the forwarded close.
Known window: a non-owner whose create was forwarded caches the stream's absence for the registry's 5 s TTL
and answers 404 there until it passes (recorded in the edge change).
Ledgers: owners.json (forward_to_owner), test inventory, mutation owners (http_forward) and measured sizes
(queue_maintenance 124 -> 125 was already on slate), DST README, WIRE-MATRIX. Formal: none. Edge change #235.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…er move cooldown, to a member below the share The ring's rendezvous hash (FNV-1a over "<shard> streams-<n>") scores members whose names differ in one digit in step, so it spreads shards unevenly: 17/11/8/28 of 64 over 4 members on Habanero's services bench, and one member still held 28 at 8 members after 8 minutes. The rebalancer moved shards only for lag, and a healthy fleet has none. A platform that balances requests evenly is then capped by the busiest member. The change: fleet::count_balance. An instance that is not lagging, past its move cooldown, and owns more than ceil(shards / members) writes the same override a lag move writes for one of its shards (preferring one whose rendezvous home is the target) to a member below the share that is not lagging. Instances past the share take different targets at the same tick (each its own rank, by name), so concurrent moves do not overshoot one member. Only a shard with no override moves, so a count move never undoes a lag move or another count move, and return-home admits a return only within the same share. Canonical owner: fleet::count_balance::pick. Red: fleet::count_balance::tests::moves_bring_every_member_within_the_fair_share with the pass disabled: 2 members: 33 (src/fleet/count_balance.rs:167) Green: fleet::count_balance (4 tests). dst_tests::fleet_controller caught a first version moving an overridden shard straight back; the no-override rule is its fix. On the bench, with REBALANCE_MOVE_COOLDOWN_SECS=5, the shards spread to 32/32 in 50 s, 16 each in 85 s and 8 each in about 4 minutes; 1, 2 and 4 instances then served 3,698, 7,774 and 15,262 appends a second (105% and 103% of linear). A first version sent concurrent moves to the same member and kept moving shards for minutes; the rank rule is its fix. Also corrects #235's known window: a non-owner answers 404 for 4 to 8 s after a create at the owner too. Gate: GATEDONE. The release suite: 2,328 passed, 0 failed. Ledgers: mutation owners (fleet_count_balance) and measured sizes (fleet 202 -> 203), SCALING.md; exception growth rows for fleet::start, approved by the owner on 2026-10-31. Formal: none. Edge change #236. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ay move it again
Return-home drops an override once it is REBALANCE_RETURN_SECS old and the rendezvous home is within
the fair share. Five minutes after the count pass had spread 64 shards 4 to each of 16 members, it
began handing count placements back to their homes, and the count pass moved others to make up: on
Habanero's services bench a measured run began during those moves, and its creates met
`shard_moving` and two instances opening one shard's log.
`fleet/overrides.json` now lists the prefixes a count move placed in `counted` (absent in older
documents, which read as before). Return-home leaves them; the count pass may move them again, as
well as shards with no override, while a lag move's override still moves only by return-home. The
document's types moved to src/fleet/overrides.rs, which keeps fleet.rs within its line budget.
Canonical owner: fleet::count_balance::movable.
Green: fleet:: (103 tests), including count_balance::a_count_placement_may_move_again_and_a_lag_move_may_not
and overrides::a_document_without_counted_prefixes_reads_and_writes_as_before.
Ledgers: mutation owners (fleet_overrides) and measured sizes; exception-growth rows for fleet::start
refreshed, under the owner's delegation of 2026-10-31 ("Ask fable or decide yourself"). Formal: none.
Edge change #236 (amended).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sorenbs
force-pushed
the
forward-to-owner
branch
from
November 1, 2026 05:36
3259312 to
4679afa
Compare
sorenbs
force-pushed
the
forward-to-owner
branch
2 times, most recently
from
November 1, 2026 13:40
8469a2e to
e0a391c
Compare
Member
Author
|
Landed on 🤖 Generated with Claude Code |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #143. A fleet member that owns more than its fair share of shards moves one, per move cooldown, to a member below the share, until every member is within it.
Why
The ring's rendezvous hash uses FNV-1a over
"<shard> streams-<n>". Member names that differ in one digit score in step, so the spread is uneven:The rebalancer moved shards only for lag, and a healthy fleet has none, so nothing evened out: one member kept 28 shards for 8 minutes. A platform that balances pinned requests evenly is then capped by that member, which carried about 44% of the load.
What changes
src/fleet/count_balance.rs, called from the fleet tick when this instance is not lagging and its move cooldown has passed.ceil(shards / members)writes the same override, retirement and eager handoff that a lag move uses.Measured
On Habanero's services bench, with
REBALANCE_MOVE_COOLDOWN_SECS=5:Return-home and count placements (second commit)
Return-home dropped a count move's override 300 s after the move, which put the shard back on its crowded ring owner. The count pass then moved it again, without end. The
Overridesdocument now keeps the prefixes a count move placed, incounted:A document written before the field has none, so nothing changes for it.
fleet::overridesholds the types now, sincefleet.rshad reached its line ceiling. TLA-011 was re-recorded for it.Records
docs/SCALING.md.fleet_count_balancemutation owner, and re-measured sizes.fleet::start, which the owner approved on 2026-10-31.Tests
moves_bring_every_member_within_the_fair_sharefailed with2 members: 33.fleet::count_balancetests pass, and so doesdst_tests::fleet_controller. That suite caught a first version that moved an overridden shard straight back; the no-override rule fixes it.scripts/gate.shreached GATEDONE, and the release suite had 2,328 passed and 0 failed.Breaking change
A healthy fleet now moves shards after a start or a change of membership. A pinned client sees its stream's owner change once for each moved shard, and follows
Streams-Forwarded-ToorStreams-Replay-To.🤖 Generated with Claude Code