[sorenbs]

A Streams instance that doesn't own a stream's shard now forwards a data-plane request once to the owner, when the fleet has published the owner's URL. It used to answer 409 not_ring_owner and leave the replay to the client.

Why

A platform that balances requests across a fleet's instances reaches a non-owner for most streams. Each such request cost a 409 and a replay, and an unpinned retry lands on a random instance again. On Habanero's services bench (a cell of 4 behind an edge, 141 ms away), 64 unpinned appends got 95 appends a second, against 343 pinned. They saw 6,383 replays and 4,628 errors in 30 s.

What changes

A layer inside track_inflight, on data-plane routes only (live_wait::data_plane), in src/http/forward.rs. Its trigger:

• the handler answers 409 with Streams-Replay-To;
• the trusted peer table holds the owner's URL;
• the request isn't marked Streams-Forwarded.

When all three hold:

• One hop. The request goes to the owner once, marked Streams-Forwarded: <instance marker>. A marked request is never forwarded again; its 409 is answered as before.
• The client gets the owner's answer, with its status, headers and body, streamed, plus Streams-Forwarded-To: <owner>.
• The body isn't buffered by the layer. The handler reads it through a deferred body. A body the handler never read streams to the owner, so admission and the shed before a body is read apply at the owner. A body the handler read whole within 64 KiB, as an extractor does, is forwarded from the frames it read, which share its bytes.
• The 409 stays when:
  • the owner's URL is unknown;
  • the owner refuses the connection;
  • the handler read more than 64 KiB of the body, or only part of it.
• A failure after the request was sent answers 503 forward_failed with Retry-After: 1. The owner may have applied the request, so the client must not replay it as new.
• Waiting reads and live feeds are forwarded like other requests. Reads at a non-owner were already served through the segment-read relay, and still are.

Known window: a non-owner whose create was forwarded caches the stream as absent for the registry's 5 s TTL. Until it passes, it answers 404 for that stream; a 404 names no owner, so it isn't forwarded.

Records

• Edge change #235 (risk medium) and a WIRE-MATRIX §0 entry.
• owners.json, test inventory, the http_forward mutation owner with re-measured sizes, and the DST README.
• queue_maintenance 124 → 125 in the sizes file was already on slate.
• Formal: none stale.

Tests

• Red: create fwd1: 409 {"error":{"code":"not_ring_owner","message":"shard 00 belongs to inst-b"}} before the layer was wired.
• New: dst::dst_tests::forward_to_owner, 8 tests: append, product append, create, live-feed connect, and the controls (already forwarded, no URL, unreachable owner, a body past 64 KiB). Also seal_fence_relay::a_non_owner_raw_close_takeover_installs_then_forwards.
• Existing tests:
  • seal_fence_relay's redirect test now marks its request as already forwarded, so it still pins the 409.
  • livefeed_ownership's typed refusal does the same.
• Gate: scripts/gate.sh GATEDONE. The release suite: 2,324 passed, 0 failed.

Breaking change

A client of a fleet that publishes peer URLs now gets, at a non-owner, the owner's answer where it used to get a 409. A router or SDK that followed Streams-Replay-To sees fewer 409s. It still has to handle them in the cases above.

🤖 Generated with Claude Code