Skip to content

Serialize Strong observation and per-name admission - #811

Draft
wolfy-j wants to merge 4 commits into
mainfrom
fix/naming-owner-admission-20260920
Draft

wolfy-j wants to merge 4 commits into
mainfrom
fix/naming-owner-admission-20260920

Conversation

@wolfy-j

@wolfy-j wolfy-j commented Sep 20, 2026

Copy link
Copy Markdown
Contributor

What changes

Strong naming uses an ordered KV-watch owner with a bounded worker pool. Pending attempts have stable IDs, and waiters, retry timers, exclusions, and terminal results remain bound to their attempt. Slow votes cannot block unrelated watch delivery. Each voter checks LOCAL/EVENTUAL claims and installs its exclusion under the shared per-name admission gate. ACK/NACK, promotion, and expiry use conditional KV transactions; snapshot, record, and vote-read failures close admission.

This revision also closes the omitted-member gap. Before a node opens LOCAL/EVENTUAL admission, it commits a fresh activation marker into a KV-backed naming-participant roster, waits for that marker locally, opens the watch, and coherently seeds pending and active claims. Strong pending creation reads that roster and conditions its transaction on the exact roster version. If enrollment wins first, the new participant is required to vote; if pending creation wins first, the joining participant sees the pending claim during seed before it can admit a weaker name. Roster changes that supersede a node's activation close its admission. Activation has a 10-second startup bound, including a stalled Raft submission, and fails closed on timeout.

Operational boundaries

  • This assumes one process exercises naming authority for a stable NodeID at a time. The activation marker detects replacement, but is not a fence against two overlapping same-ID processes.
  • Participant removal is intentionally not automatic. Gossip departure and Raft removal do not prove weaker bindings are gone. A crashed or failed-start participant remains required and can block new Strong claims until an explicit, safe retirement mechanism is implemented.
  • Mixed-version Strong creation is unsafe: older nodes do not enroll in the roster. Roll out this change as a coordinated cohort, with Strong creation stopped until every naming participant is on this version. Older leaders also lack terminal-result events, so mixed versions can produce uncertain caller outcomes.
  • This is based on merged Serialize LOCAL and EVENTUAL name admission #831 (shared admission coordinator) and Expose coherent local KV prefix scans for naming reconciliation #829 (coherent local snapshot scanning). It does not claim 100-node load validation.

Validation

  • Affected KV, boot, and real-cluster suites, including Strong promotion and leader-kill fail-closed coverage.
  • Race tests for KV naming, boot components, and real-cluster tests.
  • Regression tests for roster/pending races, pre-enrollment seed, stale activation snapshots, corruption, transient leader errors, cancellation, and a late old activation completing after retry.
  • Wider boot/topology tests, go vet, and git diff --check.

Astra reviewed the roster protocol and found no additional safety blocker under the stated NodeID assumption. Keep draft until CI and the fail-closed availability/rollout tradeoffs are accepted.

@wolfy-j
wolfy-j requested a review from skhaz September 20, 2026 21:33
@wolfy-j
wolfy-j force-pushed the fix/kv-observation-revisions-20260920 branch from 96dbc67 to ecc9a9c Compare September 21, 2026 22:23
Base automatically changed from fix/kv-observation-revisions-20260920 to main September 21, 2026 22:35
@wolfy-j

wolfy-j commented Sep 22, 2026

Copy link
Copy Markdown
Contributor Author

Re-reviewed against current main with Astra. Keeping this PR draft; the published head is not merge-ready yet.

Blocking integration work:

  1. Failed or malformed KV observations must close naming admission. The published head returns from Strong snapshot/decode failures without closing it and ignores malformed active/pending watch records. Preserve the current fail-closed reconciliation behavior in the ordered-owner model.
  2. One Strong attempt must never commit both ACK and NACK. Check both vote keys in the transaction and retain the TestStrongAckAndNackCannotBothCommit regression.
  3. Preserve the current cross-scope LookupOther(name, proposedPID) behavior, escaped vote-key grammar, reconnect cleanup, and attempt-bound timer protections. A semantic rebase is required; resolving text conflicts alone is insufficient.
  4. Port the current regression tests to the new ordered-owner and committed-terminal-event APIs without dropping their safety assertions. An isolated integration attempt still fails test compilation on obsolete waiter/terminal helpers and a malformed vote-route test fixture.

This is separate from #796 (inert participant inventory) and #799 (authority snapshot reader). Neither by itself activates participant enrollment or grants ongoing naming admission.

@wolfy-j

wolfy-j commented Sep 23, 2026

Copy link
Copy Markdown
Contributor Author

Integration update (main after #796/#799): I split the missing coherent local prefix scanner into #829. #811 still conflicts with main and remains draft. The rebase must preserve main’s escaped vote-key grammar, proposed-PID cross-scope checks, local-only EVENTUAL revocation, and authority/snapshot behavior. Before marking it ready, I will also require fail-closed observation errors and an atomic check that ACK and NACK cannot both be committed for one node/attempt. The initial automatic-merge trial does not compile and has not been pushed. Activation/retirement authority and 100-node evidence remain separate cluster-readiness work as already noted in this PR.

@wolfy-j

wolfy-j commented Sep 23, 2026

Copy link
Copy Markdown
Contributor Author

Prerequisite #829 is now merged on main (a060c74). The remaining #811 integration is the naming owner/admission change itself; this draft is not merge-ready yet.

@wolfy-j
wolfy-j force-pushed the fix/naming-owner-admission-20260920 branch from 3077813 to a2d23fb Compare September 23, 2026 13:35
@wolfy-j

wolfy-j commented Sep 23, 2026

Copy link
Copy Markdown
Contributor Author

Adversarial follow-up (def75cd): Strong voting now inspects every live EVENTUAL dot for the name, not just the visible winner. EVENTUAL lookup suppresses a conflicting result while this node holds a pending or active Strong exclusion; the raw CRDT state remains intact for convergence. Regressions cover hidden local and remote dots, delayed gossip after reservation, and a real KV Strong NACK. Affected race suites, 20-run targeted stress, and go vet pass locally. Participant retirement is still deliberately fail-closed and remains a separate design/rollout decision; a gossip timeout is not fencing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants