Skip to content

feat(omnigraph): pin 0.14.0, storage format 16 and HTTP contract 0.14 - #573

Merged
blarghmatey merged 8 commits into
mainfrom
feat/omnigraph-0.14-pin
Nov 4, 2026
Merged

blarghmatey merged 8 commits into
mainfrom
feat/omnigraph-0.14-pin

Conversation

@blarghmatey

@blarghmatey blarghmatey commented Oct 29, 2026 •

Copy link
Copy Markdown
Member

What are the relevant tickets?

witan task tk-bump-the-omnigraph-pin-to-0-14-contract-0-14-sto-e7e5c7, the pin-bump step of the omnigraph 0.14 upgrade (spec in docs/internals/design/omnigraph-0-14-upgrade-spec.md, rollout step 3). Follows #564, #568, #569 and #571.

Description (What does it do?)

The installer and both images pin omnigraph 0.13.0 (storage format 14, HTTP contract 0.13). This moves all three to the released v0.14.0 assets (tag commit 60bfd19). The digests were hashed locally and matched against each published .sha256 and the release API. omnigraph version on the binary reads internal-schema 16 (serves v14 to v16) and http-contract 0.14, so _OMNIGRAPH_INTERNAL_SCHEMA and both image labels move 14 to 16 (the release notes say 15) and HTTP_API_CONTRACT moves to 0.14. The data-tier image now selects the patched CLI from #569 (OMNIGRAPH_CLI_SOURCE=patched), which check-omnigraph-pins requires once the release tag equals the patch base. The witan image and witan setup keep the stock release binary.

Both images also carry a new label, edu.mit.ol.omnigraph.min-internal-schema (14). It's the low end of that served range, declared as _OMNIGRAPH_MIN_INTERNAL_SCHEMA and checked against the binary by check-omnigraph-format and across all three declarations by check-omnigraph-pins. ol-infrastructure's preview guard refuses any image whose internal-schema label differs from the stack's served format, so the 16 label alone fails every stack still on fmt14. mitodl/ol-infrastructure#6801 makes the guard accept a served format anywhere in the image's range.

Merging this is one-way for any graph once 0.14 publishes to it. 0.14 opens a format-14 graph in place, but the first write a 0.14 binary makes stamps that branch 16, and every 0.13 binary refuses it from then on. So this has to merge in the cutover window, and there is no rollback short of restoring the pre-cutover backup.

Merging also auto-deploys CI's pulumi-omnigraph and pulumi-witan in no fixed order, and publishes witan-core 0.50.0 to PyPI. A 0.13 server and a pinned 0.14 client refuse each other's contract header, but #580 (stacked on this branch) makes witan-core negotiate the contract per server, 0.14 or 0.13, so the two stacks can deploy in either order and the merge doesn't need witan's CI deploy paused. The ol-infrastructure council probe pins 0.13 until mitodl/ol-infrastructure#6811, which makes it discover the contract the same way, is deployed. The published witan-council 0.46.0 (witan-core>=0.48,<1) and witan-code 0.27.1 (>=0.46,<1) resolve 0.50.0 on a fresh install or upgrade. That doesn't put the 0.14 contract on any deployed data tier: omnigraph-server is ClusterIP-only with no HTTPRoute (ol-infrastructure applications/witan/ci_indexer.py), so users reach it only through the deployed witan MCP tier, whose image builds from this workspace. What it does change is local: witan setup installs the 0.14 binary, which stamps a developer's local stores 16 on their first write.

Two client changes ride along, because the 0.14 CLI breaks the existing failure handling:

  • 0.14 prints the resolved target on stderr for every call (target <store> (inline): direct graph …), so stderr is never empty. _cli_failure classified stderr whenever it had text, so every read's JSON error object on stdout (a recovery barrier, a conflict, an unreachable server) was classified from the target line alone and reported as fatal. This is what failed test_apply_schema_upgrades_an_old_store. The error object now wins whenever there is one.
  • _classify_cli_error reads 0.14's exits (command_outcome.rs exit_for at v0.14.0). 2 (partial or unknown effects) is never retried, and a write raises WriteIndeterminate; it is checked before the retryable markers so a dropped connection can't turn it into a retry. 64 (a refused command line) is fatal. 75 now covers any refusal with effects: none, action: retry, so one that carries a 503 gets the unavailable-server budget rather than the admission cap's.
  • The target lines 0.14 prints before every command (target <name> (<provenance>): ..., target write scope: branch=<name>) are kept in the message but left out of the classified text. They carry graph ids, branches and paths, and 0.13's --quiet that suppressed them is gone.

The 0.13-worded markers needed no change: 0.14 prints the same sentences with 0.14 in place of 0.13, checked on the binary and in graph_http.rs. witan-core is bumped to 0.50.0 here, and both servers' witan-core floors move to >=0.50 for the new contract.

A write whose HTTP error carries a 0.14 disposition with effects partial or unknown is now OUTCOME_UNKNOWN and raises WriteIndeterminate instead of being retried. 0.14 answers a load whose fork published before its rows were refused with a 409 that says partial, which the 409 rule would otherwise have replayed. A FATAL response is left alone: 0.14 reports unknown for every owned write that isn't a typed refusal, a plain 400 included, and group commit replays a FATAL composed group one write at a time. Whether that replay is safe stays with tk-classify-omnigraph-failures-from-the-0-14-dispos-2ae3ca.

Test changes for 0.14:

  • test_binary_contract.py gets a format-16 row, measured on 0.14.0 (every surface matches format 14), and addresses upgrade with --target as witan does.
  • testsupport/omnigraph_server.py applies and force-unlocks with 0.14's --target, so the served tests run against a 0.14 server with the 0.14 contract.
  • The bridge-recreate test used the drop of an indexed property, which 0.14 now does in place. It uses a String? to String change instead, which both 0.13.0 and 0.14.0 refuse in place (OG-MF-106 on 0.14.0; the test passes against both binaries).
  • The s3 export proxy test pins a pre-0.14 CLI, since stock 0.14 opens no s3:// store and witan-core already tests that refusal.

How can this be tested?

Run against the 0.14.0 binaries (a scratch HOME whose ~/.local/bin holds the release's omnigraph and omnigraph-server, TMPDIR on disk):

  • just test-all: witan-core 888 passed, 7 skipped; witan-council 1801 passed, 4 skipped, 1 failed; witan-code 711 passed, 1 failed; agent-config-kit 564 passed; ol-agent-kit 1 passed. The two failures are harness-only: test_pre_upgrade_candidates_exclude_the_current_binary (two omnigraph binaries on PATH under the scratch HOME) and test_rebuild_refuses_a_file_path_with_real_git (a long TMPDIR wraps the asserted phrase). Both pass in the default environment. The served group-commit and branch-view tests ran against a real 0.14 server.
  • The bridge tests in witan-code/tests/test_store.py also pass against the 0.13.0 binary.
  • just check-omnigraph-pins, just check-omnigraph-format, just check-versions, just docs-check and uv run prek run --all-files pass.
  • just check-core-floor passes against the bumped workspace witan-core.
  • For the min-format label: just check-omnigraph-format --binary <0.14.0> reports "opens 14 to 16 in place, as declared", and test_omnigraph_install.py covers the 0.14 and 0.11 version lines. After the label commit, witan-core's suite was re-run with the 0.13.0 binary on PATH: 880 passed, plus one expected failure (test_the_binary_on_path_is_the_declared_format, which needs 0.14). check-omnigraph-pins, check-versions, docs-check and prek on the changed files pass too.

Not verified here: the Concourse image builds (the patched CLI stage was qualified by omnigraph-cli-patch.yml in #569), and anything against a deployed data tier. The cutover itself is tk-cut-over-ci-qa-then-production-to-omnigraph-0-14-be7663.

Checklist:

https://claude.ai/code/session_01FGZebFvLxbZTiKey4rCL6B

Copilot AI balanced review requested due to automatic review settings October 29, 2026 17:07

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Writes with unknown effects can be replayed, the core-version mismatch leaves dependency checks failing, and paths containing spaces can cause incorrect retries.

3 open issues
What changed in this PR

Upgrades Witan’s Omnigraph integration to 0.14.0, aligning the installer, container images and HTTP client with storage format 16 and contract 0.14. This follows the earlier CLI-addressing, patched-S3 and remote-branch work and requires a coordinated data-tier-first cutover.

Changes:

  • Updates binary pins, checksums and image labels; enables the patched data-tier CLI.
  • Adapts CLI error classification to 0.14 output and exit codes.
  • Raises both servers’ core dependency floors and updates compatibility tests and changelog fragments.
File Description
testsupport/​omnigraph_server.py Uses version-aware cluster targeting in served tests.
packages/​witan-core/​witan_core/​omnigraph.py Handles 0.14 error output, target reports and exit codes.
packages/​witan-core/​witan_core/​omnigraph_install.py Pins 0.14.0 assets and declares storage format 16.
packages/​witan-core/​witan_core/​omnigraph_http.py Selects contract 0.14 and adds an unknown-outcome kind.
packages/​witan-core/​tests/​test_omnigraph.py Covers 0.14 CLI failure classification.
packages/​witan-core/​tests/​test_omnigraph_http.py Updates HTTP contract expectations.
packages/​witan-core/​tests/​test_binary_contract.py Adds format-16 expectations and target-aware upgrade calls.
packages/​witan-core/​changelog.d/​20261029_125338_blarghmatey_omnigraph_0_14_pin.md Records core compatibility and error-handling changes.
mcp/​servers/​witan/​tests/​test_remote_proxy.py Explicitly selects the legacy dialect for S3 export coverage.
mcp/​servers/​witan/​tests/​test_group_commit_served.py Uses shared HTTP contract constants.
mcp/​servers/​witan/​pyproject.toml Raises the core dependency floor to 0.50.
mcp/​servers/​witan/​changelog.d/​20261029_125338_blarghmatey_omnigraph_0_14_pin.md Documents the dependency and image updates.
mcp/​servers/​witan-code/​tests/​test_store.py Uses a schema change that still requires bridge recreation.
mcp/​servers/​witan-code/​pyproject.toml Raises the code server’s core dependency floor to 0.50.
mcp/​servers/​witan-code/​changelog.d/​20261029_125338_blarghmatey_omnigraph_0_14_pin.md Documents the new compatibility floor.
docker/​witan.Dockerfile Updates CLI pins, checksums and storage-format label.
docker/​omnigraph-server.Dockerfile Updates server pins and selects the patched CLI.
docker/​omnigraph_server_entrypoint.py Updates documentation for the default 0.14 image.
.github/​workflows/​omnigraph-cli-patch.yml Updates comments describing patched-image builds.

Review effort: Balanced

Comment thread mcp/servers/witan/pyproject.toml Resolved
The installer and both images pinned omnigraph 0.13.0 (format 14, HTTP
contract 0.13). This moves all three to the released v0.14.0 assets. The
binary reports "internal-schema 16 (serves v14 to v16)" and http-contract
0.14, so `_OMNIGRAPH_INTERNAL_SCHEMA` moves to 16 (the release notes say
15) and `HTTP_API_CONTRACT` to 0.14. The data-tier image selects the
patched CLI, which `check-omnigraph-pins` requires once the release tag
equals the patch base. Both servers' witan-core floors move to 0.50.

Two client changes the 0.14 CLI needs:

- 0.14 prints the resolved target on stderr for every call, so stderr is
  never empty, and `_cli_failure` classified stderr whenever it had text.
  Every read's error object on stdout (recovery barrier, conflict,
  unreachable server) was then classified from the target line alone and
  reported as fatal. The error object now wins whenever there is one.
- `_classify_cli_error` reads the new exits: 2 (partial or unknown
  effects) is never retried and a write raises WriteIndeterminate, 64 (a
  refused command line) is fatal, and 75 from a 503 gets the
  unavailable-server budget rather than the admission cap's.

Tests: a format-16 row in the binary contract table (measured on 0.14.0,
same as 14's), `upgrade` addressed by `--target`, the served-server
helper applies and unlocks with 0.14's `--target`, the bridge-recreate
test uses a type change 0.14 still refuses in place (it now drops an
indexed property in place), and the s3 export test pins a pre-0.14 CLI.

The first write a 0.14 binary makes to a format-14 graph stamps it 16,
and 0.13 binaries then refuse it. This merges only in the cutover window,
with the data tier moving before the witan clients.

Claude-Session: https://claude.ai/code/session_01FGZebFvLxbZTiKey4rCL6B
…fication

0.14 prints `target <name> (<provenance>): ...` and `target write scope:
branch=<name>` on stderr before every command, and 0.13's `--quiet`
that suppressed them is gone. Those lines carry the graph id, branch and
paths, so a marker without spaces (`recovery_required`,
`too_many_requests`) could match a name and misroute an ordinary
failure. The message keeps them; only the classified text drops them.

Claude-Session: https://claude.ai/code/session_01FGZebFvLxbZTiKey4rCL6B
The witan-policy CI job installs the pinned omnigraph, so with the pin at
0.14.0 policy/check.sh failed at its first cluster command with
"target_required". 0.14 takes the cluster root as --target on cluster apply
(--config is only the source) and replaced --cluster with --target on policy
validate and policy test.

Claude-Session: https://claude.ai/code/session_01FGZebFvLxbZTiKey4rCL6B
Both servers' floors move to witan-core>=0.50 in this PR, so check-core-floor needs the workspace at 0.50.0. Collects the 0.14 pin fragment and #572's StoreQuarantined removal.

Claude-Session: https://claude.ai/code/session_01FGZebFvLxbZTiKey4rCL6B
@blarghmatey
blarghmatey force-pushed the feat/omnigraph-0.14-pin branch from d95aa41 to 47a9bdf Compare October 29, 2026 19:24
…ects landed

0.14 answers a /load whose fork published before its rows were refused with
HTTP 409 and disposition effects "partial". The 409 rule classified that
RETRYABLE, so the retry loop could replay a write that had already changed
the graph. A retried kind (409/429/503) on a non-idempotent request now
becomes OUTCOME_UNKNOWN when the disposition says partial or unknown, and
the write raises WriteIndeterminate. FATAL responses are left alone,
because 0.14 reports "unknown" for every owned write that isn't a typed
refusal (a plain 400 included) and they aren't retried anyway.

The 0.14 target-report regex matched the label as \S+, so an inline path
with a space left the line in the classified text, where a marker such as
too_many_requests could turn a precondition failure into a retried
admission cap. It now matches up to the provenance delimiter.

Claude-Session: https://claude.ai/code/session_01FGZebFvLxbZTiKey4rCL6B

@feoh feoh left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code looks right. I checked the exit-code mapping against upstream exit_for at v0.14.0, and the digests against the release API. Two merge-time problems are inline. Minor items:

  • The body is out of date: the witan-core bump is already done (47a9bdf), and f391196 fixes the deferred "HTTP still retries every 409" low, so tk-classify-omnigraph-failures-from-the-0-14-dispos-2ae3ca may be closable.
  • Unchanged code, but it bites now: _download_omnigraph replaces any non-pinned binary, so an older witan's setup puts 0.13 back over 0.14. That build then refuses local stores already stamped 16. Recoverable from omnigraph-0.14.0, but there's no warning.

Comment thread packages/witan-core/pyproject.toml Resolved
Comment thread docker/omnigraph-server.Dockerfile Resolved
ol-infrastructure's preview refuses an image whose internal-schema label
differs from the stack's served format, so the 16 label fails every stack
still on fmt14 even though 0.14 opens format 14 in place. This declares
the low end of `omnigraph version`'s "serves v14 to v16" as
_OMNIGRAPH_MIN_INTERNAL_SCHEMA and stamps it on both images as
edu.mit.ol.omnigraph.min-internal-schema, so the guard can accept any
served root inside the range. check-omnigraph-format compares it against
the binary and check-omnigraph-pins keeps the three declarations and the
runtime-stage labels in step.

Raised in review: #573 (comment)

Claude-Session: https://claude.ai/code/session_01FGZebHdoMkhaQ1ewuGV2Ni
A version line whose "serves" clause isn't in the "vN to vM" shape read
as serving only the written format, so check-omnigraph-format advised
raising the min label to it, which would make ol-infrastructure refuse
every fmt14 root. It now raises instead. check-omnigraph-pins also
refuses a min above the written format, and the format check's remedy
says to update the min in the same change.

Claude-Session: https://claude.ai/code/session_01FGZebHdoMkhaQ1ewuGV2Ni
@blarghmatey

Copy link
Copy Markdown
Member Author

@feoh On the minor items: the body is updated. tk-classify-omnigraph-failures-from-the-0-14-dispos-2ae3ca stays open, because f391196 closes only the HTTP 409 replay. The task still covers classifying on the disposition instead of message text, plus 0.14 reporting effects: unknown on a plain 400 that group commit then replays one write at a time. The _download_omnigraph downgrade is filed as tk-witan-setup-silently-downgrades-a-newer-omnigrap-6f7ce5.

@blarghmatey
blarghmatey added this pull request to stack #581 November 4, 2026 14:54
@blarghmatey
blarghmatey merged commit ba19147 into main Nov 4, 2026
25 checks passed
@blarghmatey
blarghmatey deleted the feat/omnigraph-0.14-pin branch November 4, 2026 17:44
blarghmatey added a commit that referenced this pull request Nov 5, 2026
Both servers now pass `actor=` to OmnigraphClient, a keyword this change
adds to witan-core. With the floors at 0.50 an external install could
resolve 0.50.0, which imports cleanly and then raises TypeError on the
first deployed request or ingest. The floor checker accepts an
unpublished floor only when it equals the workspace version, so the bump
and the floors move together, as in #573.

0.51.0 is a minor bump because it also removes
reported_internal_schema() and reported_internal_schema_range() (#583).

Claude-Session: https://claude.ai/code/session_01FGZebE9nnZy6bBL3eprDdQ
blarghmatey added a commit that referenced this pull request Nov 5, 2026
* feat(omnigraph): assert the actor on per-actor requests

witan resolves a different bearer token per actor on every request
(ADR-0004), and a mapping that hands out the wrong token writes under
someone else's name. HTTP contract 0.14 lets the client say which actor it
expects (Omnigraph-If-Actor), and the server refuses a mismatch with 412
actor_mismatch before graph lookup or any effect.

OmnigraphClient takes the actor its token belongs to. The pooled transport
sends it on every request, and a write over the 0.14 CLI passes --as, which
the CLI sends as the same header. witan's per-actor client and witan-code's
ingest client set it. A 0.13.0 server ignores the header, so it goes out
whatever contract the transport speaks.

ACTOR_MISMATCH is its own kind because it shares 412 with a failed
precondition and means the opposite thing for a CAS caller: re-reading does
not fix a wrong token, so it is terminal and never a lost race, and group
commit fails the whole group rather than the fenced member.

Task: tk-send-omnigraph-if-actor-on-per-actor-requests-so-f4fc67

Claude-Session: https://claude.ai/code/session_01FGZebE9nnZy6bBL3eprDdQ

* fix(witan-code): pass actor to witan-core only when there is one

witan-code's OmnigraphClient forwarded actor= to the base class on every
construction, so against the published witan-core 0.50.0, which has no
such keyword, every witan-code client raised TypeError, local use
included. It now leaves the keyword off without an actor, so only a
deployment's per-actor ingest client needs the witan-core release that
adds it.

Claude-Session: https://claude.ai/code/session_01FGZebE9nnZy6bBL3eprDdQ

* test(witan): make the served actor test refuse an actual write

task_create runs a duplicate search first, and a 0.14 server checks the
Omnigraph-If-Actor assertion on reads too, so over HTTP the refusal came
from the search and the test passed without attempting the write. It now
stubs the search and records change_many, so both transports have to
refuse the mutation itself.

Claude-Session: https://claude.ai/code/session_01FGZebE9nnZy6bBL3eprDdQ

* version: witan-core 0.51.0, raise both server floors to it

Both servers now pass `actor=` to OmnigraphClient, a keyword this change
adds to witan-core. With the floors at 0.50 an external install could
resolve 0.50.0, which imports cleanly and then raises TypeError on the
first deployed request or ingest. The floor checker accepts an
unpublished floor only when it equals the workspace version, so the bump
and the floors move together, as in #573.

0.51.0 is a minor bump because it also removes
reported_internal_schema() and reported_internal_schema_range() (#583).

Claude-Session: https://claude.ai/code/session_01FGZebE9nnZy6bBL3eprDdQ
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants