Repository navigation
fix(schema)!: refuse a body-level @unique on a list or vector property - #1306
Conversation
A body-level @unique(p) on a list or vector property was admitted, though the property-level shorthand refuses it and the write path's scalar key cannot enforce it, so every write that set the property failed. Schema admission now refuses list and vector members of a body-level @unique, as it refuses Blob members. A persisted contract that already holds one still parses, so existing graphs keep opening. Closes ModernRelay#1305
ragnorc
left a comment
There was a problem hiding this comment.
Review of 5c8a178
Why. A body-level @unique on a list or vector property passed schema admission. After that, every write that gave the property a value failed in canonical_scalar_key. The graph could store no value in that property, and the constraint could not be removed in place.
What. The Constraint::Unique arm of validate_type_constraints now refuses list and vector members, as it already refused Blob members. The refusal applies in admission mode only. A persisted contract still parses, so a graph that already holds such a constraint still opens.
How it fits. The admission rule now matches the write path exactly. Every type that the parser still accepts as a @unique member (String and enum, I32/I64/U32/U64, F32/F64, Bool, Date, DateTime as Date64) has a branch in canonical_scalar_key (crates/omnigraph/src/loader/mod.rs:4099). Every type it now refuses (list, FixedSizeList vector, Blob) has none. Interface properties are covered too: resolve_interfaces desugars their constraints into the node before validate_constraints runs.
Impact. This removes a schema shape that the engine cannot serve. It adds no state, API or invariant. The cost is one breaking admission refusal. It follows the Blob and #1071 precedents, and the note and the guide document the exit.
Outcome: ready to merge. No blocking findings. Three optional suggestions follow.
Blocking
None.
Optional
O1. One predicate for "this member can be a key or unique member". The parser states this rule three times: in the property-level annotation loop (parser.rs:937-962), in the @key arm (parser.rs:1164, list/vector/Blob checks below it) and in the @unique arm (parser.rs:1233). canonical_scalar_key states it a fourth time at runtime. This defect is that duplication drifting: the @unique arm fell behind the other two. A small fn scalar_key_member_refusal(&PropType) -> Option<&'static str> could be used by all three parser sites, with the admission-only split kept at the @unique call site. Then the next member-type rule would change in one place. To make drift against the write path loud, an engine unit test beside canonical_scalar_key_handles_every_supported_non_string_key_type could do this: for every ScalarType that the predicate accepts, build a one-row array of to_arrow() and assert that canonical_scalar_key returns Ok. Cost: one function and one test. It removes duplicated rules that can drift. It is not required for this fix.
O2. Pin the documented exit with a test. The guide now promises two things: dropping every property a historical @unique names removes the constraint in one apply, and a constraint that also names another member or @src/@dst cannot be removed in place. Only the PR description shows these, by hand. The lifecycle test already builds a historical graph with publish_schema_contract_row (crates/omnigraph/tests/integration/lifecycle.rs:27). Adding one apply_schema step there would pin the user-facing promise: drop tags, re-add it without the constraint, then write a value. I read ends_with_dropped_properties (catalog/schema_plan.rs:963), and it does what the guide says. The test would stop a later planner change from breaking the exit silently.
O3. Cluster wording in the note. validate_bundle_with_root parses every graph's schema in admission mode (crates/omnigraph-cluster/src/deployment/execution.rs:833). It runs on capture, preview, execution and reconcile. So after the upgrade, a deployment that contains one such graph refuses every cluster apply, even one that changes only another graph's queries or policies. It also refuses to reconcile a deployment captured before the upgrade. The guide's existing line ("Complete outstanding deployments ... before upgrading") covers the second case. The changelog note says only "before reapplying its source". You could say directly that any cluster apply of that deployment is refused until the source changes. Wording only.
Checks
- Regression proof (cheapest owner): the compiler parser tests and the engine lifecycle test. GQT cannot express this: a
.gqtschema section is a fixture that must parse at case load (omnigraph-gqt-core/src/lib.rs, theschemasection andrefuse_embed_schema), and the format has no schema-refusal step. I agree with the PR on this.- Before the fix: head tests with
crates/omnigraph-compiler/src/schema/parser.rsrestored from0b468487.cargo dev-nextest -E 'test(/issue_1305/)' --no-fail-fastgave 0 passed, 3 failed, each for the predicted reason.test_reject_body_unique_..._issue_1305panicked atexpect_err("node-list")because the schema was accepted.test_persisted_contract_..._issue_1305failedparse_schema(historical).is_err().lifecycle::open_accepts_historical_body_unique_but_init_rejects_it_issue_1305panicked withnew init must refuse: ... list property Document.tags. - After the fix, at
5c8a1782: the same command gave 3 passed. The temporary edit is reverted, and the worktree is clean.
- Before the fix: head tests with
- Focused suites at
5c8a1782(clean detached worktree, debug,RUST_MIN_STACK=16777216):cargo dev-build-testshad no warnings.cargo dev-nextestover all ofomnigraph-compilerplus enginelifecycle::,validators::andissue_1305gave 556 passed, 0 failed. Enginebinary(schema_apply)gave 50 passed, 0 failed. - CLI built at head:
initrefuses@unique(tags)on[String]?and refuses edge@unique(@src, tags)with the new messages.lint --schemarefuses the node case too. - Admission callers checked: the engine opens through
parse_persisted_schema_contract(db/schema_state.rs:326), so historical graphs still open.lint --targetuses the opened catalog. Clusterconfig.rsparses only desired source. The cluster bootstrap check runs only when no revision has been applied yet. - Docs:
check-docs.pyOK (473 files).check-agents-md.shOK.release_notes.py verify --version v0.15.0 --target HEADOK.check-release-note.pywith this title OK.git diff --checkclean. - Exact-head CI at
5c8a1782: 27 success, 4 skipped (the release-tag and vocabulary-guard jobs), 0 failed. - Lance: not involved. Lance has no whole-value uniqueness for
ListorFixedSizeListcolumns. Its label-list index answers per-element membership (array_has), not whole-list equality. Refusing at admission therefore matches the substrate; enforcing would need a new Omnigraph key encoding and semantics. - Other open PRs: none changes constraint validation or the migration planner. #1291 also touches
loader/mod.rsand merges cleanly with this head.
Limits
- I did not run the drop-and-re-add exit or the historical-graph reads end to end with the CLI. That needs a graph written by the base binary. I checked the planner rule by reading the code (see O2).
- I did not run the GQT corpus, DST or the canonical workspace graph locally. Exact-head CI covers them.
- No performance claims; the change does no storage I/O.
Address review O2 and O3 on ModernRelay#1306: a lifecycle test plants a historical @unique(tags) on a list, shows a write is refused, drops tags to remove the constraint, re-adds it and writes; a constraint that also names a kept property is refused in place. The breaking note and the schema guide now say that every cluster apply of a deployment holding such a graph is refused.
|
Reply to the review of O2: taken, in
O3: taken, in the same commit. The changelog note now says that every apply of source that keeps the constraint is refused, including any O1: not in this PR, kept as a follow-up. I agree that one predicate would stop the parser sites from drifting apart. But the shared function would also change the property-level and Local checks on this code, debug, |
…vector # Conflicts: # docs/releases/v0.15.0.md
ragnorc
left a comment
There was a problem hiding this comment.
Review of 4007578 (delta from 5c8a178)
Why. The earlier review at 5c8a178 found no blocking defects and gave three optional suggestions. This review covers only what changed since then.
What changed. There are four new commits:
40e9d435adds a lifecycle test that pins the documented exit (O2). It also rewords the breaking note and the schema guide for clusters (O3).852f7f28and40075781regenerate the v0.15.0 release snapshot.0dc55b42mergesmainatf0df4673(#1307, release workflow only).
git range-diff shows the four earlier commits unchanged, so the fix in parser.rs is byte-identical to the reviewed one.
Impact. The delta adds one test and corrects two sentences. It adds no API, state or invariant, and it removes a gap: the exit the guide promises is now checked in CI, not only shown by hand in the PR description.
Outcome: ready to merge. No blocking findings. O2 and O3 are fixed. O1 is still open and still optional. One small new wording suggestion follows.
Blocking
None.
Fixed since 5c8a178
- O2 (pin the exit).
historical_body_unique_list_ends_by_dropping_its_property_issue_1305(crates/omnigraph/tests/integration/lifecycle.rs:606) plants a historical@unique(tags)on a list through the contract row. It shows a write that setstagsis refused with thecanonical_scalar_keyerror, dropstags, adds it back without the constraint and writes the row. It then shows that@unique(label, tags)cannot be removed in place, and that the accepted schema does not move. That matchesends_with_dropped_properties(catalog/schema_plan.rs) and the guide text. - O3 (cluster wording). The note and the guide now say that every
cluster applyof such a deployment is refused. I checked this against the code:validate_bundle_with_rootparses each graph schema of the desired bundle withparse_schema(admission mode) atcrates/omnigraph-cluster/src/deployment/execution.rs:834. So an apply that changes only another graph, query or policy is refused while the source keeps the constraint. An apply whose source drops the named properties is accepted, so the in-place exit also works in a cluster. The otherparse_schemacallers outside the compiler are tests,lint --schemaand clusterconfig.rs(desired source only), so no path re-parses an applied schema in admission mode.
Optional
O1 (still open). One predicate for "this member can be a key or unique member". Unchanged from the earlier review. The parser still states the rule in three places, and the defect came from those copies drifting apart. It is not required for this fix.
O4 (new). "until the constraint is removed" (docs/user/schema/index.md:116). Strictly, the apply that removes the constraint is itself a cluster apply of that deployment, and it is accepted. "until the deployment's source no longer declares it" says when applies start to pass. Wording only.
O5 (new, minor). Shared setup in the lifecycle test. The new historical_graph helper (lifecycle.rs:642) repeats the setup in open_accepts_historical_body_unique_but_init_rejects_it_issue_1305 (init the base schema, resolve the persisted shape, publish the contract row, reopen). If a third historical-contract test appears, one module-level helper would keep the planting steps in one place. Not worth a change for two tests.
Checks
- Regression proof (cheapest owner: compiler parser tests and engine lifecycle tests; GQT cannot express an admission refusal, as the earlier review explained).
- Before the fix: tests at
40075781, withcrates/omnigraph-compiler/src/schema/parser.rsrestored frommainatf0df4673.cargo dev-nextest -E 'test(/issue_1305/)' --no-fail-fastgave 3 failed and 1 passed. The three fix tests failed for the predicted reasons: the node-list schema was accepted (parser_tests.rs:354),parse_schema(historical).is_err()failed (parser_tests.rs:400), andinitaccepted the list schema (lifecycle.rs:573). The new exit test passed. That is correct: it pins the planner's drop rule, which does not depend on the admission fix. - After the fix, at
40075781: all 4 passed. I reverted the temporary edit, and the worktree is clean.
- Before the fix: tests at
- Focused suite at
40075781(clean detached worktree, debug,RUST_MIN_STACK=16777216):cargo dev-nextest -E 'test(/issue_1305/) | test(/^lifecycle::/)'gave 28 passed, 0 failed. - Docs:
check-docs.pyOK (473 files).check-agents-md.shOK.release_notes.py verify --version v0.15.0 --target HEADOK (60bfd19d..40075781).check-release-note.pywith this title OK.git diff --check origin/main...HEADclean. - Merge of main: it brings only #1307 (
build-release-archives.yml, install docs and its note). It does not touch constraint validation, the planner or the lifecycle tests. - Exact-head CI at
40075781: 27 success, 4 skipped (the vocabulary guard and release-tag jobs), 0 failed. - Lance: not involved in the delta. The earlier point still holds. Lance's label-list index answers per-element
array_has/array_has_all/array_has_any, not whole-list equality, so refusing at admission matches the substrate. - Other open PRs: #1301 changed only by merging the same
maincommit. No open PR changes constraint validation or the migration planner.
Limits
- I did not run the CLI end to end against a graph written by the base binary. The new engine test now covers the drop-and-re-add exit and the in-place refusal at the engine API.
- I did not run a real cluster apply for the O3 wording. I checked it by reading
validate_bundle_with_rootand its callers. - I did not run the full workspace graph, GQT or DST locally. Exact-head CI covers them.
- No performance claims. The delta does no storage I/O.
# Conflicts: # docs/releases/v0.15.0.md
|
Merged
|
|
I merged
|
ragnorc
left a comment
There was a problem hiding this comment.
Review of 2a337f1 (delta from 4007578)
Why. The review of 4007578 found no blocking defects. Since then, the branch has merged main two times. This review covers only that delta, plus one documentation point found in the existing guide text.
What changed. There are four new commits:
c8ba40a6mergesmainat2e7921c4(#1301).c8130db9regenerates the v0.15.0 snapshot.81603bb0mergesmainatead05c40(#1310).2a337f1dregenerates the snapshot again.
The PR's own files (parser, tests, guide, skill, changelog note) are byte-identical to 4007578. The only PR-owned change is the generated snapshot. Against main, the snapshot diff is the new breaking note, its link definition and the provenance comment. The two merges bring no change to constraint validation, the schema planner or the lifecycle tests.
Impact. The code fix is unchanged and still correct. The delta adds no API, state or invariant.
Outcome: no blocking code defects. One optional documentation correction, with a test-backed reason, follows. There are also two release-order points to check before merge.
Blocking
None.
Before merge (release process, not code)
- R1. Release order. #1310 says
ead05c40is the commit that v0.15.0 will be tagged on. If the tag comes first, this PR must not editdocs/releases/v0.15.0.md.docs/dev/documentation.mdallows--replaceonly before the tag exists, and a published snapshot does not change. In that case, drop the snapshot hunk, and the note goes to the next release. Then re-read the breaking note's upgrade wording for that release. - R2. PR link after the squash. The snapshot's
linksmap has no entry forchangelog.d/unique-refuses-list-and-vector-members.breaking.md, because the squash commit does not exist yet. After the merge, the note will ship without its PR link. #1301 had the same gap, and #1310 fixed it. Plan the same follow-up before the tag.
Optional
O6 (new). One guide sentence disagrees with the planner (docs/user/schema/index.md:118-120). The guide says: "A constraint that also names another property or @src/@dst cannot be removed in place: rebuild the graph without it." The planner does not work that way. ends_with_dropped_properties (crates/omnigraph-compiler/src/catalog/schema_plan.rs:963) ends any non-key constraint when the plan drops every property it names. It refuses only when a named property stays or the constraint names a system field. The new test's own comment says this correctly ("names a property that stays", lifecycle.rs:610), and so does the changelog note ("Drop the properties it names in one change").
I proved it with a temporary step at the end of historical_body_unique_list_ends_by_dropping_its_property_issue_1305, at 2a337f1. On the historical @unique(label, tags) graph, apply_schema with only title succeeded. apply_schema(BASE_SCHEMA) then added both properties back, and the tags row loaded. cargo dev-nextest -E 'test(/issue_1305/) | test(/^lifecycle::/)' gave 28 passed. I removed the step afterwards.
So a user with @unique(label, tags) reads "rebuild" when one schema change can remove the constraint, at the cost of the label values too. A suggested wording: "A constraint that also names a property you keep, or @src/@dst, cannot be removed in place: drop all of its properties, or rebuild the graph without it." This also covers O4 if you rewrite the same paragraph.
O4 (still open). "until the constraint is removed" (docs/user/schema/index.md:116). Wording only, as before.
O1 (still open). One predicate for "this member can be a key or unique member". Unchanged and still optional.
Context (no action in this PR). Four other open PRs (#952, #1175, #1184, #1308) also regenerate docs/releases/v0.15.0.md. CI does not verify that snapshot on a PR, and its provenance comment records the generating commit. Thus each merge to main puts every other open copy in conflict, and each squash leaves a missing PR link (R2). This PR needed two main merges for this reason only. If the release-prep PR regenerated the snapshot one time, near the tag, the conflicts and the follow-up link PRs would stop. That decision belongs to the release process, not to this PR.
Checks
All at exact head 2a337f1, in a clean detached worktree (CARGO_TARGET_DIR shared, RUST_MIN_STACK=16777216). I restored every temporary edit, and the worktree was clean at the end.
- Regression proof (cheapest owner: compiler parser tests and engine lifecycle tests; GQT cannot express an admission refusal).
- Before the fix:
crates/omnigraph-compiler/src/schema/parser.rsrestored frommainatead05c40.cargo dev-nextest -E 'test(/issue_1305/)' --no-fail-fastgave 3 failed and 1 passed. The failures had the predicted reasons: the list schema was accepted (parser_tests.rs:354),parse_schema(historical).is_err()failed (parser_tests.rs:400), andinitaccepted the list schema (lifecycle.rs:573). The exit test passed, which is correct: it pins the planner's drop rule, not the admission fix. - After the fix:
cargo dev-nextest -E 'test(/issue_1305/) | test(/^lifecycle::/)' --no-fail-fastgave 28 passed, 0 failed.
- Before the fix:
- Compiler suite at c8130db (identical code to 2a337f1):
cargo dev-nextest -E 'package(omnigraph-compiler) | test(/issue_1305/) | test(/^lifecycle::/)'gave 519 passed, 0 failed. - Docs and format:
check-docs.pyOK (474 files).check-agents-md.shOK.release_notes.py verify --version v0.15.0 --target HEADOK (60bfd19d..2a337f1d).cargo fmt --all --checkclean.git diff --check origin/main...HEADclean. - Merge state: 2a337f1 merges cleanly into
mainatead05c40. - Exact-head CI at 2a337f1: 27 success, 4 skipped (the vocabulary guard and release-tag jobs), 0 failed.
- Other open PRs: none changes
schema/parser.rsorschema_plan.rs. #1101 editsdocs/user/schema/index.mdin another section (type namespaces), so it does not overlap. - Lance: not involved in the delta.
Limits
- I did not run the CLI end to end against a graph written by the base binary. The engine tests cover the admission refusal, the open path and both exits.
- I did not run a real cluster apply. The cluster wording was checked by code reading in the earlier review.
- I did not run the full workspace, GQT or DST locally. Exact-head CI covers them.
- No performance claims. The delta does no storage I/O.
Summary
Schema admission now refuses a body-level
@uniquewhose member is a list or vector property, as it already refuses the property-level shorthand and Blob members. Before this change,init,lint,schema planandcluster validateaccepted such a schema, and then every write that gave the property a value failed withscalar key: unsupported column type List(...) for @unique/@key enforcement.Closes #1305
Reproduction
On main
0b4684875ded4ecc5e829a2e7f7f208a06e65a04, with the issue's schema (tags: [String]?plus@unique(tags)):The
Vector(2)?variant fails the same way withFixedSizeList(...), and so does an edge with@unique(@src, tags). Verdict: reproduced, all three variants.The new parser tests fail on main:
test_reject_body_unique_containing_blob_list_or_vector_property_issue_1305(the schema is accepted) andtest_persisted_contract_parser_allows_only_historical_body_unique_issue_1305(parse_schemaaccepts the historical source).Root cause
crates/omnigraph-compiler/src/schema/parser.rschecks constraints in two places. The property-level annotation check refuses@uniqueon list and vector properties. The body-level check,validate_type_constraints, refuses list, vector and Blob members in itsConstraint::Keyarm, but itsConstraint::Uniquearm refused only Blob members. The write path'scanonical_scalar_key(crates/omnigraph/src/loader/mod.rs) builds unique keys from scalar columns only, so a list or vector member always fails there.Decisions
@keyarm refuses them, and the docs already say list and vector properties cannot be keys. List or vector uniqueness would need a new key encoding and semantics (whole-list equality? order? null elements?) that nothing asks for. Refusing makes both spellings of the same constraint agree.ConstraintValidationMode::Admission.parse_persisted_schema_contractstill accepts these members, so a graph whose accepted schema already declares one keeps opening for reads and export. Refusing in both modes would make such graphs fail to open.@src/@dstcannot be removed in place; such a graph needs a rebuild. That limit is the existingremoving constraints ... is not supported in schema migration v1rule, which this change does not alter.Alternative considered: also give
canonical_scalar_keya typed refusal for list and vector columns. Not needed once admission refuses them; the only graphs that reach it are the historical ones, where the existingfixdisposition is accurate.Before and after
0b468487initwith@unique(tags)on[String]?parse error: @unique is not supported on list property U.tagsinitwith@unique(v)onVector(2)?parse error: @unique is not supported on vector property U.v@unique(@src, tags)parse error: @unique is not supported on list property T.tagsAfter, with this branch's CLI:
For the edge graph, dropping
tagsplansremoving constraints from 'T' is not supported in schema migration v1, as the guide says.Tests
crates/omnigraph-compiler/src/schema/parser_tests.rs:test_reject_body_unique_containing_blob_list_or_vector_property_issue_1305extends the Blob refusal cases with node list, composite list, node vector, edge list with@srcand edge vector.test_persisted_contract_parser_allows_only_historical_body_unique_issue_1305extends the persisted-contract cases with list and vector.crates/omnigraph/tests/integration/lifecycle.rs:open_accepts_historical_body_unique_but_init_rejects_it_issue_1305now runs over Blob, list and vector:initrefuses, and a graph whose contract row holds the historical schema still opens.No
.gqtcase: a GQT case needs a schema that loads, so it cannot express an admission refusal.Local results (this box,
CARGO_TARGET_DIRshared, aliases from.cargo/config.toml):cargo test -p omnigraph-compiler --locked: 493 + 2 passed; the two new tests fail on main.cargo dev-nextestoveromnigraph-compiler,omnigraph-cluster, engineschema_apply,lifecycle::,validators::,legacy_columns::and CLIcli_schema_config::/lint: 819 run, 818 passed. The one failure,schema_apply::apply_schema_column_changes_keep_data_files_ids_and_index_coverage_issue_1014, is a stack overflow at this box's default stack size; it passes withRUST_MIN_STACK=16777216, which CI sets.cargo dev-clippyandcargo fmt --all --check: clean.python3 scripts/check-docs.pyandbash scripts/check-agents-md.sh: OK..gqtcase, benchmark or fixture declares a body@uniqueon a list or vector (scanned every@unique(in the tree), so no corpus schema changes meaning.RUST_MIN_STACK=16777216 cargo test -p omnigraph-gqt --locked: 559 passed, 0 failed (all 464 corpus cases).Docs
docs/user/schema/index.md: list and vector properties cannot be unique, and how a graph that already declares one gets out.skills/omnigraph/SKILL.mdandskills/omnigraph/references/schema.md: the reference said "list/vector declarations may parse but writes fail scalar-key validation"; it now says they are refused.changelog.d/unique-refuses-list-and-vector-members.breaking.md, and the regenerateddocs/releases/v0.15.0.mdsnapshot.Risks and what was not verified
schema applyon a standalone graph, not withcluster apply(it uses the same planner).Workflow run: run-wf-reproduce-modernrelay-omnigraph-issue-20261108T2142Z-0f5c4c
Response run: run-wf-respond-modernrelay-omnigraph-pr-20261109T0110Z-b8b23b
Response run: run-wf-respond-modernrelay-omnigraph-pr-20261109T0210Z-5b2fb3