Skip to content

Fixes #35988: Add abstract, source, and deprecation fields to native ontology classes - #35997

Open
harshach wants to merge 16 commits into
harshach/ontology-studio/02-definition-statusfrom
harshach/ontology-studio/03-class-model
Open

harshach wants to merge 16 commits into
harshach/ontology-studio/02-definition-statusfrom
harshach/ontology-studio/03-class-model

Conversation

@harshach

@harshach harshach commented Oct 22, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #35988

Native ontology definitions record abstraction, definition source, deprecation and replacements. RDF import and export preserve those fields, and grouped working diffs remain revision-bound. With definition status in the parent, captured release summaries now include the deprecated count.

The published stack targets release 2.1.0. Each linked Shipping issue is In Progress with Product, P1 and Governance metadata.

main → PR1 #35990 → PR2 #35994 → PR3 #35997 → PR4 #35995. Two chains start at PR4:

This PR targets PR2 #35994. Its child is PR4 #35995. PR9, PR11, PR13, PR14 and PR16 are not published yet.

Current assessment for eb95757841f, recorded 2026-10-28.

Verdict. The 2026-10-25 source integration review passed at 33601808a794. The plan audit after it closed with no change. This head merges PR2's review-bot fix 20e50e5e265. PR3's patch-id against PR2 is 36506fafb14 before and after that merge, with the same edit lines in all 58 files, so the verdict carries to this head.

Test evidence at this head. The merge gate passed package and spotless, and 190 of 190 unit tests. docs/generated/api-reference.md regenerates unchanged, the Java impact map is complete, and the Playwright map is current.

CI. Every check at this head passes except maven-collate-ci. That job fails with HTTP 422 (No ref found for: harshach/ontology-studio/02-definition-status). The Collate build is pointed at this PR's base branch, and the Collate repository has no branch of that name. That is a limit of the CI setup for stacked PRs, not a fault in this PR's code. PR1, whose base is main, passes it.

Still owed. Docker is down on the build machine, so none of these has run. Run each IT on -Pmysql-elasticsearch and then -Ppostgres-opensearch, and confirm a Tests run: line.

  • The full merge-queue IT suite, which the impact planner selects because the stack changes ListFilter.java (area core). Run first IndependentOntology*IT, Ontology*ResourceIT, TaskResourceIT, TaskChangeEventIT, and WorkflowDefinitionResourceIT in the global-state lane.
  • The container-backed unit classes OntologyEntityDAOTest and OntologyReleaseComparisonStorageTest.
  • Changed-class coverage, which needs the ITs above.
  • The plan's live and performance lanes for this PR.

Not verified yet. This PR is verified only when its unit, live and performance boxes in the plan are all checked. The operator lands the stack.

Earlier description and verification records

These records preserve the earlier design notes, test results and attachments. Earlier statements about the current head or readiness apply to their recorded revisions. The integration assessment above supersedes those statements.

Describe your changes:

Fixes #35988

Native ontology classes could not say that they are abstract, where their definition comes from, or that they are deprecated and which definition replaces them. This PR adds abstract, definitionSource, deprecated and replacedBy to concepts, and deprecated and replacedBy to properties. It projects them as om:abstract, dcterms:source, owl:deprecated and dcterms:isReplacedBy and reads them back on import. Release and working diffs classify a member whose deprecated turned true as DEPRECATED. GET /api/v1/ontologies/{id}/working/diff?groupBy=concept serves rows grouped by concept (change type, parent labels, property count, member changes), paged by concept under the same revision binding (a stale cursor returns 412). A property with a single enumeration value is refused with An enumeration needs at least two allowed values. Adds a decision record for the new fields.

Also in this PR: a MySQL deadlock fix. OntologyReferenceIndex.replace no longer deletes the reference rows of a definition that does not exist yet. Deleting a missing key range takes a gap lock, and two concurrent creates then deadlocked on their inserts and returned 500 after the retries ran out.

Left out: the changeSummary deprecated count (PR2 is not in the base) and any UI.

Deviations from the plan:

  • Grouped diff keeps concept-less rows (ontology root, orphans), so the grouped view is a lossless partition of the plain diff.
  • A change set advances the working revision once per applied operation, not once. That is how OntologyWorkingState behaves on main, so the IT asserts it.

What could break: the enumeration rule applies only when a one-value list differs from the stored value, so stored definitions stay editable. replacedBy is validated as a same-kind, same-ontology reference, so deleting a replacement is refused.

Type of change:

  • New feature

Tests:

  • Unit: mvn test -pl openmetadata-service -Dtest='Ontology*Test,NativeOntology*Test,RdfOntologyContractTest,OwlProfileGuardTest': 595 run, 0 failures. Line coverage from unit tests alone: projection 98%, importer 98%, validator 95%, OntologyConceptChanges 100%, comparison 69%, property repository 71%; the resource and mappers are covered by the ITs.
  • IT: IndependentOntologyClassModelIT and NativeOntologyMemoryIT (new cases: class change set with two parents, an equivalent class, a disjoint class and three properties; rollback when the last operation fails), plus the affected ontology ITs, on -Pmysql-elasticsearch and -Ppostgres-opensearch: 739 run, 0 failures, 96 skipped on each.
  • Before the deadlock fix, NativeOntologyMemoryIT failed on MySQL in 3 of 3 solo runs; after it, 3 of 3 pass.
  • Not run: the full java-affected-tests plan (about 400 ITs) and a perf probe.

Checklist:

  • I have read the CONTRIBUTING document.
  • My PR title is Fixes <issue-number>: <short explanation>
  • My PR is linked to a GitHub issue via Fixes #<issue-number> above.

RetriggerView in GreptileConfidence Score: 5/5

The reviewed changes appear safe to merge.

Summary

Adds abstraction, source, deprecation, and replacement fields to native ontology definitions.

  • Concepts and properties carry class details through API storage and RDF exports.
  • Newly deprecated definitions appear as a separate kind of release change.
  • Working diffs can show changed members together under their concept.
  • Data-property enumerations must contain at least two allowed values.
  • Concurrent ontology creation no longer deletes an empty reference range.

No actionable issue was established. No previous review threads were supplied. The full PR diff was reviewed because the previous and current commits have different merge bases with the target. Tests were not run during this review.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A[Definition writes or RDF import] --> B[Working definitions]
  B --> C[Reference and structure checks]
  B --> D[Release capture]
  D --> E[RDF export]
  B --> F[Compare with active release]
  F --> G[Flat member changes]
  F --> H[Concept groups]
  G --> I[Revision-bound paging]
  H --> I
Loading

Reviews (8) · Last reviewed commit: "Merge definition status into class model" · Reviewed by Greptile

Adds the fields to the concept and property schemas, the create
requests, the release payload allowlist, and the RDF catalog vocabulary.
Adds the DEPRECATED change type, the REPLACED_BY reference role, and the
concept-grouped rows of the working diff. Regenerates the TypeScript
types.
@harshach
harshach requested a review from a team as a code owner October 22, 2026 15:18
@harshach harshach added safe to test Add this label to run secure Github workflows on PRs backend labels Oct 22, 2026
@github-actions

github-actions Bot commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

✅ PR checks passed

The linked issue has a description and all required Shipping project fields set. Thanks!

@github-actions

github-actions Bot commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Jest test Coverage

UI tests summary

Lines Statements Branches Functions
Coverage: 79%
79.75% (131262/164590) 66.26% (46308/69878) 69.21% (26273/37957)

Concepts and properties now carry deprecation and replacement, and
concepts carry abstract and a definition source:

- A replacement must name another definition of the same kind in the
  same ontology. It is indexed as a REPLACED_BY reference, so deleting
  a replacement target is refused like any other dangling reference.
- An enumeration needs at least two allowed values. One-value
  enumerations stored before this rule stay editable until their values
  change.
- Release projection writes om:abstract, dct:source, owl:deprecated,
  and dct:isReplacedBy, and native import reads them back.
- Release and working diffs classify a member whose deprecated flag
  turned true as DEPRECATED.
- GET /ontologies/{id}/working/diff?groupBy=concept returns the working
  changes grouped by the concept they define, with parent labels and a
  property count, paged by group under its own revision-bound cursor.
…sion

IndependentOntologyClassModelIT covers the field round trips, the
enumeration minimum and its legacy carve-out, replacedBy integrity, the
DEPRECATED diff row, and grouped diff paging with the 412 cursor rule.
NativeOntologyMemoryIT adds a class-create change set and its rollback.
Existing ITs that stored one-value enumerations now use two values.
Adds the fields to the concept and property schemas, the create
requests, the release payload allowlist, and the RDF catalog vocabulary.
Adds the DEPRECATED change type, the REPLACED_BY reference role, and the
concept-grouped rows of the working diff. Regenerates the TypeScript
types.
Concepts and properties now carry deprecation and replacement, and
concepts carry abstract and a definition source:

- A replacement must name another definition of the same kind in the
  same ontology. It is indexed as a REPLACED_BY reference, so deleting
  a replacement target is refused like any other dangling reference.
- An enumeration needs at least two allowed values. One-value
  enumerations stored before this rule stay editable until their values
  change.
- Release projection writes om:abstract, dct:source, owl:deprecated,
  and dct:isReplacedBy, and native import reads them back.
- Release and working diffs classify a member whose deprecated flag
  turned true as DEPRECATED.
- GET /ontologies/{id}/working/diff?groupBy=concept returns the working
  changes grouped by the concept they define, with parent labels and a
  property count, paged by group under its own revision-bound cursor.
…sion

IndependentOntologyClassModelIT covers the field round trips, the
enumeration minimum and its legacy carve-out, replacedBy integrity, the
DEPRECATED diff row, and grouped diff paging with the 412 cursor rule.
NativeOntologyMemoryIT adds a class-create change set and its rollback.
Existing ITs that stored one-value enumerations now use two values.
@harshach
harshach requested a review from a team as a code owner October 25, 2026 21:46
@harshach
harshach requested a review from stout-yeoman October 25, 2026 21:46
@harshach
harshach changed the base branch from main to harshach/ontology-studio/02-definition-status October 25, 2026 21:49
@greptile-apps

This comment has been minimized.

@gitar-bot

gitar-bot Bot commented Oct 27, 2026 •

Copy link
Copy Markdown

Reviewing your code

Code Review ✅ Approved 3 closed / 3 findings

🟡 Medium risk · Adding ontology metadata, RDF round-tripping, and replacement validation changes definition writes and imports; errors could omit metadata or reject valid definitions.

Adds abstract, definition source, and deprecation/replacement fields to native ontology classes and properties, projected through RDF import and export and reflected in grouped working diffs. Resolved the missing DEPRECATED change type emission, RDF export dropping the new semantic fields, and the missing decision record for new cross-component contracts. No open issues found.

✅ 3 closed
✅ Bug: DEPRECATED change type is declared but the comparison never emits it

📄 openmetadata-spec/src/main/resources/json/schema/api/data/ontologyReleaseDiff.json:15
The schema and the generated TypeScript now include DEPRECATED, and the description says such members are classified DEPRECATED. But OntologyReleaseComparison.changeType only ever returns ADDED, REMOVED or CHANGED. Because deprecated is now one of the semantic fields, setting it to true makes the digest differ, so the member is reported as CHANGED. Clients written against the new enum will never see DEPRECATED. The grouped-diff list in the description's "follow in the next pushes" does not cover this. The fix is to pass both snapshots to changeType and return DEPRECATED when before.deprecated is false and after.deprecated is true.

✅ Bug: New semantic fields are sealed into releases but dropped from RDF export

📄 openmetadata-service/src/main/java/org/openmetadata/service/ontology/OntologyReleaseContent.java:33-47 📄 openmetadata-spec/src/main/resources/rdf/ontology/openmetadata.ttl:4169-4175
abstract, definitionSource, deprecated and replacedBy are now part of the sealed release member content. Reviewers approve them, and they affect the digest. However, nothing under service/rdf/ (OntologyReleaseProjection, the concept and property writers, NativeOntologyRdfImporter) reads these fields. An activated release therefore puts no deprecation or abstract marker into Fuseki, and Turtle, RDF/XML and JSON-LD exports drop them without any error. The Accepted release-sealing ADR says "Unsupported members fail export rather than disappearing." The fix is to land the projection in this PR, using owl:deprecated, dct:isReplacedBy, om:abstract and om:definitionSource. RelationshipTypeRdfWriter already uses DCTerms.isReplacedBy. The alternative is to keep these fields out of semanticFields until the projection supports them.

✅ Quality: New cross-component contracts and limits have no decision record

📄 openmetadata-spec/src/main/resources/json/schema/api/data/ontologyReleaseDiff.json:15 📄 openmetadata-spec/src/main/resources/json/schema/api/data/ontologyWorkingDiff.json:10 📄 openmetadata-spec/src/main/resources/json/schema/api/data/ontologyWorkingDiff.json:63-69 📄 openmetadata-spec/src/main/resources/json/schema/entity/data/ontologyProperty.json:105 📄 openmetadata-spec/src/main/resources/json/schema/entity/data/ontologyProperty.json:112-115 📄 openmetadata-spec/src/main/resources/json/schema/type/ontologyReference.json:32 📄 openmetadata-service/src/main/java/org/openmetadata/service/ontology/OntologyReleaseContent.java:33-47
This PR adds contracts that the UI, Python and Java all have to agree on, and none of them is recorded in docs/decisions/. No ontology ADR mentions deprecation, abstract classes, grouped diffs or the minimum size of an enumeration. The records would need to say these things:

  1. DEPRECATED change type. "A release or working-diff member whose deprecated turned true is classified DEPRECATED, not CHANGED." This comes from the description's "classifies a member whose deprecated turned true as DEPRECATED". The record should also say what happens when other fields change in the same step, and what happens when the flag goes back to false.
  2. Grouped working diff. "A property belongs to the group of its first domain in this ontology. A change that belongs to no concept forms its own group. When grouping is requested, data is empty, paging counts groups, and a page holds at most 1000 groups." The 1000 cap was a judgment call.
  3. Enumerations. "An enumeration must have at least two allowed values." This is the description's "refuses an enumeration with a single value".
  4. REPLACED_BY reference role. "replacedBy must name a member of the same ontology."
  5. Release content. "abstract, definitionSource, deprecated and replacedBy join the sealed ontology-release-v1 member content without a new encoding version." The Accepted record ontology-releases-seal-canonical-sql-members defines that content contract and warns against reinterpreting it under a revised contract, so this change needs to be recorded or given a new version.

The fix is to add a record, or amend the release-sealing ADR, so it states these rules.

Verification

Rules No rules evaluated

Functional validation Not enabled

Set up

Auto-approval Not enabled

Set up

For more verification details, comment gitar display:verbose.

Options

Display: compact → Showing compact verification results.

Comment with these commands to change the behavior for this request:

Compact
gitar display:verbose         

Was this helpful? React with 👍 / 👎 | Powered by Gitar — free for open source

@github-actions

Copy link
Copy Markdown
Contributor

❌ UI Checkstyle Failed

❌ Bundle Runtime Check

A route chunk loads but one of its chunks still exports undefined: a chunking change broke module initialisation. Run yarn build && yarn bundle:runtime locally.


Fix locally (fast - only checks files changed in this branch):

make ui-checkstyle-changed

@sonarqubecloud

Copy link
Copy Markdown

@sonarqubecloud

Copy link
Copy Markdown

This branch was successfully deployed

1 active deployment
test — eb957578 Deployed Oct 27, 2026 by harshach via RDF Playwright execution #40351
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend safe to test Add this label to run secure Github workflows on PRs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Native ontology classes need abstract, source, and deprecation fields

1 participant