Skip to content

Fixes #36715: Generate SQL from bindings and hand live data to a Collate extension - #36725

Merged
harshach merged 377 commits into
mainfrom
harshach/ontology-studio/14-live-data
Oct 30, 2026
Merged

harshach merged 377 commits into
mainfrom
harshach/ontology-studio/14-live-data

Conversation

@harshach

@harshach harshach commented Oct 30, 2026 •

Copy link
Copy Markdown
Collaborator

The stack targets release 2.1.0.

PR1 #35990, PR2 #35994, PR3 #35997, PR4 #35995, PR5 #36183, PR6 #36187, PR7 #36188, PR8 #36190, PR9 #36721, PR10 #35991, PR11 #36723, PR12 #35996, PR13 #36724 and PR15 #35989 are merged.

This PR targets main. PR16 is not published yet.

Current assessment for 4a082e4810e, recorded 2026-10-30.

Verdict. PASS. The sixth independent review passed 0f09ecd3792 with no blockers and no majors. The review of gitar-bot's round passed b548ede16d0 with no blockers and no majors. 5351e1709ba and e4405de5026 merge PR13's newer heads without conflict, and 4a082e4810e catches up with main after PR13 landed. PR14's own lines are unchanged.

What moved since the last assessment. gitar-bot's review raised one finding, and it is fixed. gitar-bot now approves with no open thread. The search-around picker was a core Box with a border, background and padding. It is now a core Card (size="sm", variant="muted") with Card.Content. In light mode it looks as before, a white panel with a gray-200 edge on the white rail. The section title is now a real h3 in place of a span with role="heading". It keeps not-prose tw:m-0 tw:text-primary, as other headings in the UI do, because core wraps a heading in .prose and has no primary text colour.

5351e1709ba merges PR13 at c2feefb2bcd. That brings PR13's direction per hop, its review fixes, PR11's review fixes and main with #36721 landed. PR14's stable patch-id is 9de987ca6d9 before and after, with every file the same. No Java, SQL, schema or bootstrap file differs from b548ede16d0 at 5351e1709ba.

PR11 then landed as 5bf55a51aa0, on top of #36726, and PR13 caught up with main at 727d5a0f91c. e4405de5026 merges that head. It brings #36726's backend changes, which add hooks to EntityRepository and a version read to EntityDAO. PR14's Java uses both classes and overrides neither. PR14's stable patch-id is still 9de987ca6d9, and no UI file differs from 5351e1709ba. The ten Java classes in PR14's gate were run again at e4405de5026.

PR13 then landed as d07d7174631, on top of #36179 and #36727. c6528891b75 merges 6459c4f717c, the commit before PR13's squash, without conflict. Those two PRs change JWT identity, MCP server and change request code, and no Java file in PR14's diff references a class they change. 4a082e4810e then merges main and keeps PR14's tree, because main's tree equals PR13's final head merged with them. PR14's stable patch-id is still 9de987ca6d9, over the same 49 files.

The reviews declared these minor issues, which do not block.

  • A request whose hop has no direction gets 400 [query param direction must not be null]. The message does not say which hop is wrong, and it calls a body field a query param. The shared ConstraintViolationExceptionMapper writes that text for every endpoint, so naming the hop belongs in its own PR.
  • The Live data section renders only in the classic concept inspector. main's AI-mode inspector does not mount it. See AI mode.
  • The picker test checks only that the old layout-box classes are gone, so another restyle on the Card would pass it.
  • A hop option label longer than about 33 characters may be cut off at the card edge, because core Button does not wrap and Card clips. This is inferred from the classes and was not rendered.
  • The title is a real h3, and its sibling section titles in the rail still use role="heading" at level 3. Assistive technology reads them the same.
  • The not-prose workaround now appears in seven files. Core could drop the prose heading style when a heading gets an explicit size, in its own PR.

Test evidence. At e4405de5026, the ten Java classes in PR14's gate pass 99 of 99 tests, the same count as before #36726. They were not run again at 4a082e4810e, as above. The UI gates ran at 5351e1709ba, whose UI tree equals this head's. Jest passes 1,688 of 1,688 tests in 163 suites, covering every Studio suite, the placeholder wrappers and the import guard test. TypeScript reports only main's error at IngestionWorkflowForm.tsx(413,9). ESLint, Prettier and organize-imports leave no change on PR14's UI and Playwright files. The i18n check passes for 21 locales, the import guard is clean, the Playwright impact map is up to date, and the decision record check is clean. The sixth review ran the Java gates at 0f09ecd3792. The ten Java classes passed 99 of 99 there too. OntologyConceptQueryIT passes 10 of 10 on -Pmysql-elasticsearch and 10 of 10 on -Ppostgres-opensearch. At a6e632bebba, OntologyConceptQueryServiceTest passes 10 of 10 after its tests were renamed. The gitar round's review ran the inspector suites at b548ede16d0 and passed 49 of 49.

CI. This head was pushed on 2026-10-30, and CI is running. With main as the base, GitHub links Fixes #36715, so Validate PR Metadata and maven-collate-ci can pass.

Still owed. None of these has run yet.

  • Playwright OntologyNativeLiveData.spec.ts against a live stack built from this head. The spec now asserts the posted direction.
  • The plan's ten live lanes. Lane 3 includes the self-referencing employees table, the four chips between Employee and Manager, and the chip from the restriction "managed by some Manager", which must return each manager's manager with the comment managed by.
  • The performance lane. It fails when the rail's median open time exceeds trunk by more than 10%, or when generation with three hops exceeds 150 ms at p95.
  • ROW_NUMBER() in the binding read on the CI images mysql:8.0 and postgres:15. The review's IT run used mysql:8.3.0.
  • A check in the classic inspector that the title reads 14px semibold with no margin, level with the Not stored badge, that the open picker shows its edge against the rail, and that a long hop label is not cut off.
  • The review screenshots and video.

AI mode. main's AI-mode restyle #36433 gates new screens on AI mode, and it reached PR14 through the PR13 merge. PR14 mounts the Live data section only in the classic inspector panel, renderClassicPanel (OntologyConceptInspector.tsx:1094-1099 at 0f09ecd3792). The AI-mode panel, renderAiPanel (:862-943), ends with the Assets section, so a user in AI mode sees no SQL, no Search around and no Collate preview. This follows the stack's default for AI mode. A control that lands only in a classic branch stays classic-only until the operator decides to mount it in AI mode. The classic inspector is light-only, so the section has no dark rendering to check.

Not verified yet. This PR is verified only when its unit, live and performance boxes in the plan are all checked. The operator lands the stack.

Describe your changes

Fixes #36715

The concept rail in Ontology Studio now shows the SQL that a native concept's bindings describe. The new Live data section shows a SELECT over the concept's bound columns from its bound table with LIMIT 50, a copy control, and a not stored badge. Search around adds one JOIN per relation hop, commented with the relation label, up to five hops.

OpenMetadata generates the SQL and never runs it. Running a query needs the source's credentials and a query engine, which OpenMetadata does not ship. The section asks OntologyExplorerClassBase for a live-data extension. The OSS class base returns none, so the OSS build shows the SQL, the copy control and the badge, and no preview button. Collate's override supplies Preview 3 rows, labelled Live result · not cached, and Open in SQL Studio through its query runner. That Collate change merges after this PR.

This is PR14 of the sixteen-part Ontology Studio program for release 2.1.0.

Type of change

  • New feature

High-level design

The endpoint. POST /api/v1/ontologyConcepts/{id}/query (generateOntologyConceptQuery) takes an OntologyConceptQuery body, defined in api/data/ontologyConceptQuery.json. The request sends only hops, at most five. Each hop names an OBJECT property, a target concept and a required direction, either outgoing or incoming, which is the way the Studio's chip follows the property. The response returns the hops with their labels, the sql, the dialect, the service, the serviceType, and the tables the SQL reads in join order. OntologyConceptQueryService serves it and shares one OntologyConceptBindingStore with the binding service. The call writes nothing.

The SQL generator. OntologyConceptSqlGenerator follows these rules.

  • The service type picks a dialect, ansi, snowflake, bigquery, mysql, spark or tsql. T-SQL limits with SELECT TOP 50 and the others with LIMIT 50. Any other service type is a 400, "SQL generation does not support ".
  • The backtick and bracket dialects quote every identifier. Snowflake double-quotes every identifier, because the connector stores each name in the case Snowflake reports. ansi quotes only a name that is not lowercase or that is a reserved word of an engine mapped to it.
  • A hop follows its property's object-property binding when the previous table or a candidate table binds it, and the previous table's binding wins. OntologyBindingRules now accepts OBJECT properties for this. The join uses a FOREIGN_KEY constraint whose columns equal the bound column, else the one key whose columns contain it, else the target concept's identifier column. Every candidate's equal key is tried before any containing key. A foreign key on another column never counts.
  • A hop whose property no table binds joins through the one foreign key between the two tables. Several keys are a 400 that names both tables. A key from a table to itself counts once, and the binding picks the direction. A hop whose property is unbound but whose declared inverse is bound takes the converse of the inverse's join. When both are unbound over a self key, the hop is a 400 that asks for one of them to be bound.
  • The server labels and joins each hop by the direction the request carries. An incoming hop reads as a hop over the declared inverse, so the SQL comment and the chip always name the same relation. The server never infers the direction from the declared domain and range.
  • Reads are batched whatever the hop count. One statement reads every concept's first 100 table bindings, and each concept uses its first ten visible tables in asset-id order.

The Collate hand-off. OntologyConceptLiveData.tsx renders the section, calls generateOntologyConceptQuery in rest/ontologyAPI.ts, and sends each hop with its chip's direction. OntologyExplorerClassBase.getConceptLiveDataExtension() returns no component in OSS. When an override returns one, the section renders it with conceptId and the generated OntologyConceptQuery. Both repositories share those props, so a change to them needs a paired Collate change.

Visibility. The endpoint authorizes like the binding read, with VIEW_BASIC on the concepts and hop properties. A bound table the caller may not view counts as unbound, so its columns never appear in the SQL or an error. The SQL comment uses an inverse label only when the reader may view that inverse. A concept or hop without a viewable bound table, or a hop with no join, is a 400 that names the concept and the hop.

The live-data decision record holds these rules and their consequences. It follows ADR:2026-10-16-ontology-ai-proposal-review-is-a-collate-extension.

Tests

Use cases covered

The tests cover these cases.

  • The SQL for one table and for hops through constraints, through an object-property binding, and through an identifier column.
  • Quoting and row limits for each dialect, and the 400 for an unsupported service type.
  • A self-referencing table in both directions, a hidden inverse, composite keys, and an equal key on a later candidate that wins over containing keys.
  • An incoming and an outgoing hop with the same property and target, and a hop without a direction.
  • Missing bindings and joins as named 400s, tables the reader may not view, and a reader who may view only the twelfth bound table.
  • The rail's SQL, copy, search-around, the five-hop limit, the badge, and the preview only when the class base returns an extension.
  • No entity count changes after a call.

Unit tests

OntologyConceptSqlGeneratorTest and OntologyConceptQueryServiceTest are new, and OntologyConceptBindingDaoTest and OntologyBindingRulesTest gain cases. The generator and service tests assert literal SQL strings, labels and 400 messages. OntologyConceptLiveData.test.tsx is new, and OntologyConceptInspector.test.tsx and rest/ontologyAPI.test.ts gain cases. See Test evidence above for the counts and mutants.

Backend and ingestion integration tests

OntologyConceptQueryIT is new. It asserts the SQL for bound concepts over HTTP, the 400 reasons, the joins of two hops that differ only in direction, and unchanged entity counts after a call. .github/java-tests/impact-map.json routes the new unit tests. PR14 changes no ingestion code.

Playwright and UI recording

playwright/e2e/Features/OntologyNativeLiveData.spec.ts checks the generated SQL, one search-around hop and the not stored badge, and asserts the posted direction. It has not run against a stack built from this head. The UI recording and review screenshots are not made yet.

Performance

Not measured yet. The plan's lane times the concept rail's open at trunk and head, and SQL generation with three hops.

Checklist

  • I have read the CONTRIBUTING document.
  • My PR title is Fixes <issue-number>: <short explanation>.
  • My PR is linked to a GitHub issue via Fixes #36715 above.
  • I have commented on my code, particularly in hard-to-understand areas.
  • The JSON Schema change adds a request schema for a new endpoint and stores nothing, so it needs no migration.
  • The PR build has not run in Docker with sample data yet, and no verified recording is embedded.
  • I have added tests and listed them above.
  • The final diff passed independent review, and the minor issues it declared are listed above.
  • PR14 adds the live-data decision record linked above.

The topology cursor codec now seals and opens cursors with keys derived
from the server's Fernet keys: HMAC-SHA256 over each key's 32 bytes and
the label native-ontology-topology-cursor/v1, in the same order. A value
that a server key seals for another purpose no longer opens as a cursor,
and a rotated key list still opens cursors sealed under an older key.

OntologyResource builds the codec through forServerKeys. The codec's
constructor is package-private, so code outside the package cannot build
a codec from the server keys themselves.
Without a configured Fernet key, the topology cursor codec seals cursors
with one key that the server generates at start and holds for the life
of the process. A page whose topology needs a continuation now returns
its first page and a cursor that the next request accepts, instead of
failing. A cursor sent to another server or after a restart gets the
same 400 as any other invalid cursor.
The combined graph amendment now says that each cursor key is derived
from one Fernet key, that a server without a configured key seals
cursors with a key it generates at start, and that a 412 can follow a
change to a hidden axiom at the sealed scan position. The Guard line
names the new unit tests.
Cover the wasDraftBeforeApply guard: a retried apply that loses its response while the re-read still shows the earlier APPLY_FAILED keeps Retry and the lock. Assert OntologyGraphView passes keepCanvasOnAdd to the graph.
The test seals a cursor with a random key and checks that a codec
without a server key refuses it with the same message as any other
invalid cursor. It builds no second process key, so the test name and
the decision record's Guard line now say what it checks.
Regenerate .github/playwright/impact-map.generated.json: the merged tree routes OntologyReview.spec.ts through main's changed Playwright support files, so the generator adds it to several entries.
…ition status queries

The status resolver and the unpublished-change counter chunked only the member IDs and bound every release ID into each chunk. A large page could pass the database bind-parameter limit and fail the list request. Both lists now share one budget and every release and member pair is still queried once.
Regenerate .github/playwright/impact-map.generated.json: the merged tree routes OntologyReviewTasks.spec.ts through main's changed Playwright support files, so the generator adds it to several entries and adds the new review banner, tasks drawer and captureOntologyRelease sources.
Regenerate .github/playwright/impact-map.generated.json: the merged tree routes the PR8 specs through main's changed Playwright support files, so the generator adds OntologyNativeTreeShell.spec.ts and other entries (1009 entries, +35 lines).
A user who can create axioms but not properties is offered only the
relations an axiom completes, and a reused pair is offered from the side
whose inverse the user can edit. Submitting a choice the user cannot
complete shows the permission message and writes nothing.
The native relation picker offered every registry type to any user who
could create axioms. Picking a type can also declare the properties the
ontology lacks, which needs create rights on properties, or point a
reused property at its partner, which needs edit rights on that
property. A user without those rights picked the type, and the server
refused the first write.

The picker now offers a type only when the user holds every right its
planned writes need, and it reads edit rights per property. Submitting
a type the user cannot complete shows the permission message before any
write. Hierarchy relations and pairs that the ontology already holds
linked need only the axiom right, so they stay available.

Planning every type on each model refresh made the planner quadratic in
the number of properties, because each plan built a set of all property
names and scanned the properties by IRI. The planner now indexes the
properties by IRI once per model and builds the name set only when it
declares a property. With 100 registry types and 10,000 properties,
planning the whole catalog takes about 41 ms instead of about 1.9 s, and
the plans match the previous planner's.

The decision record gains a v3 amendment that states the rule.
Regenerate docs/generated/api-reference.md: both sides counted 1988 endpoints, this branch with the release validation endpoint and binding rows with the data graph pages endpoint, so the merged file listed 1989 endpoints under a count of 1988.
…mment

The record now says iri and namespace must parse as a URI and that the server
has no uniqueness check on namespace, so the duplicate refusal is client-only.
It no longer claims other code reads the onto/ shape, and the comment above
ontologyIris keeps only the record citation.
The wrapper now maps an optional per-feature action to a title button, which
the record said it never would. Add a v3 revision and a dated amendment, and
leave the Decision text unchanged.
A Load more pressed before the second 412 still sends its one queued request
afterwards, so the record no longer says nothing is sent. It also names the
hook's Generation a load generation, to keep it apart from the server's
topology generation, and says which tests guard the amendment now that
OntologyConceptDataGraph.test.tsx is gone.
…t it proves

The AI-mode heading is a surfaced card and the classic heading is not. The
logo tile test says it checks the card surface colour rather than visibility.
…tion

Core Button sizes set their own radius and padding and TypographyColor has no
primary or secondary-text step, so the full-bleed asset rows and three text
colours stay styled at the call site. A scoped record and a v4 amendment to the
styling decision say so, and the call sites cite it. The asset name keeps its
primary colour because the tertiary button would mute it. The Data layer title
drops its primary colour because the page text is already primary. The Load
more strip puts its border and padding on a plain div instead of a core Box.
The organize-imports and ESLint pass in CI inserts this blank line.
Takes the decision record's last wording fix from PR9's final head, which main now holds as #36721.
main's tree equals the class authoring head merged in the parent commit, so this merge records the squash and keeps the first-run tree as it is.
The glossary empty state now takes its muted colour from Typography's color prop, so the exception record's closing sentence holds for every Typography in Data mode. The bindings amendment names the PR by number and the view test that pins the stale message, and the AI-mode heading test resets the mode in afterEach.
… its scroll host

The placeholder kept flex-shrink 1 and min-h-full replaced its automatic
minimum, so it shrank to the host height and its centred content spilled
above the scroll origin. Use flex-1 with h-auto so it grows past the host
and stays centred when it fits. Put the no-concepts placeholder in
Card.Content so the card supplies the padding.
At 720x450 the headline must sit at or below the scroll host top and
Create from scratch must scroll into view.
…ding

A relative iri also gets 400, iriHash covers iri and not namespace, and
the context no longer says the server stores whatever it is sent. The
v3 amendment follows the v2 heading and lead-line style.
Without tw:h-auto the shell keeps its own h-full, shrinks to the scroll host and clips the headline again, so the scroll host test now asserts it.
Takes PR11 at 4f9d666: the empty states through the placeholder wrappers, the IRI record, the scrollable first-run screen, and main with #36721 landed.
Takes PR13 at c2feefb: the per-hop direction, the recorded page size and reload budget, the core styling fixes and exception record, and PR11 with main after #36721 landed.
Takes #36726, which changes only backend repositories and their tests.
main's tree equals the first-run head this branch already holds, merged with #36726 in the parent commit, so this merge records the squash and keeps the Data mode tree as it is.
Takes PR13 at 727d5a0, which records main with #36723 and #36726. #36726 changes backend repositories that the live data endpoint does not override.
Base automatically changed from harshach/ontology-studio/13-data-mode to main October 30, 2026 19:18
Takes #36179 and #36727, which change JWT identity, MCP server and change request code and their tests. None of their files is in this branch's diff.
main's tree equals the Data mode head this branch already holds, merged with #36179 and #36727 in the parent commit, so this merge records the squash and keeps the live data tree as it is.
@gitar-bot

gitar-bot Bot commented Oct 30, 2026

Copy link
Copy Markdown
Code Review ✅ Approved 1 closed / 1 findings

🔴 High risk · The new query endpoint's VIEW_BASIC checks govern which bound table metadata appears in generated SQL; an access-control mistake could expose otherwise hidden schema details.

Adds a Live data section to the Ontology Studio concept rail that shows generated SQL for a native concept's bindings, with search-around hops joined by direction. The restyle of the Live data panel onto core Card and Typography components was addressed. No open issues found.

✅ 1 closed
✅ Quality: Live data panel restyles core Box/Typography via className

📄 openmetadata-ui/src/main/resources/ui/src/components/governance/ontology/OntologyConceptInspector/OntologyConceptLiveData.tsx:195-199 📄 openmetadata-ui/src/main/resources/ui/src/components/governance/ontology/OntologyConceptInspector/OntologyConceptLiveData.tsx:230-235
The search-around picker gives a core Box a border, radius, tw:bg-surface background and padding through className (lines 195-199). The section title passes className="tw:text-primary" to Typography instead of using its color prop (lines 230-235). Both break the styling rule that callers must not set padding, border, radius, background or text colour on core components, and that Box is for layout only with no background (ADR:2026-10-11-migrated-antd-components-adopt-core-styling, ADR:2026-10-12-dark-mode-targets-ai-mode-core-surfaces-only). The practical effect: the picker's surface and border are set at the call site, so they won't follow changes to the core surface tokens. Use a core Card for the picker and Typography color="primary" for the title. The title also uses role="heading" on a non-heading element where a semantic heading would do.

Verification

Rules No rules evaluated

Functional validation Not enabled

Set up

Auto-approval Not enabled

Set up

For more verification details, comment gitar display:verbose.

Options

Display: compact → Showing compact verification results.

Comment with these commands to change the behavior for this request:

Compact
gitar display:verbose         

Was this helpful? React with 👍 / 👎 | Powered by Gitar — free for open source

@github-actions

Copy link
Copy Markdown
Contributor

⚠️ UI Checkstyle passed — lint findings in changed files

🔍 ESLint findings in this PR's files — 0 error(s), 6 warning(s)

Errors block the build. Warnings do not yet — they are rules whose backlog is still
being worked down, listed so this PR does not add to it. See docs/ui-code-quality-gate.md.

0 error(s), 6 warning(s) across 2 changed file(s).

Count Rule
5 openmetadata-imports/review-sequential-api-calls
1 openmetadata-imports/no-api-calls-in-iteration
All findings
Location Rule Message
🟡 src/rest/ontologyAPI.test.ts:280:12 openmetadata-imports/review-sequential-api-calls Review these sequential API requests. If they are independent, start them together with Promise.all/Promise.allSettled; keep sequencing only when data-dependent
🟡 src/rest/ontologyAPI.test.ts:288:12 openmetadata-imports/review-sequential-api-calls Review these sequential API requests. If they are independent, start them together with Promise.all/Promise.allSettled; keep sequencing only when data-dependent
🟡 src/rest/ontologyAPI.test.ts:319:12 openmetadata-imports/review-sequential-api-calls Review these sequential API requests. If they are independent, start them together with Promise.all/Promise.allSettled; keep sequencing only when data-dependent
🟡 src/rest/ontologyAPI.test.ts:341:5 openmetadata-imports/review-sequential-api-calls Review these sequential API requests. If they are independent, start them together with Promise.all/Promise.allSettled; keep sequencing only when data-dependent
🟡 src/rest/ontologyAPI.test.ts:477:24 openmetadata-imports/review-sequential-api-calls Review these sequential API requests. If they are independent, start them together with Promise.all/Promise.allSettled; keep sequencing only when data-dependent
🟡 src/rest/ontologyAPI.ts:133:28 openmetadata-imports/no-api-calls-in-iteration Avoid issuing one API request per item. Fetch at the data owner, use a bulk endpoint, or use useQueries with an intentional concurrency policy.

Fix locally (fast - only checks files changed in this branch):

make ui-checkstyle-changed

@sonarqubecloud

Copy link
Copy Markdown

@sonarqubecloud

Copy link
Copy Markdown

@harshach
harshach merged commit 344652d into main Oct 30, 2026
76 checks passed
@harshach
harshach deleted the harshach/ontology-studio/14-live-data branch October 30, 2026 20:04
harshach added a commit that referenced this pull request Oct 30, 2026
Takes #36725 and #36734. The ontology REST client test keeps both new imports, exportOntologyRelease and generateOntologyConceptQuery, and the Playwright impact map is regenerated from the merged sources.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend safe to test Add this label to run secure Github workflows on PRs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Ontology Studio: generate SQL from bindings and hand live data to a Collate extension

1 participant