Fixes 36357: Hex data models and column-level lineage from project SQL cells by ulixius9 · Pull Request #36371 · open-metadata/OpenMetadata

ulixius9 · 2026-10-27T15:25:35Z

Describe your changes:

The Hex connector ingested each project as a dashboard and nothing more: no data models, no column lineage, and table → dashboard lineage only from the warehouse query history (needs warehouse credentials, 7-day window). This adds one HexDataModel per project, with the columns each SQL cell outputs, and column-level lineage from the warehouse table columns to those cells, then data model → dashboard, the way Power BI / Tableau / Sigma do.

Type of change:

New feature

High-level design:

Source of the SQL. Each project's SQL cells come from POST /v1/projects/export (YAML). The export is needed because only it names each cell's result variable: in real Hex notebooks most dashboard-facing cells are chained SQL that query another cell's result (FROM base_opptys) rather than a warehouse table. GET /v1/cells stands in only when the export isn't allowed (403/404). The warehouse query-history lineage stays, and also adds any table that no cell resolved.

Columns and lineage (hex/cell_lineage.py). A Hex SQL cell is a plain SELECT with no target table, so LineageParser returns no column lineage for it. Instead:

Parse with sqlglot in the connection's dialect (from GET /v1/data-connections, which also gives the connection's default database/schema; DuckDB for dataframe SQL). Jinja input parameters ({{ x }}) become NULL so parameterized cells still parse.
Inline upstream cells as CTEs (each nested with its own WITH, so its CTE names can't capture the downstream cell's tables), which is how Hex runs chained SQL, so every output column traces back to the warehouse through any number of cells. A warehouse cell chains only to cells on its own connection, as in Hex. Dataframe SQL cells are kept when they chain off warehouse cells; the names they read themselves are dataframes, never looked up as warehouse tables. A table an upstream cell couldn't resolve stays unresolved downstream.
Resolve each table reference in OpenMetadata, hand sqlglot the table schemas (for SELECT * and unqualified columns), and walk sqlglot.lineage per output column. Columns resolve case-insensitively, quoted or not, so Snowflake "REGION" and Postgres "createdAt" both match. As a result output names are lower-cased, except a column passed straight through, which keeps its table's casing. A type carries over only when every hop is a bare column.

Entities (hex/metadata.py). One DashboardDataModel per project (<service>.model.<projectId>), with one STRUCT column per SQL cell holding its output columns, so names never clash across cells. Each group is named by the cell's stable id (the export's cellId, which is the cells API's staticId), with the cell label as its displayName, so renaming or reordering cells keeps the curated descriptions, tags and lineage. Edges: table → data model (with columnsLineage to <model>.<cell>.<column>), data model → dashboard. The server already validates nested child columns as lineage endpoints. With includeDataModels off, the tables link straight to the dashboard.

Safety.

Lineage doesn't guess. A table reference resolves only when exactly one table matches across all DB service prefixes, and only with a schema (from the SQL, the connection's defaults, or the prefix). A bare name matching any service is never used.
Transient failures don't rewrite or delete anything. If the export fails transiently, the cells or data connections can't be read, or the table search errors, the project's existing data model is kept as is: marked as seen and still linked. Rebuilding it from incomplete input would drop columns along with their descriptions, tags and lineage.
The Hex client retries 429s: Hex allows about 30 requests per minute per endpoint, and the OM REST client treated 429 as a hard limit.

Schema. HexDataModel is added to dataModelType in dashboardDataModel.json. No migration is needed: it only adds an enum value, and no stored entity changes.

Alternatives rejected. The cells API alone has no result-variable names, so lineage stops at the first chained cell. The warehouse query history needs warehouse credentials, a lookback window, and misses cached results. A flat column list per project (DataHub's InputFields) has no equivalent in OM: column lineage must end on a child of the target entity.

Tests:

Use cases covered

A project's SQL cells become one data model with a column group per cell, and table columns → data-model columns → dashboard lineage.
Chained SQL across cells (warehouse → warehouse → dataframe SQL), Jinja parameters, quoted aliases, SELECT * expansion, CTEs, joins, and stars over Python dataframes.
Export not allowed → cells API. Transient export failure, unreadable cells or data connections, or table search down → existing data model kept and linked.
Same table in two services, or across two prefixes → no lineage. Bare table name → needs a schema from the connection or a prefix.
Query-history tables no cell resolved → still linked. includeDataModels off → table → dashboard.
Quoted identifiers in any case. Chained cells reading tables found through the connection's defaults, never re-resolving what the upstream cell couldn't, and never chaining across connections. UNION with a * over an unknown table. Outputs sharing a name (* over a join) all kept. Column names sanitized and kept unique. dataModelFilterPattern honored. A cell's CTE never shadows another cell's table. A renamed cell keeps its column group. WITH RECURSIVE, Jinja {% if %}/{% else %}, a bare * over a join with a dataframe. Notebooks with long chains stay bounded (a cell over 10,000 nodes isn't inlined).

Unit tests

I added unit tests for the new/changed logic.
Files added/updated: ingestion/tests/unit/topology/dashboard/test_hex_cell_lineage.py (new), test_hex.py.
PYTHONPATH=src python -m pytest tests/unit/topology/dashboard/test_hex*.py → 146 passed (test_hex.py, test_hex_cell_lineage.py, test_hex_client.py, test_hex_ingestion_flow.py, test_hex_lineage.py).
Coverage on changed files (pytest-cov): cell_lineage.py 94%, client.py 90% (the rest is the untouched test_project), metadata.py 90%, models.py 100%.
Also clean: ruff check / ruff format --check, check_ruff_suppressions.py --check, and basedpyright on hex/ (0 errors).

Backend integration tests

Not applicable (no backend API changes). The schema enum change's impacted Java tests ran locally:

Local Java test run: PASSED

Commit: 33b47d3c0db1 (base origin/main); later commits change only ingestion Python, a lint baseline and a decision record
Finished: 2026-10-27 13:43 UTC · 7.8 min

Step	Scope	Classes	Tests	Failed	Skipped	Min	Result
unit	openmetadata-service (11 classes)	11	678	0	0	1.4	passed
integration	mysql-elasticsearch · parallel	10	1884	0	156	6.4	passed

Ingestion integration tests

Not applicable: there are no Hex integration tests (tests/integration/ has no Hex directory). The connector was verified end to end against a live Hex workspace and a local server instead (below).

Playwright (UI) tests

Not applicable (no UI changes).

Manual testing performed

Built this branch (mvn -DskipTests clean package) and started ./docker/run_local_docker.sh -m ui -d mysql -s true -i false; /api/v1/system/version → 2.0.0-SNAPSHOT.
Created a Snowflake service with the three B2B_DEMO_DATA.PROD tables that Hex's demo projects read (Hex's demo warehouse credentials aren't exposed, so the tables were created through the API).
Ran the Hex metadata workflow from the host against a live Hex trial workspace (two demo projects on a Snowflake connection) → 0 errors, 0 warnings. Re-ran on the final commit: same results.
Verified on the server: both data models are HexDataModel. Sales Revenue Dashboard has 7 cell groups, displayed by label (Opportunities, Won Opportunities, Current Quarter Pipeline, Join Opportunities and Targets, Quarterly ARR Progress, S0/S1 Pipeline), and the server accepted all 74 column links (none filtered as invalid). Chained columns resolve across cells, e.g. Quarterly ARR Progress.qtd_actual (SUM(ARR), 3 cells downstream) ← dim_opportunities.arr.
Checked the data model's columns and the table → data model → dashboard lineage in the UI. A re-run is idempotent.

UI screen recording:

Not applicable: ingestion connector change with no UI code. The new entities render through the existing data model and lineage pages (checked by hand, step 5 above).

Checklist:

Confidence Score: 5/5

The PR appears safe to merge, with a non-blocking loss of nested type details.

Findings

Nested types lose details ▶

Summary

Adds one HexDataModel per Hex project, with stable column groups for SQL cells and warehouse column lineage through chained cells.

Hex projects show SQL-cell models with lineage back to warehouse columns.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  Export[Hex project SQL cells] --> Resolve[Resolve tables and chained cells]
  Resolve --> Model[HexDataModel with one group per cell]
  Tables[Warehouse table columns] -->|Column lineage| Model
  Model --> Dashboard[Hex dashboard]
  History[Additional query-history tables] --> Dashboard

_{Reviews (6) · Last reviewed commit: "Hex: keep a kept data model linked, copy..." · Reviewed by Greptile}

Each Hex project now gets one HexDataModel with a struct column per SQL cell, holding the columns that cell outputs. Column lineage runs from the warehouse table columns to those cells, then data model -> dashboard. The SQL comes from the project export (falling back to the cells API), which names each cell's result variable, so chained SQL cells are inlined as CTEs the way Hex runs them and trace back to the warehouse. Input parameters ({{ x }}) parse as values, and sqlglot resolves each output column with table schemas from OpenMetadata. The warehouse query history path stays as the fallback when no cells can be read. The Hex client retries 429 responses, as Hex rate-limits each endpoint to about 30 requests per minute.

A project whose cells or data connections can't be read keeps the data model an earlier run created, marked as seen and still linked to its dashboard, instead of being deleted over a failed API call. A failed data connections read is retried for the next project rather than cached. A table reference matching several tables (the same schema.table in two services, or a bare name) now resolves to none instead of the first search hit.

…ailures - A table reference resolves only when exactly one table matches across all db service prefixes, and only with a schema: from the SQL, the Hex connection's defaults (now read from the data connections API) or the prefix. A bare name is no longer matched in any service. - Tables the warehouse query history found but no cell resolved still link to the dashboard, instead of cells replacing that lineage. - The cells API stands in for the export only when the export is not allowed (403/404). A transient export failure, or a failed table search, now leaves the project's data model as it is rather than rewriting it from incomplete cells. - The data model lookup for the dashboard link can no longer fail the dashboard, and an unread project's model is marked as seen without one.

… names - Columns resolve case-insensitively, quoted or not, so Snowflake's "REGION" and Postgres' "createdAt" match the table's columns instead of losing their lineage (a table-qualified one used to drop the whole cell). A failed qualify now keeps the columns, without lineage. A column passed straight through keeps its table's casing. - A cell is stored for the cells chained after it once its references name the tables found, so tables it found through its connection's defaults still resolve from a dataframe cell. - Column names are made unique after they are made valid, so two outputs can no longer collapse into one rejected name. - Data connections the token can't read (403/404) are treated as none known, rather than as a transient failure that would keep every data model unchanged forever.

- A table reference a cell could not resolve now carries the database and schema it was searched under, so a chained cell inlining it, which has no connection defaults of its own, can't match a table elsewhere. - A quoted alias keeps the spelling it was given in the data model.

- A table reference a cell could not resolve is marked, and a chained cell inlining it leaves it unresolved, rather than searching it again without that cell's connection defaults (which could match a table in another database through a prefix). - A warehouse cell only inlines upstream cells on its own connection, as Hex does; SQL on dataframes can read any cell's result. - A UNION branch selecting * from an unknown table no longer drops the cell: sqlglot's IndexError there now keeps the columns without lineage.

…odels - The names a dataframe SQL cell reads are dataframes, so they are never looked up as warehouse tables; such a cell reaches the warehouse only through the cells it chains to. - Outputs sharing a name (o.id, c.id, or * over a join) are all kept, the repeats suffixed, instead of all but the first being dropped. - dataModelFilterPattern applies to Hex data models.

- An inlined upstream cell keeps its own WITH, and table references are found scope by scope, so a CTE defined inside an upstream cell can no longer capture a warehouse table of the same name the downstream cell reads. - Each cell's column group is named by its stable id (the export's cellId, the cells API's staticId) with the label as its display name, so a renamed or reordered cell keeps its columns' descriptions, tags and lineage. - Lineage never targets a data model excluded by dataModelFilterPattern, which this run deletes.

- Add ADR:2026-10-27-hex-project-is-one-data-model-with-a-column-group-per-cell-id: the data model and its column groups are named by project and cell id, tables resolve only uniquely and with a schema, and a failed read keeps the existing data model. - A cell over 10,000 nodes isn't stored for inlining, so a notebook where each cell joins the ones before it no longer grows exponentially; the cells chained to it keep their columns, without lineage. - Inlining keeps WITH RECURSIVE, so a recursive CTE isn't looked up as a table. - Only the first branch of a Jinja {% if %} is kept, so the SQL parses. - A bare * over a join with a dataframe is spelled out per table, so the known table's columns are kept.

The kept-model link now applies dataModelFilterPattern like the data model and lineage stages, so a dashboard is never pointed at a data model this run deletes.

Copilot

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

github-actions · 2026-10-27T15:25:52Z

❌ PR checklist incomplete

This PR cannot be merged until the following are addressed on its linked issue:

Linked issue Hex connector: ingest data models and column-level lineage from project SQL cells #36357 is not added to the Shipping project.

The fields live on the linked issue in the Shipping project (open the issue → right sidebar → Projects). After you set them, re-run this check (or push a commit) — issue/project changes do not re-trigger it automatically.

Maintainers can bypass this check by adding the skip-pr-checks label.

github-actions · 2026-10-27T15:31:47Z

✅ Generated Sources Auto-Updated

The generated TypeScript types (src/generated/) and dereferenced JSON
schemas (src/jsons/, public/jsons/) have been automatically updated
based on JSON schema changes in this PR.

Copilot

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

…caps - A t.* is dropped only where t names an unknown table in that SELECT's own scope, so an alias reused across CTEs no longer loses the known table's columns (and their lineage). - The table lookup memo is an LRUCache(1000), and a project keeps at most 100,000 nodes of cell queries for chaining; cells past it are stored as unresolvable rather than read as warehouse tables. - The decision record now also states the child column casing, the inlining caps and the 429 retry budget.

Copilot

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

github-actions · 2026-10-27T16:05:52Z

Jest test Coverage

UI tests summary

Lines	Statements	Branches	Functions
	80.02% (132079/165045)	66.59% (46761/70220)	69.63% (26509/38071)

Copilot

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

github-actions · 2026-10-28T13:13:55Z

🚦 Removed from the merge queue — `merge_conflict` (2026-10-28T13:13:39Z)

The entry left the queue before it was built, so no checks ran against it.

Conflicts with the latest main — rebase, then re-queue.

Keep the data model ingested without a link when the Hex workspace id can't be fetched, as main now does for the dashboard.

Copilot

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

gitar-bot · 2026-10-28T13:50:31Z

Code Review ✅ Approved 2 closed / 2 findings

🟡 Medium risk · Adding Hex project data models and chained SQL column lineage changes ingested metadata and could misattribute or omit lineage when resolution is incomplete.

Adds a Hex data model per project with a column group per SQL cell, plus column-level lineage from warehouse tables through chained cells to the dashboard. The unrecorded decisions around the 429 retry budget, inline node cap and column casing were resolved, and the passthrough struct tag/profile copy was closed without a fix. No open issues found.

✅ 2 closed

✅ Quality: Unrecorded decisions: 429 retry budget, inline node cap, column casing

📄 ingestion/src/metadata/ingestion/source/dashboard/hex/client.py:73-76 📄 ingestion/src/metadata/ingestion/source/dashboard/hex/cell_lineage.py:35-49 📄 ingestion/src/metadata/ingestion/source/dashboard/hex/metadata.py:646-650 📄 docs/decisions/2026-10-27-hex-project-is-one-data-model-with-a-column-group-per-cell-id.md:22-31
The PR adds one ADR (ADR:2026-10-27-hex-project-is-one-data-model-with-a-column-group-per-cell-id). It covers the data model's shape, the cell-id naming, table resolution and how failures are handled. Three more judgment calls in the PR have no record:

429 retry budget. client.py:73-76 sets retry_codes=[429, 503, 504] and limit_codes=[]. On the shared REST client this means up to 3 retries with escalating sleeps of 30, 60 and 90s, per call, for every project export. The description only says "The Hex client retries 429 responses because Hex limits requests per endpoint", and that rule exists nowhere in the tree.

Inlining cap. cell_lineage.py:49 sets MAX_INLINED_NODES = 10_000. Above it, chained cells silently lose their lineage.

Child column casing. Output column names are lower-cased, except for passed-through columns and quoted aliases (metadata.py:646-650, cell_lineage.py:35-39). The ADR itself says data model column names are a contract that users curate, so their casing is part of that contract. If this rule changes later, existing curated child columns are orphaned.

Add these as decisions in the existing record, or in a second one. Sentences it could assert:

"The Hex client retries 429/503/504 up to the default 3 times with the default escalating 30s wait, and never treats 429 as a hard limit, because Hex allows ~30 requests/minute per endpoint."

"A cell's query is stored for chaining only up to 10,000 sqlglot nodes; larger cells leave the cells chained to them untraced."

"Child column names are the lower-cased output name, except a quoted alias keeps its spelling and a passthrough column keeps its table's casing."

Closed: Bug: Passthrough struct children copy the table's tags, profile and descriptions

📄 ingestion/src/metadata/ingestion/source/dashboard/hex/metadata.py:779-793
_output_column builds a passed-through struct's children with _without_fqns(child), which is model_copy of the warehouse table's Column with only fullyQualifiedName cleared. Each nested child therefore also takes the source table's description, tags (including auto-classified PII tags), constraint, profile and customMetrics. All of these go into the CreateDashboardDataModelRequest on every run. So the data model's nested columns pick up table tags and profiler data that nobody curated on the data model. A user who removes those tags gets them back on the next ingestion. The top-level passthrough column copies only type fields, so the two levels also behave differently. Fix: when copying a table column, keep only the structural fields (name, type, arrayDataType, dataTypeDisplay, children). Keep _without_fqns for re-sending the data model's own existing groups, where keeping curated metadata is the point.

Verification

Rules No rules evaluated

Functional validation Not enabled

Set up

Auto-approval Not enabled

Set up

_{For more verification details, comment gitar display:verbose.}

Options

Display: compact → Showing compact verification results.

Comment with these commands to change the behavior for this request:

`Compact`
`gitar display:verbose`

_{Was this helpful? React with 👍 / 👎 | Powered by Gitar — free for open source}

sonarqubecloud · 2026-10-28T14:10:53Z

Quality Gate passed for 'open-metadata-ui'

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

sonarqubecloud · 2026-10-28T14:18:26Z

Quality Gate passed for 'open-metadata-ingestion'

Issues
5 New issues
0 Accepted issues

Measures
0 Security Hotspots
95.3% Coverage on New Code
0.4% Duplication on New Code

See analysis details on SonarQube Cloud

github-actions · 2026-10-28T18:14:23Z

🚦 Removed from the merge queue — `failed_checks` (2026-10-28T18:14:08Z)

Blocked the queue: integration-tests-postgres-opensearch

Maven Collate Tests — maven-collate-ci
Integration Tests - PostgreSQL + OpenSearch — integration-tests-postgres-opensearch, Integration Test Lane (parallel)
Prune dead merge-queue refs — cancel-stale-runs
Integration Test Report (postgres-opensearch, parallel)

github-actions · 2026-10-29T05:41:57Z

🚦 Removed from the merge queue — `failed_checks` (2026-10-29T05:41:45Z)

Blocked the queue: py-tests-status

Maven Collate Tests — maven-collate-ci
py-tests-postgres — py-tests-status, python / Verify Expected Jobs, python / Integration Tests (shard-2)

…odel in the generated types Matches the resolution on main (#36371) of the data model sourceUrl against the workspace-scoped Hex URLs, which return None when the workspace id can't be read. The generated TypeScript enum value is the one #36371 generated on main; the type-generation workflow fails on 2.0 so it never added it here.

Matches the resolution on main (#36371) of the data model sourceUrl against the workspace-scoped Hex URLs, which return None when the workspace id can't be read.

2.0 now has docs/decisions/, so the record #36371 added on main (with its v2 amendment from #36541) comes along instead of leaving the ADR citation in hex/metadata.py dangling.

…(2.0) (#36388) * Fixes 36357: Hex data models and column-level lineage from project SQL cells (2.0) Backport of #36371 to 2.0, squashed from 33b47d3, 16ad476, 22d96d4, 6af0b43, 791ff4a, c42a857, cacf30b, c5d461c, 61debe6, 05f7d5c and d996669. Adapted to 2.0: - es_search_from_fqn has no raise_on_error here and returns None for a failed search, so the Hex source runs the FQN search itself and treats a search that didn't answer as an error, not as "no such table". - The project's tag handling (tag_utils) is kept as it is on 2.0. - The decision record is not added: 2.0 has no docs/decisions. The code cites ADR:2026-10-27-hex-project-is-one-data-model-with-a-column-group-per-cell-id on main. * Hex: keep models without data connections, keep broken cells' groups - Data connections the token can't read raise HexAccessDeniedError and are asked for once per run. Without them an existing data model is left as it is instead of being rebuilt without dialects and connection defaults; only a project without one gets one. - A cell whose SQL fails to parse keeps its column group from the existing data model, with what users curated on it. - A column name is parsed out of the lineage leaf rather than split on dots, so a quoted "customer.id" is one name. - A column passed straight through keeps its table column's array element type, display type and nested fields. (cherry picked from commit a6b8920) * Hex (2.0): a failed read of a table the search found keeps the data model The 2.0 FQN search reads each table it finds; an error there now counts as a failed search, so the project's existing data model is kept rather than rebuilt without that cell. * Hex: keep a kept data model linked, copy only the type of passed-through columns - A data model kept because its existing groups couldn't be read stays linked to the dashboard: the data model stage records what it kept and the dashboard stage links it, instead of only when the cells couldn't be read. - A passed-through struct's fields copy only the table column's type, not its description, tags, constraint or profile. (cherry picked from commit c02d176) * Hex: a data model without a dashboard URL is still ingested; HexDataModel in the generated types Matches the resolution on main (#36371) of the data model sourceUrl against the workspace-scoped Hex URLs, which return None when the workspace id can't be read. The generated TypeScript enum value is the one #36371 generated on main; the type-generation workflow fails on 2.0 so it never added it here. * Hex: cells reading a broken cell keep their data model groups A cell whose SQL fails keeps its existing group, but the cells reading its result lost theirs: dataframe SQL over it was dropped and a warehouse cell searched its name as a table, coming back with UNKNOWN columns. A cell reading a failed cell now fails too, down the chain, so each keeps its group. (cherry picked from commit ea7e122) * Hex: add the data model decision record the code cites 2.0 now has docs/decisions/, so the record #36371 added on main (with its v2 amendment from #36541) comes along instead of leaving the ADR citation in hex/metadata.py dangling.

…(2.1) (#36387) * Hex: ingest data models and column lineage from project SQL cells Each Hex project now gets one HexDataModel with a struct column per SQL cell, holding the columns that cell outputs. Column lineage runs from the warehouse table columns to those cells, then data model -> dashboard. The SQL comes from the project export (falling back to the cells API), which names each cell's result variable, so chained SQL cells are inlined as CTEs the way Hex runs them and trace back to the warehouse. Input parameters ({{ x }}) parse as values, and sqlglot resolves each output column with table schemas from OpenMetadata. The warehouse query history path stays as the fallback when no cells can be read. The Hex client retries 429 responses, as Hex rate-limits each endpoint to about 30 requests per minute. (cherry picked from commit 33b47d3) * Hex: keep data models on failed API reads, never guess a table A project whose cells or data connections can't be read keeps the data model an earlier run created, marked as seen and still linked to its dashboard, instead of being deleted over a failed API call. A failed data connections read is retried for the next project rather than cached. A table reference matching several tables (the same schema.table in two services, or a bare name) now resolves to none instead of the first search hit. (cherry picked from commit 16ad476) * Hex: resolve tables only when unambiguous, keep models on transient failures - A table reference resolves only when exactly one table matches across all db service prefixes, and only with a schema: from the SQL, the Hex connection's defaults (now read from the data connections API) or the prefix. A bare name is no longer matched in any service. - Tables the warehouse query history found but no cell resolved still link to the dashboard, instead of cells replacing that lineage. - The cells API stands in for the export only when the export is not allowed (403/404). A transient export failure, or a failed table search, now leaves the project's data model as it is rather than rewriting it from incomplete cells. - The data model lookup for the dashboard link can no longer fail the dashboard, and an unread project's model is marked as seen without one. (cherry picked from commit 22d96d4) * Hex: resolve quoted identifiers, keep chained defaults, unique column names - Columns resolve case-insensitively, quoted or not, so Snowflake's "REGION" and Postgres' "createdAt" match the table's columns instead of losing their lineage (a table-qualified one used to drop the whole cell). A failed qualify now keeps the columns, without lineage. A column passed straight through keeps its table's casing. - A cell is stored for the cells chained after it once its references name the tables found, so tables it found through its connection's defaults still resolve from a dataframe cell. - Column names are made unique after they are made valid, so two outputs can no longer collapse into one rejected name. - Data connections the token can't read (403/404) are treated as none known, rather than as a transient failure that would keep every data model unchanged forever. (cherry picked from commit 6af0b43) * Hex: don't widen an unresolved upstream reference, keep quoted aliases - A table reference a cell could not resolve now carries the database and schema it was searched under, so a chained cell inlining it, which has no connection defaults of its own, can't match a table elsewhere. - A quoted alias keeps the spelling it was given in the data model. (cherry picked from commit 791ff4a) * Hex: never re-resolve an unresolved upstream table, chain per connection - A table reference a cell could not resolve is marked, and a chained cell inlining it leaves it unresolved, rather than searching it again without that cell's connection defaults (which could match a table in another database through a prefix). - A warehouse cell only inlines upstream cells on its own connection, as Hex does; SQL on dataframes can read any cell's result. - A UNION branch selecting * from an unknown table no longer drops the cell: sqlglot's IndexError there now keeps the columns without lineage. (cherry picked from commit c42a857) * Hex: dataframe names aren't tables, keep same-named outputs, filter models - The names a dataframe SQL cell reads are dataframes, so they are never looked up as warehouse tables; such a cell reaches the warehouse only through the cells it chains to. - Outputs sharing a name (o.id, c.id, or * over a join) are all kept, the repeats suffixed, instead of all but the first being dropped. - dataModelFilterPattern applies to Hex data models. (cherry picked from commit cacf30b) * Hex: nest inlined cells, name column groups by cell id, honor the filter - An inlined upstream cell keeps its own WITH, and table references are found scope by scope, so a CTE defined inside an upstream cell can no longer capture a warehouse table of the same name the downstream cell reads. - Each cell's column group is named by its stable id (the export's cellId, the cells API's staticId) with the label as its display name, so a renamed or reordered cell keeps its columns' descriptions, tags and lineage. - Lineage never targets a data model excluded by dataModelFilterPattern, which this run deletes. (cherry picked from commit c5d461c) * Hex: record the data model naming, bound chained SQL, more SQL shapes - Add ADR:2026-10-27-hex-project-is-one-data-model-with-a-column-group-per-cell-id: the data model and its column groups are named by project and cell id, tables resolve only uniquely and with a schema, and a failed read keeps the existing data model. - A cell over 10,000 nodes isn't stored for inlining, so a notebook where each cell joins the ones before it no longer grows exponentially; the cells chained to it keep their columns, without lineage. - Inlining keeps WITH RECURSIVE, so a recursive CTE isn't looked up as a table. - Only the first branch of a Jinja {% if %} is kept, so the SQL parses. - A bare * over a join with a dataframe is spelled out per table, so the known table's columns are kept. (cherry picked from commit 61debe6) * Hex: don't keep a filtered-out data model linked when Hex can't be read The kept-model link now applies dataModelFilterPattern like the data model and lineage stages, so a dashboard is never pointed at a data model this run deletes. (cherry picked from commit 05f7d5c) * Hex: drop unknown stars per scope, bound per-project lookups, record caps - A t.* is dropped only where t names an unknown table in that SELECT's own scope, so an alias reused across CTEs no longer loses the known table's columns (and their lineage). - The table lookup memo is an LRUCache(1000), and a project keeps at most 100,000 nodes of cell queries for chaining; cells past it are stored as unresolvable rather than read as warehouse tables. - The decision record now also states the child column casing, the inlining caps and the 429 retry budget. (cherry picked from commit d996669) * Update generated TypeScript types and dereferenced schemas * Hex: keep models without data connections, keep broken cells' groups - Data connections the token can't read raise HexAccessDeniedError and are asked for once per run. Without them an existing data model is left as it is instead of being rebuilt without dialects and connection defaults; only a project without one gets one. - A cell whose SQL fails to parse keeps its column group from the existing data model, with what users curated on it. - A column name is parsed out of the lineage leaf rather than split on dots, so a quoted "customer.id" is one name. - A column passed straight through keeps its table column's array element type, display type and nested fields. (cherry picked from commit a6b8920) * Hex: keep a kept data model linked, copy only the type of passed-through columns - A data model kept because its existing groups couldn't be read stays linked to the dashboard: the data model stage records what it kept and the dashboard stage links it, instead of only when the cells couldn't be read. - A passed-through struct's fields copy only the table column's type, not its description, tags, constraint or profile. (cherry picked from commit c02d176) * Hex: a data model without a dashboard URL is still ingested Matches the resolution on main (#36371) of the data model sourceUrl against the workspace-scoped Hex URLs, which return None when the workspace id can't be read. * Hex: cells reading a broken cell keep their data model groups A cell whose SQL fails keeps its existing group, but the cells reading its result lost theirs: dataframe SQL over it was dropped and a warehouse cell searched its name as a table, coming back with UNKNOWN columns. A cell reading a failed cell now fails too, down the chain, so each keeps its group. (cherry picked from commit ea7e122) --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

ulixius9 added 10 commits October 27, 2026 19:05

Hex: don't keep a filtered-out data model linked when Hex can't be read

05f7d5c

The kept-model link now applies dataModelFilterPattern like the data model and lineage stages, so a dashboard is never pointed at a data model this run deletes.

ulixius9 requested a review from a team as a code owner October 27, 2026 15:25

ulixius9 requested review from akashverma0786 and a balanced review from Copilot October 27, 2026 15:25

Copilot AI reviewed Oct 27, 2026

github-actions Bot added Ingestion safe to test Add this label to run secure Github workflows on PRs labels Oct 27, 2026

gitar-bot Bot reviewed Oct 27, 2026

View reviewed changes

Comment thread ingestion/src/metadata/ingestion/source/dashboard/hex/client.py

greptile-apps Bot reviewed Oct 27, 2026

View reviewed changes

Comment thread ingestion/src/metadata/ingestion/source/dashboard/hex/cell_lineage.py Outdated

Comment thread ingestion/src/metadata/ingestion/source/dashboard/hex/metadata.py Outdated

Update generated TypeScript types and dereferenced schemas

4790784

Copilot AI balanced review requested due to automatic review settings October 27, 2026 15:31

Copilot AI reviewed Oct 27, 2026

Copilot AI balanced review requested due to automatic review settings October 27, 2026 15:46

Copilot AI reviewed Oct 27, 2026

Merge branch 'main' into ulixius9/hex-datamodels-powerbi

38dd494

Copilot AI balanced review requested due to automatic review settings October 28, 2026 05:36

Copilot AI reviewed Oct 28, 2026

IceS2 approved these changes Oct 28, 2026

View reviewed changes

ulixius9 added this pull request to the merge queue Oct 28, 2026

github-merge-queue Bot removed this pull request from the merge queue due to a conflict with the base branch Oct 28, 2026

github-actions Bot added the merge conflict Conflicts with the latest main; removed from the merge queue label Oct 28, 2026

ulixius9 enabled auto-merge October 28, 2026 13:47

Merge branch 'main' into ulixius9/hex-datamodels-powerbi

b32608b

Keep the data model ingested without a link when the Hex workspace id can't be fetched, as main now does for the dashboard.

github-actions Bot removed the merge conflict Conflicts with the latest main; removed from the merge queue label Oct 28, 2026

Copilot AI balanced review requested due to automatic review settings October 28, 2026 13:49

Copilot AI reviewed Oct 28, 2026

ulixius9 added this pull request to the merge queue Oct 28, 2026

github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 28, 2026

ulixius9 added this pull request to the merge queue Oct 29, 2026

github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 29, 2026

ulixius9 added this pull request to the merge queue Oct 29, 2026

sonika-shah mentioned this pull request Oct 29, 2026

test(search): stop SearchRankingRelevanceIT plural test flaking the merge queue #36521

Merged

Merged via the queue into main with commit ee6dc4f Oct 29, 2026
93 of 94 checks passed

ulixius9 deleted the ulixius9/hex-datamodels-powerbi branch October 29, 2026 07:46

ulixius9 mentioned this pull request Oct 29, 2026

Fixes 36357: Hex cells reading a broken cell keep their data model groups #36541

Open

15 tasks done

Conversation

ulixius9 commented Oct 27, 2026 • edited by greptile-apps Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Describe your changes:

Type of change:

High-level design:

Tests:

Use cases covered

Unit tests

Backend integration tests

Ingestion integration tests

Playwright (UI) tests

Manual testing performed

UI screen recording:

Checklist:

Confidence Score: 5/5

Findings

Summary

Diagram

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Oct 27, 2026

❌ PR checklist incomplete

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Oct 27, 2026

✅ Generated Sources Auto-Updated

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Oct 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Jest test Coverage

UI tests summary

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Oct 28, 2026

🚦 Removed from the merge queue — merge_conflict (2026-10-28T13:13:39Z)

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

gitar-bot Bot commented Oct 28, 2026

Uh oh!

sonarqubecloud Bot commented Oct 28, 2026

Quality Gate passed for 'open-metadata-ui'

Uh oh!

sonarqubecloud Bot commented Oct 28, 2026

Quality Gate passed for 'open-metadata-ingestion'

Uh oh!

Uh oh!

github-actions Bot commented Oct 28, 2026

🚦 Removed from the merge queue — failed_checks (2026-10-28T18:14:08Z)

Uh oh!

Uh oh!

github-actions Bot commented Oct 29, 2026

🚦 Removed from the merge queue — failed_checks (2026-10-29T05:41:45Z)

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ulixius9 commented Oct 27, 2026 •

edited by greptile-apps Bot

Loading

github-actions Bot commented Oct 27, 2026 •

edited

Loading

🚦 Removed from the merge queue — `merge_conflict` (2026-10-28T13:13:39Z)

🚦 Removed from the merge queue — `failed_checks` (2026-10-28T18:14:08Z)

🚦 Removed from the merge queue — `failed_checks` (2026-10-29T05:41:45Z)