fix(dq): allow attaching test cases to nested columns by TeddyCr · Pull Request #36385 · open-metadata/OpenMetadata

TeddyCr · 2026-10-27T22:27:10Z

Description

Fixes #36381 Backend slice of #36278: the API rejected column-level test cases on nested columns (<#E::table::…::columns::address.city>) with Invalid column name, because every test-case resolution site matched the entityLink's array field against top-level Table.columns only — while the storage format itself (nested Column.fullyQualifiedName dotted paths, FullyQualifiedName.getTableFQN, the UI's FqnPart.NestedColumn link writer) is already nested-aware.

What changed

ColumnUtil.resolveColumnByPath / validateColumnPath (new): resolve a dotted path by walking Column.children. FQN-split so a column literally named a.b arriving as "a.b" stays one segment; case-exact match first, case-insensitive fallback; a dotted literal column name arriving unquoted (legacy links) still resolves as a whole name when the segment walk fails; all keeps its meaning.
TestCaseRepository.validateColumnTestCase and validateDimensionColumns route through the new validation, so nested links attach via the API and a truly missing nested path still returns 400.
Tag inheritance (fetchAndSetTestSuiteTags bulk site + inheritTags) resolves nested columns, so tags on nested sub-columns now surface on their test cases.
CSV round-trip (convertFQNToEntityLink): exports the full quoted remaining FQN path (address.city, per-segment quoteName) instead of truncating nested FQNs to their first column segment.
No schema change, no migration (entityLink is an opaque string on the wire).
Decision record: ADR:2026-10-27-nested-column-dq-tests-address-nested-paths — segment-walk representation + per-dialect executability boundary.

Scope

Executing a test against a nested column is per-dialect and lands separately: ingestion (#36382: BigQuery/Databricks/Unity Catalog/Athena), Snowflake (#36383, needs structured-type ingestion first), UI picker (#36384).

How Has This Been Tested?

New ColumnUtilTest (11 tests): nested walk, multi-level nesting, case-insensitive + exact-case preference, quoted/unquoted dotted-name disambiguation, missing segments, all, and the 400 message. All green locally.
Regression: TestCaseRepositoryTest, TestSuiteRepositoryTest, ColumnRepositoryValidationTest — all green locally.
TestCaseResourceIT passed locally on MySQL + Elasticsearch: 360 passed, 16 skipped, no failures. The nested-column test now verifies tag inheritance through GET/list and programmatic result submission, history, and latest-result retrieval. The focused method also passed independently.
mvn spotless:apply clean; .github/java-tests/impact-map.json check passes.

Checklist

Links to issues: fixes Backend: allow attaching DQ test cases to nested columns (validate + tag inheritance) #36381 (sub-issue of epic Ability to attach test cases to deeply nested columns #36278)
High-level design: ADR included; nested-path representation + per-dialect executability documented
Unit tests for validation and resolution; ITs for the API attach path
No DB/schema migration needed
No UI change in this PR

🤖 Generated with Claude Code

Summary by Gitar

PII masking:
- Masks nested column profiles and test cases through PII tags on parent structs
- Fails closed for profiles whose column no longer resolves
Type conformance:
- Applies the supportedDataTypes check to nested column types
Ingestion runner:
- Reports nested-column tests as unsupported instead of crashing
Dimension columns:
- Keeps dimensionColumns top-level, since the runner only groups by top-level columns
Tag inheritance:
- Resolves nested columns so sub-column tags surface on their test cases
CSV round-trip:
- Exports full nested column paths instead of truncating to the first segment

_{This will update automatically on new commits.}

Confidence Score: 5/5

The PR appears safe to merge; the previously reported test failure is fixed.

Summary

The PR lets the API attach test cases to nested columns.

Test-case links can point to nested columns and keep their full paths.
PII tags on nested columns also mask their profiles and test cases.
The runner reports nested-column tests it cannot execute.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A[Column test case link] --> B[Resolve quoted column path]
  B --> C[Walk column children]
  C --> D[Validate column and supported type]
  C --> E[Read column tags]
  E --> F[Mask sensitive profiles and test cases]
  D --> G[Save test case]

_{Reviews (12) · Last reviewed commit: "fix: address automated review findings (..." · Reviewed by Greptile}

Follow-up integration verification

The tested worktree changes are now committed as 71da0c1370b01864502e615a2c08769a0766ffaf (no source changes after the run began). The planner captured the previous HEAD while the commit hooks were finishing, hence the generated report's uncommitted-changes notation. This run covers the follow-up test-only delta from e4e0fa01ce7e, not all production changes in the PR.

Local Java test run: PASSED

Commit: e4e0fa01ce7e (base e4e0fa01ce7e754d5a3b53b5a1fc6dd67378d85f) — uncommitted changes were present
Finished: 2026-10-28 18:12 UTC · 5.0 min

Step	Scope	Classes	Tests	Failed	Skipped	Min	Result
integration	mysql-elasticsearch · parallel	1	376	0	16	5.0	passed

Full local test details

Self-review: 71da0c1370b01864502e615a2c08769a0766ffaf, follow-up test-only delta reviewed against REVIEW.md; no substantive findings. Spotless and commit/push hooks passed.

Test case column validation, tag inheritance and the CSV FQN round-trip all resolved the entityLink's column against top-level Table.columns only, so nested paths like <#E::table::...::columns::address.city> were rejected at creation and silently lost tags. Resolve the column by walking Column.children with a quoted-aware dotted path (case-exact first, case-insensitive fallback; a dotted literal column name arriving unquoted still resolves). Route validateColumnTestCase, dimensionColumns and both tag-resolution sites through it, and serialize the full remaining FQN path on CSV export instead of truncating to the first column segment. Executing against the nested column remains per-dialect and is scoped to PRs 2/2b (ingestion). ADR:2026-10-27-nested-column-dq-tests-address-nested-paths Fixes open-metadata#36381

Copilot