Skip to content

Antalya 26.8: Added support for iceberg v3 unknown data type - #2537

Merged
zvonand merged 3 commits into
antalya-26.8from
feature/antalya-26.8/pr-2363
Oct 10, 2026
Merged

zvonand merged 3 commits into
antalya-26.8from
feature/antalya-26.8/pr-2363

Conversation

@zvonand

@zvonand zvonand commented Oct 10, 2026

Copy link
Copy Markdown
Member

Changelog category (leave one):

  • New Feature

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Support for iceberg v3 unknown datatype which is maps to Nullable(Nothing) in Clickhouse. Read and write path(Parquet) (#2363 by @subkanthi).

CI/CD Options

Exclude tests:

  • Fast test
  • Integration Tests
  • Stateless tests
  • Stateful tests
  • Performance tests
  • All with ASAN
  • All with TSAN
  • All with MSAN
  • All with UBSAN
  • All with Coverage
  • All with Aarch64
  • All Regression
  • Disable CI Cache

Regression jobs to run:

  • Fast suites (mostly <1h)
  • Aggregate Functions (2h)
  • Alter (1.5h)
  • Benchmark (30m)
  • ClickHouse Keeper (1h)
  • Iceberg (2h)
  • LDAP (1h)
  • Parquet (1.5h)
  • RBAC (1.5h)
  • SSL Server (1h)
  • S3 (2h)
  • S3 Export (2h)
  • Swarms (30m)
  • Tiered Storage (2h)

Cherry-picked from #2363.

…next commit)

---
Original cherry-pick message follows:

Merge pull request #2363 from Altinity/iceberg_unknown_data_type

Antalya 26.6: Added support for iceberg v3 unknown data type
# Conflicts:
#	src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergWrites.cpp
#	src/Storages/ObjectStorage/DataLakes/Iceberg/MultipleFileWriter.cpp
#	src/Storages/ObjectStorage/DataLakes/Iceberg/tests/gtest_iceberg_metadata_generator.cpp
#	src/Storages/ObjectStorage/StorageObjectStorageSource.cpp
#	tests/integration/test_storage_iceberg_with_spark/test_writes_field_ids_spark_read.py
Kept antalya-26.8 versions of extendSchemaForPartitions and
test_writes_field_ids_spark_read.py, which already contain the source PR's content.
Dropped bundled-in tests from other PRs in gtest_iceberg_metadata_generator.cpp.
Dropped the setStorageColumnEncoding context line in MultipleFileWriter: on antalya-26.8
the column mapper is already initialized via createColumnMapperFromFields.
Adapted the tuple-subcolumn extraction in StorageObjectStorageSource to iterate
requested_columns_copy, since antalya-26.8 has no row-lineage columns_to_extract list.
@zvonand zvonand added releasy Created/managed by RelEasy antalya-26.8 Session label (releasy session config) forwardport This is a frontport of code that existed in previous Antalya versions ai-resolved Port conflict auto-resolved by Claude labels Oct 10, 2026
@zvonand zvonand mentioned this pull request Oct 10, 2026
52 tasks done
@github-actions

github-actions Bot commented Oct 10, 2026 •

Copy link
Copy Markdown

Workflow [PR], commit [c299cbd]

…ya-26.8/pr-2363

# Conflicts:
#	src/Storages/ObjectStorage/DataLakes/Iceberg/tests/gtest_iceberg_metadata_generator.cpp
@zvonand

zvonand commented Oct 10, 2026

Copy link
Copy Markdown
Member Author

@blau-ai

@blau-ai

blau-ai commented Oct 10, 2026

Copy link
Copy Markdown
Collaborator

CI triage

Verdict: 0 of the red checks are caused by this PR. Everything failing is infra, a missing CI baseline, or known-environmental swarm flakiness. This PR is a self-contained Iceberg data-lake read-path change, and none of the failing areas (container CVE scans, parser-memory baseline, swarm cluster/resilience tests) are reachable from the code it touches.

Head SHA checked: c299cbd9eb8 · base: antalya-26.8

Check Class PR-caused?
Grype Scan keeper (4 high/critical) image CVE scan (infra) No
Grype Scan server-alpine (1 high/critical) image CVE scan (infra) No
Parser memory check (1/2) missing master baseline artifact No
Regression swarms — aarch64 (3/1535) environmental/flaky No
Regression swarms — release (3/1535) environmental/flaky No
SQLLogic test / PR root still RUNNING at report time n/a

Why the diff can't be responsible

The change only affects the data-lake / object-storage read path:

  • InterpreterCreateQuery.cpp — adds a single && !storage.isDataLake() guard to checkForUnsupportedColumns, so non-data-lake tables behave exactly as before.
  • StorageObjectStorageSource.cpp — tuple-element / nested-subcolumn handling, only inside the object-storage reader.
  • Everything else is under Storages/ObjectStorage/DataLakes/Iceberg/** (+ its gtests and two Iceberg integration tests).

None of this touches cluster discovery, node-failure resilience, the SQL parser, or Docker image contents.

Per-failure detail

Grype Scan (keeper + server-alpine) — these scan the built container images for OS/dependency CVEs (keeper: 4 high/critical; server-alpine: 1). They flag packages in the base image, not ClickHouse source, and fail on essentially every PR on this branch. Not actionable from this PR; needs a base-image/dependency bump at the branch level (separate from this feature).

Parser memory check — reported ERROR, Failures: 1/2. The failing sub-step is Resolve master binary with info "No master clickhouse-examples artifact was found". This is a CI-baseline problem (the master comparison binary wasn't available), not a parser regression. Safe to re-run; if it persists it's a pipeline/artifact issue, not code.

Regression swarms (aarch64 + release) — both arches report 13 features (10 ok, 3 failed), 1535 scenarios (1518 ok, 3 failed), and the same 3 scenarios fail on both:

  1. cluster discovery / check cluster discovery with wrong cluster name
  2. node failure / cpu overload
  3. node failure / cpu overload all swarm nodes

All three fail on Code: 279. DB::Exception: Cannot connect to any replica for query execution. (ALL_CONNECTION_TRIES_FAILED). These are swarm cluster-discovery / node-resilience scenarios (two of them are deliberate CPU-overload stress tests) failing on connectivity assertions — classic resource-contention/environmental flakiness, and they're identical across both architectures, which points to the test environment rather than the binary. Recent MasterCI runs on the base branch are also red, consistent with these being pre-existing on the branch. Suggested next step: re-run the swarms suite; no code change in this PR addresses or affects it.

Bottom line

CI is functionally green for this PR — the real test gates (builds, stateless, integration, stress, AST fuzzer, BuzzHouse, compat/install) are all OK. The remaining red is infra/flaky noise that exists independently of this change. No fix is needed on the PR itself; a re-run of the parser-memory and swarms jobs should clear those, and the Grype/CVE findings belong to the branch's base image rather than this feature.

🤖 automated triage by @blau-ai · evidence: praktika result_pr.json, swarms job logs, PR diff

@zvonand
zvonand merged commit 7fe3860 into antalya-26.8 Oct 10, 2026
312 of 323 checks passed
@zvonand zvonand added the port-antalya PRs to be ported to all new Antalya releases label Oct 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ai-resolved Port conflict auto-resolved by Claude antalya-26.8 Session label (releasy session config) forwardport This is a frontport of code that existed in previous Antalya versions port-antalya PRs to be ported to all new Antalya releases releasy Created/managed by RelEasy

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants