[draft] Master catalog spi 11 hive#65474
Open
morningman wants to merge 320 commits into
Open
Conversation
…ive + FORMAT_TEXT vs CSV vs JSON (dormant) Fix two read-correctness bugs in the hive connector's file-format detection by reproducing legacy HMSExternalTable.getFileFormatType faithfully. Dormant today (hms is not in SPI_READY_TYPES; only the hive connector emits the text-family tokens, so the fe-core vocabulary split below is hive-only in effect). Bug 1 (JSON read as CSV): HiveFileFormat.detect was input-format-first, and a standard hive JSON table has inputFormat=TextInputFormat (its JSON-ness lives only in the serde), so it resolved to text and was read by the CSV reader. Bug 2 (default hive-text read as CSV): the connector collapsed LazySimpleSerDe / MultiDelimitSerDe / OpenCSVSerde all to one "text" token, and fe-core mapFileFormatType mapped "text" -> FORMAT_CSV_PLAIN. But BE dispatches FORMAT_TEXT (LazySimpleSerDe/MultiDelimit) to a DISTINCT reader from FORMAT_CSV_PLAIN (OpenCSV) -- the text reader honors hive collection/map delimiters, \N nulls and hive escaping. So the DEFAULT hive text serde was mis-read for nested/complex columns, hive nulls and escapes. - HiveFileFormat: two-stage, serde-authoritative for the text family (inputFormat -> parquet/orc/text-file family by substring; text family -> serde decides json/csv/hive-text by EXACT FQCN). Adds a CSV member. Reproduces the OpenX-JSON read_hive_json_in_one_column branch (one CSV column when the first column is a string; fail-loud otherwise). Fails loud on an unrecognized inputFormat/serde (legacy parity, user sign-off) instead of silently degrading to FORMAT_JNI. - fe-core PluginDrivenScanNode.mapFileFormatType: split the fused case so "text" -> FORMAT_TEXT and "csv" -> FORMAT_CSV_PLAIN. Neutral, pre-existing tokens; no if(engine)/instanceof (iron-rule clean); hive is their sole emitter. Must land atomically with the connector's distinct csv emission or OpenCSV (correct-by-accident today) would regress. - HiveScanPlanProvider: pass the session read_hive_json_in_one_column flag + handle firstColumnIsString into detect; widen the text-props gate to TEXT||CSV||JSON; stop the dead per-partition detect (leave null; the whole node reads one table-level format) so a partition's unusual serde can't fail-loud. - HiveTableHandle + getTableHandle: carry firstColumnIsString (STRING-only, legacy parity), precomputed from the already-loaded metastore table (no extra fetch). - HiveTextProperties: add the legacy hive-2 JsonSerDe + the serde2 MultiDelimit FQCN so the emitted text params match the serdes the detector recognizes. read_hive_json_in_one_column reproduced (user sign-off). Test: HiveFileFormatTest rewritten as the legacy serde truth table (12 cases: text/csv/json split, the JSON-as-CSV regression, OpenX one-column on/off, fail-loud on unknown). Full hive+hms suites green (153/150, no regression); fe-core compiles; checkstyle 0; import gate clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx
…stPartitions Record the file-format read-correctness fix (c320df2) — serde-authoritative detection fixing JSON-read-as-CSV + the FORMAT_TEXT/CSV/JSON distinction + OpenX one-column + fail-loud, plus the atomic fe-core mapFileFormatType text/csv split — and re-point the next substep to listPartitions. Recon: wf_470a2e51-373. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx
…(no silent 32767 cap)
ThriftHmsClient.listPartitionNames mapped a non-positive maxParts to Short.MAX_VALUE
(32767) and passed it to HMS get_partition_names, hard-capping the result at 32767
partitions. Legacy ThriftHMSCachedClient passed the Config.max_hive_list_partition_num
default of -1 straight through, which HMS treats as unbounded ("all"). So a table with
>32767 partitions was silently truncated.
This already defeated two existing dormant callers that pass -1 intending "all":
HiveWritePlanProvider.buildExistingPartitions and HudiConnectorMetadata (the latter
carries an explicit "no silent partition truncation" comment). The scan/prune callers
pass 100000, which narrows to a negative short (-31072) and is therefore already
unbounded -- so the connector was internally inconsistent (full scan universe vs capped
listing).
Fix: a non-positive maxParts now maps to (short) -1 (unbounded), extracted into a
testable static toThriftMaxParts helper. Positive values pass through, narrowing to
short -- a value above Short.MAX_VALUE narrows to a negative short (unbounded), which
preserves the pre-existing behavior of the 100000-cap callers. No behavior change for
any positive caller; the -1 callers now get the "all" listing they always intended.
Test: ThriftHmsClientMaxPartsTest pins the mapping (-1/0 -> unbounded, positive
pass-through, >Short.MAX_VALUE -> negative/unbounded) without a live metastore.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx
…mant) Implements the ConnectorTableOps partition-listing SPI on HiveConnectorMetadata so a flipped hive table's fe-core partition view (PluginDrivenMvccExternalTable. materializeLatest / getNameToPartitionItems), SHOW PARTITIONS names branch and MetadataGenerator resolve real partitions instead of degrading to empty. - listPartitionNames / listPartitions share a collectPartitionNames helper that lists the metastore's rendered partition names (get_partition_names, -1 = all). An unpartitioned table lists nothing without touching the metastore (mirrors paimon's collectPartitions guard). - lastModifiedMillis (and rowCount/sizeBytes/fileCount) are left UNKNOWN(-1) by design (user sign-off): legacy's hot partition-pruning path listed names ONLY -- HiveExternalMetaCache.loadPartitionValues never did the per-partition get_partitions_by_names round-trip; the per-partition transient_lastDdlTime was fetched only at MTMV-refresh time. Filling it in listPartitions (run on every partitioned-hive query) would regress the hot path. Per-partition MTMV freshness is rewired connector-side in the later MVCC/MTMV step. maxcompute precedent also emits -1; hive declares no SUPPORTS_PARTITION_STATS so the -1 stats never surface (SHOW PARTITIONS takes the names-only branch). - The filter is ignored (legacy materialized the full set and pruned FE-side). - Partition values are decoded via HiveWriteUtils.toPartitionValues (byte-faithful port of legacy HiveUtil.toPartitionValues, incl. path unescaping) and keyed by remote partition-column name, which is how PluginDrivenExternalTable.getNameToPartitionItems reads them back. Dormant: hive is not in SPI_READY_TYPES, so this is unreachable online today. Verified: 3-lens adversarial review (legacy-parity / flip-consumer trace / code-edge) converged on a single defect -- the -1 partition cap -- fixed in the preceding ThriftHmsClient commit; value-decoding byte-parity, iron rules, and no-NPE-from-(-1) confirmed. fe-connector-hive 160 tests + 7 new partition tests green, checkstyle + import gate clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx
…d fix; next = getCapabilities Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx
…ta-preload (dormant) Declares the connector-wide capabilities the flipped hms catalog needs, each a faithful port of a legacy HMSExternalTable/HMS admission, alongside the already-declared SUPPORTS_VIEW: - SUPPORTS_COLUMN_AUTO_ANALYZE: legacy StatisticsUtil.supportAutoAnalyze admitted HMS tables of dlaType HIVE (and ICEBERG) into the background per-column FULL auto-analyze; the capability replaces that instanceof-HMSExternalTable arm. FULL is forced (sample analyze throws for external SQL-driven tables), and admitting hive needs no new SPI (FULL runs a SQL scan over the already-built read path). - SUPPORTS_METADATA_PRELOAD: legacy HMSExternalTable.supportsExternalMetadataPreload() returned true; replaces the legacy engine-name "jdbc" gate. Opt-in via enable_preload_external_metadata (default off), pure lock-latency optimization, no correctness effect. Gates only supportsExternalMetadataPreload -- plugin tables keep the false default for supportsLatestSnapshotPreload, so hive does not snapshot-preload. Deliberately withheld (verified sound): - SUPPORTS_MVCC_SNAPSHOT: the mixed hms catalog DOES need it (iceberg/hudi-on-HMS are MvccTable; GSON single-row maps HMSExternalTable -> PluginDrivenMvccExternalTable and buildTableInternal selects the Mvcc subclass from this catalog-level capability), but it lands WITH the MVCC/MTMV machinery (freshness-aware getTableSnapshot + hive empty-pin + iceberg sibling delegation), not as a bare flag. This corrects the earlier "hive is non-MVCC" note -- the deferral is timing, not permanent. - SUPPORTS_SHOW_CREATE_DDL: needs the connector to emit the table location + a generic-vs-hive-specific rendering decision (own substep). - SUPPORTS_PASSTHROUGH_QUERY / SUPPORTS_PARTITION_STATS: no query() TVF; names-only SHOW PARTITIONS. - SUPPORTS_TOPN_LAZY_MATERIALIZE: per-table marker (already emitted, orc/parquet). - SUPPORTS_NESTED_COLUMN_PRUNE: deferred like MVCC -- legacy parquet/orc pruned nested columns, but the connector must first carry nested field-ids or BE reads NULL leaves. Dormant: hms not in SPI_READY_TYPES, so getCapabilities is never consulted for hive tables today. Verified: 3-lens adversarial review confirmed the declared caps are faithful legacy ports with preconditions met and every deferral sound; residual = connector-wide auto-analyze over-admits hudi-on-HMS at the flip (legacy excluded hudi), to be gated per-handle or accepted at the delegation substep. fe-connector-hive 164 tests (+4 new capability tests) green, checkstyle + import gate clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx
…preload); MVCC/SHOW_CREATE deferred not forbidden; next = HiveTableFormatDetector Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx
…il-loud + view short-circuit (dormant) Closes the last read-side format-detection gaps vs legacy HMSExternalTable: - LZO text InputFormats: HiveTableFormatDetector.detect now classifies the 3 LZO-text InputFormats (com.hadoop.compression.lzo.LzoTextInputFormat, com.hadoop.mapreduce. LzoTextInputFormat, com.hadoop.mapred.DeprecatedLzoTextInputFormat) as HIVE. The connector's SUPPORTED_HIVE_INPUT_FORMATS now exactly matches legacy SUPPORTED_HIVE_FILE_FORMATS (7 entries); without them an LZO-text table would fail the now-fail-loud format check even though legacy read it as hive text (FORMAT_TEXT). - Fail-loud: getTableHandle now throws DorisConnectorException on a null / unrecognized input format (UNKNOWN), replacing the old silent UNKNOWN-type handle. This mirrors legacy HMSExternalTable.supportedHiveTable() throwing NotSupportedException, at the same lazy per-table-access point (getTableHandle is the resolveConnectorTableHandle analogue of makeSureInitialized; SHOW TABLES uses listTableNames and is unaffected). - View short-circuit: the fail-loud is skipped for a view (isViewTable = view text present), matching legacy which returned true for a view before the format check. A view has no data files (usually null input format) and is served by the view SPI, so its handle keeps the UNKNOWN type (never scanned) rather than being rejected. Dormant: hms not in SPI_READY_TYPES. Verified: 3-lens adversarial review (classification- parity / fail-loud blast-radius / exception-catch-parity), verdict ship — format sets match legacy exactly, no bulk-listing abort (throw is lazy per-access), view predicate + detect ordering equivalent. Two residuals owned by the flip (not this step, pre-existing cross- connector SPI properties already exhibited by iceberg): existence-via-getTableHandle now fails loud for an unsupported non-view table (CREATE-IF-NOT-EXISTS/DROP), and a query on an unsupported table surfaces DorisConnectorException which fe-core ConnectProcessor classifies as a generic error rather than ERR_NOT_SUPPORTED_YET -- both fixable only in fe-core read-path handling at the flip. fe-connector-hive 171 tests (+7 new) green, checkstyle + import gate clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx
…de §4.2 complete); next = column-stats SPI Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx
…n col stats (dormant) Preserves the legacy HMSExternalTable.getColumnStatistic / getHiveColumnStats / setStatData query-planner fast path (no-scan HMS/spark column stats, consulted by ColumnStatisticsCacheLoader on a stats-cache miss) via a new SPI, following the established "connector returns raw facts, fe-core does the Doris-type math" split (D1=preserve). - fe-connector-api: new ConnectorColumnStatistics DTO (rowCount / ndv / numNulls / avgSizeBytes) + ConnectorStatisticsOps.getColumnStatistics(session, handle, columnName), a default returning empty (other connectors are unaffected and fall back to full ANALYZE). - fe-connector-hms: new neutral HmsColumnStatistics DTO; HmsClient gets a default getTableColumnStatistics (empty) so the test doubles need no change; ThriftHmsClient overrides it and convertColumnStatistics extracts ndv/numNulls/avgColLen per HMS ColumnStatisticsData variant -- a byte-faithful port of setStatData (avgColLen only for string; boolean/binary/timestamp -> ndv=0/numNulls=0, exactly as legacy's unhandled path). - fe-connector-hive: HiveConnectorMetadata.getColumnStatistics serves plain-hive tables only (iceberg-on-HMS goes to the sibling; hudi had no fast path), requires a positive numRows table param as the data-size basis (NO spark fallback here, unlike the table-size branch), fetches the HMS col stats and returns the raw facts. - fe-core: PluginDrivenExternalTable.getColumnStatistic override calls the SPI and toColumnStatistic() does the Doris-type-dependent size math the connector cannot (must not import fe-type): a string column -> round(avgColLen * count), every other type -> count * slot-width; avgSizeByte = dataSize / count; min/max = builder defaults (legacy's NEGATIVE/POSITIVE_INFINITY). Extracted static for direct unit testing. Scope: this is the query-planning column-stat fast path. The partition-level col stats (getHivePartitionColumnStats -> HMSAnalysisTask metadata ANALYZE path) and the iceberg connector's getIcebergColumnStats port are separate follow-ups. Dormant: hms not in SPI_READY_TYPES. Verified: 3-lens adversarial review (math byte-parity / guards+flow / architecture), verdict ship, zero confirmed defects -- field-by-field parity for every HMS variant, guards diverge only strictly-safer toward the ANALYZE fallback, no fe-core imports leak into the connector, SPI defaults keep other connectors inert. New tests: hms 4 (variant extraction) + hive 5 (fast-path gates) + fe-core 4 (Doris math); full fe-connector-hms 56 + fe-connector-hive 176 green, checkstyle + import gate clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx
…§4.3 MVCC/sys-table + freshness Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx
…tition snapshots (dormant) Fix the stale-MV root cause for hive base tables served by the connector SPI, and declare SUPPORTS_MVCC_SNAPSHOT for the flipped hms catalog. PluginDrivenMvccExternalTable.getTableSnapshot hardcoded MTMVSnapshotIdSnapshot; for a hive empty pin the snapshot id is a constant -1, so an MV over a hive base table would compare equal forever and never refresh. getPartitionSnapshot likewise read the pin's -1 (hive listPartitions is names-only on the scan hot path). Fix = make the table/partition MTMV snapshots freshness-kind-aware, driven by a per-handle signal the connector puts ON THE QUERY-BEGIN PIN (no engine branch in fe-core): - fe-connector-api: ConnectorMvccSnapshot gains lastModifiedFreshness (default false); ConnectorMetadata gains getTableFreshness / getPartitionFreshnessMillis (default empty) + a neutral ConnectorTableFreshness DTO. All consulted only on the MTMV refresh path. - fe-core PluginDrivenMvccExternalTable: getTableSnapshot/getPartitionSnapshot read the pin flag; last-modified (hive) -> MTMVMaxTimestampSnapshot / MTMVTimestampSnapshot from the on-demand SPI (byte-parity legacy HiveDlaTable). A snapshot-id connector (paimon/iceberg) leaves the flag false and takes the EXACT pre-change path off the pin it already holds -- zero extra metadata round-trips (guarded by two verify(never) regression tests). A vanished partition (on-demand empty) raises the legacy "can not find partition". - fe-connector-hive: beginQuerySnapshot returns the empty(-1) pin flagged last-modified; getTableFreshness (unpartitioned = table transient_lastDdlTime, no round-trip; partitioned = max over get_partitions_by_names, named by the owning partition) and getPartitionFreshnessMillis port legacy transient_lastDdlTime*1000 (parse stays connector-side). HiveConnector declares SUPPORTS_MVCC_SNAPSHOT. Scan hot path (listPartitions) stays names-only; freshness is fetched only at MTMV refresh (two-path parity with legacy). Dormant for hive (not yet in SPI_READY_TYPES); paimon/iceberg byte-and-cost unchanged. fe-connector-hive 180 (9 new freshness + capabilities updated) + fe-core PluginDrivenMvccExternalTableTest 53 (7 new) + PluginDriven neighbors green; checkstyle 0; connector-imports gate net. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx
…deferred to flip; next = §4.4 sibling delegation §4.3 landed (3d78467): last-modified table/partition MTMV snapshots gated by a pin flag (so paimon/iceberg — live via P5/P6, NOT dormant — pay zero extra round-trips), + SUPPORTS_MVCC_SNAPSHOT. Records the review-caught regression fix, the deferred getNewestUpdateVersionOrTime dictionary probe + sys-tables ($partitions TVF, user-signed deferral to the flip), and the accepted flip-boundary parity nuances. Next = §4.4 cross-plugin sibling SPI + gateway delegation (D3 hard precondition). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx
Add ConnectorContext.createSiblingConnector(type, props) so a heterogeneous "gateway" connector (a flipped Hive-metastore catalog) can obtain a sibling connector of another type -- built in that type's OWN child-first plugin classloader and sharing this catalog's context -- to delegate iceberg-on-HMS tables to the iceberg connector without co-packaging (a 2nd AWS SDK in a distinct loader poisons S3 JVM-wide). - SPI default returns null, matching every other ConnectorContext default seam; non-gateway connectors are unaffected. - fe-core DefaultConnectorContext override forwards to ConnectorFactory.createConnector(type, props, this): passes `this` so the sibling reuses the catalog's id/auth/storage, and stays connector-agnostic (no property parsing -- the caller synthesizes the sibling props). - iceberg + paimon TcclPinningConnectorContext decorators delegate the new method to the raw context, keeping their exhaustive-pass-through contract true; the sibling applies its own TCCL/auth pin, so it must receive the unwrapped context (avoids double-pinning). Dormant: nothing calls createSiblingConnector yet (hms is not in SPI_READY_TYPES); the hive gateway substep wires it. First step of the sibling-connector delegation build-out. Tests: SPI default returns null; DefaultConnectorContext builds via the factory and passes props + this-context through (+ null on no-match / uninitialized manager); both decorators delegate the new method to the raw context. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx
…4 S0-S6 decomposition + wrapper-handle correction Record the sibling-connector delegation decomposition (S0-S6) from recon wf_da155752-ebb: the S0 fe-core seam is DONE (45dcca3); S1 props-synthesis, S2 gateway embed (one sibling per gateway + close-forward), S3 metadata instanceof-guard-and-forward, S4 getTableHandle iceberg divert, S5 scan-provider override, S6 name-based residuals remain. Key correction: the design's gateway-owned wrapper handle is dead (the engine threads one handle object from getScanPlanProvider selection straight into IcebergScanPlanProvider's (IcebergTableHandle) cast, no unwrap seam) -> getTableHandle returns the raw foreign iceberg handle and gateway consumers discriminate by instanceof HiveTableHandle (own type), never casting the foreign handle. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx
Add IcebergSiblingProperties.synthesize(catalogProps): the flipped HMS gateway
will hand this map to createSiblingConnector("iceberg", ...) to build the
embedded iceberg-on-HMS connector. It copies the gateway catalog's whole
property map verbatim and injects the literal iceberg.catalog.type=hms.
- Copy-all (not a hand-picked subset) is the robust choice: it cannot drop a
connectivity key (the `uri` short form, HDFS-HA dfs.*, an S3 endpoint, a
kerberos variant). It is also the exact map shape the iceberg connector
already handles for a native `type=iceberg, iceberg.catalog.type=hms` catalog
(fe-core builds that from the full catalog property map), and its create()
path does no property validation, so hive-only keys are ignored, not misused.
- The flavor key/value are hardcoded literals: iceberg's
IcebergConnectorProperties constants are child-first (invisible to the hive
loader).
Per-catalog (the sibling is table-agnostic), pure + static (unit-testable with
no connector). Dormant: no caller yet (the gateway-embed substep wires it).
Tests: injects hms flavor; carries metastore/storage/HA/kerberos/conf keys
(incl. the uri short form); does not mutate the input; overrides any
pre-existing flavor.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx
…ified); next = S2 gateway embed S1 (d971410) synthesizes the iceberg sibling's property map by copying the gateway catalog's full property map + injecting iceberg.catalog.type=hms. The copy-all-vs-subset choice was adversarially verified safe: it is the same map shape a native iceberg-on-hms catalog already feeds the connector, and hive-only delta keys are ignored (not misused) since create() does no validation. Next = S2: HiveConnector holds + lazily builds one sibling per gateway (volatile double-check) and forwards close(), held only as the parent-first Connector. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx
…ector (dormant)
S2 of the hms-cutover sibling-delegation build-out. A flipped hms catalog will be
served by the hive plugin acting as a gateway; its iceberg-on-HMS tables are
delegated to a sibling iceberg connector built in the iceberg plugin's own
child-first classloader (never co-packaged into the hive zip — a second AWS SDK
would poison S3 JVM-wide). This step adds only the holder + lazy build +
close-forward; there is no consumer yet, so it is fully dormant until hms enters
SPI_READY_TYPES.
- HiveConnector.getOrCreateIcebergSibling(): volatile double-checked lazy build
via context.createSiblingConnector("iceberg", IcebergSiblingProperties
.synthesize(properties)), mirroring the existing getOrCreateClient() idiom. One
sibling per gateway (IcebergConnector holds per-catalog caches shared across its
tables, which a per-op sibling would fragment). Held ONLY as the parent-first
Connector interface, never cast (the concrete type is invisible to the hive
loader -> a cast would CCE across the loader split). Fails loud
(DorisConnectorException, catalog-scoped message) when no iceberg provider is
available, and the failure is NOT memoized so a later-available plugin recovers.
- close() forwards to the sibling then nulls it (the engine closes only a catalog's
primary connector; the gateway owns the sibling lifecycle). No-op when the sibling
was never built.
Green: fe-connector-hive 194 tests (6 new HiveConnectorSiblingTest) + checkstyle 0 +
connector import gate clean. Adversarial review (4 dimensions + verify): 0 confirmed
defects. hms confirmed absent from SPI_READY_TYPES -> dormancy holds.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx
…adata handle-guard-forward S2 committed at 119b4fc (dormant): HiveConnector.getOrCreateIcebergSibling lazy build + single-per-gateway + never-cast + fail-loud-not-memoized + close forward. Adversarial review wf_c17ad553-53a: 0 confirmed defects. Next step = S3 (HiveConnectorMetadata per-handle instanceof-HiveTableHandle guard-and-forward). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx
…berg sibling (dormant) S3 of the hms-cutover sibling-delegation build-out. Makes HiveConnectorMetadata's per-handle methods discriminate on the concrete handle type: a HiveTableHandle runs the existing hive logic byte-identically; a foreign (iceberg-on-HMS) handle is forwarded to the embedded iceberg sibling connector (built dormant in S2). The gateway NEVER casts the foreign handle — its concrete type lives in the iceberg plugin's child-first classloader, so a cast would CCE across the loader split. Wiring: HiveConnector.getMetadata passes this::getOrCreateIcebergSibling as a Supplier<Connector>; HiveConnectorMetadata.siblingMetadata(session) resolves it per forwarded call (parity with fe-core, which acquires a ConnectorMetadata per operation; the heavy catalog/caches live on the single sibling connector). A 3-arg constructor (hive-only, test) installs a fail-loud NO_ICEBERG_SIBLING supplier. Two guard sets (a code-grounded recon classified the full 67-method surface): - 13 methods hive already overrides (getTableSchema/getColumnHandles/getTable+Column Statistics/estimateDataSizeByListingFiles/applyFilter/listPartitions[Names]/ beginQuerySnapshot/getTable+PartitionFreshness/dropTable/truncateTable) get a pure prefix guard — the hive body is unchanged for a hive handle. - 8 methods hive did NOT override but iceberg does (schema-at-snapshot, getMvccPartitionView, resolveTimeTravel, applySnapshot, applyRewriteFileScope, applyTopnLazyMaterialization, listSupportedSysTables, getSysTableHandle) get NEW guard-and-forward overrides — without them a delegated iceberg table would get hive's SPI default (a SILENT wrong answer: empty MVCC/time-travel/sys-tables or an unthreaded snapshot pin), not a fail-loud. The hive branch reproduces the SPI default. beginQuerySnapshot diverts in lockstep with the two freshness methods (its isLastModifiedFreshness flag gates whether fe-core consults them). Handle-out methods (applyFilter/apply*/getSysTableHandle) return the sibling's handle UNMODIFIED — a rewrap would poison a downstream planScan cast. Excluded by design (recon-verified): getTableHandle (S4), buildTableDescriptor + all name-based view/db/createTable/identifier methods (S6), the inert applyProjection/ applyLimit/listPartitionValues (neither side overrides), and the ALTER-DDL suite + write validations (handle-based but write-cutover, separate substep). Dormant: "hms" is absent from SPI_READY_TYPES and getTableHandle (S4, not yet done) still returns a HiveTableHandle, so every guard is an inert no-op today and a pure-hive query never builds the sibling. Green: fe-connector-hive 197 tests (3 new HiveConnectorMetadataSiblingDelegationTest, including a completeness lock on the exact forwarded-method set) + checkstyle 0 + connector import gate clean. Adversarial review (4 dimensions + verify): 0 findings. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx
… = S4 getTableHandle iceberg divert S3 committed at b80c614 (dormant): HiveConnectorMetadata's 21 per-handle methods guard-and-forward a foreign iceberg handle to the embedded sibling (13 prefix guards on hive-overridden methods + 8 new gap-fill overrides for the silent iceberg-only subset), handle-out methods return the sibling handle unmodified. Recon wf_2676a99b-efa classified the full 67-method surface; adversarial review wf_e393f9ec-b68: 0 findings. Records the scope corrections (buildTableDescriptor -> S6; ALTER-DDL/write-validation deferred to the write cutover; applyProjection/applyLimit/listPartitionValues inert). Next = S4 (getTableHandle returns the raw foreign iceberg handle for ICEBERG tables, lighting up S3). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx
…sibling at getTableHandle (dormant) §4.4 S4 — the pivot that starts a foreign iceberg handle flowing out of the hive gateway. In HiveConnectorMetadata.getTableHandle, once the format detector resolves the HMS table to ICEBERG, return the embedded iceberg SIBLING connector's OWN table handle (the raw foreign iceberg handle) verbatim instead of a HiveTableHandle stamped ICEBERG. Iceberg's scan/metadata path unconditionally casts the handle to its concrete IcebergTableHandle, so a HiveTableHandle would CCE the moment iceberg cast it; the foreign handle is never cast here (its concrete type is invisible across the classloader split). This activates the S3 guard-and-forward overrides: every gateway consumer discriminates by `instanceof HiveTableHandle` (the gateway's own hive-loader type) and forwards any non-hive handle to the sibling. An empty from the sibling (table vanished / iceberg cannot load) passes through unchanged rather than fabricating a hive handle. HUDI is intentionally NOT diverted — hudi-on-HMS delegation is a later substep, so a hudi table stays on the hive-handle path (byte-unchanged). Plain-hive and view tables are untouched. Dormant until hms enters SPI_READY_TYPES: nothing calls getTableHandle for this connector today. New HiveConnectorMetadataTableHandleDivertTest pins the contract: iceberg returns the sibling's handle unmodified (not a HiveTableHandle), the empty pass-through is proven to be forwarded from the sibling, hive/hudi never consult the sibling, a missing table short-circuits to empty, and an iceberg table with no sibling configured fails loud. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx
…; next = S5 scan-provider gateway Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx
…bling scan provider (dormant) §4.4 S5 — the scan-side pair of the getTableHandle iceberg divert. HiveConnector overrides the per-table scan-provider seam Connector.getScanPlanProvider(handle): a hive handle (the gateway's own hive-loader type) runs the hive scan provider; any foreign handle (the raw iceberg handle the sibling's getTableHandle produced) is delegated to the embedded iceberg sibling's per-handle scan provider, passed through unmodified and never cast. Because the returned sibling provider is built in the iceberg plugin's classloader, PluginDrivenScanNode.onPluginClassLoader auto-pins the scan-thread TCCL to the iceberg loader for free (it keys off provider.getClass().getClassLoader()), so no pinning is needed here. A HUDI-stamped HiveTableHandle is still a HiveTableHandle, so it stays on the hive scan path (its delegation is a later substep). Dormant until hms enters SPI_READY_TYPES: nothing selects a scan provider for this connector today. New HiveConnectorScanProviderDivertTest pins the route: a foreign handle returns the sibling's provider (with the handle reaching the sibling unmodified and the sibling built once), a hive/hudi handle resolves the connector-level hive provider without building or consulting the sibling, and a foreign handle with no iceberg plugin fails loud naming the catalog. The hive/hudi cases stub the no-arg getScanPlanProvider() to isolate the routing from HmsClient construction (a real HmsClient needs a HiveConf, off the unit-test classpath; provider construction is covered by the scan-planning suites). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx
… next = S6 name-based divert + capability residuals Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx
…NO_DIVERT); read-spine S0-S6 complete; next = write delegation
A code-grounded recon (wf_6ac0e047-768: 4 dimension readers + a
completeness/correctness critic, all HEAD-verified) proves the §4.4 S6 sketch
("re-run detect(db,table) and forward the view family + buildTableDescriptor for
ICEBERG") is unnecessary or actively harmful, so S6 ships ZERO connector code:
- buildTableDescriptor: byte-identical output (the sibling is always hms flavor
-> same HIVE_TABLE + THiveTable; legacy iceberg-on-HMS also emitted HIVE_TABLE),
so a divert only adds a wasteful getTable(detect).
- view family (viewExists/getViewDefinition/dropView): legacy-parity for the only
reachable objects (iceberg/hudi TABLES return viewExists=false and stay on the
getTableHandle divert; an iceberg-on-HMS VIEW stores hive-dialect SQL in HMS view
text and is served as a hive view) AND undiscriminable name-only (detect returns
UNKNOWN for a view: table_type='iceberg-view' != 'ICEBERG').
- listViewNames stays un-overridden (a divert would double-list views); getTableComment
stays SPI-default "" (legacy getComment() returned "" unconditionally -> a divert
would surface a comment legacy never showed).
- db/catalog methods are format-agnostic.
=> The §4.4 read-delegation spine (S0-S6) is COMPLETE. Remaining per-table
divergences are all write/DDL residuals (dropDatabase-force purge, createTable
engine-routing, per-handle getWritePlanProvider/getProcedureOps) or connector-wide
capability residuals (SUPPORTS_COLUMN_AUTO_ANALYZE hudi over-admission already
documented; supportsLatestSnapshotPreload under-admission) plus the explicit
non-goal of first-class iceberg-on-HMS views — all recorded for the flip in the
new findings doc. Dormant until hms enters SPI_READY_TYPES.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx
…DL + write-validation to sibling (dormant)
§4.4 write-delegation W1 (dormant): extend the S3 handle-guard-forward on
HiveConnectorMetadata to the 16 remaining handle-carrying mutating/validation
methods so an iceberg-on-HMS table's ALTER-DDL and write-time validation route to
the embedded iceberg sibling, while plain-hive stays byte-identical.
- 14 ALTER-DDL mutators (renameTable + add/drop/rename/modify/reorder column,
create/drop branch & tag, add/drop/replace partition-field): foreign handle ->
siblingMetadata().<same>(...); hive handle reproduces the EXACT inherited
SPI-default throw ("... not supported"). Foreign handle is never cast.
- 2 write validators (validateRowLevelDmlMode / validateStaticPartitionColumns):
foreign handle -> sibling (iceberg's real write-mode / static-partition
rejections apply); hive handle returns SILENTLY (SPI-default no-op) -- a throw
there would newly reject legal plain-hive row-level DML / static-partition
INSERTs.
Zero new SPI (all 16 already carry a ConnectorTableHandle) and zero fe-core change
(ALTER call sites are already handle-based via PluginDrivenExternalCatalog and
getTableHandle already S4-diverts iceberg to the sibling). Dormant until "hms"
enters SPI_READY_TYPES. Per user sign-off, iceberg-on-HMS gains full native-iceberg
write/ALTER capability (an intended upgrade over legacy's read-only reject).
Tests: HiveConnectorMetadataSiblingDelegationTest +2 (foreign-handle completeness
lock over all 16 + hive-handle throw/no-op + never-consult-sibling). Full module
209 tests green, checkstyle 0, connector-import gate clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx
…on doc W1-W6 + capability upgrade signed; next = W2 - Add hms-write-delegation-decomposition-2026-07-08.md: authoritative W1-W6 plan from recon wf_f508ac0e-8ec (3 divert shapes; user-signed iceberg-on-HMS capability UPGRADE to full native-iceberg; e2e requirement; flip residuals incl. hudi-on-HMS write flip-blocker). - HANDOFF: W1 landed (e7a96f439e7); next = W2 (getWritePlanProvider(handle) overload + gateway divert + fe-core provider-fetch conversion). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx
…iverts iceberg-on-HMS to sibling (dormant) §4.4 write-delegation W2 (dormant): add a per-handle write-provider seam mirroring the shipped getScanPlanProvider(handle). - Connector: new default getWritePlanProvider(ConnectorTableHandle) delegating to the connector-level getWritePlanProvider() -> behavior-preserving for every single-format connector (jdbc/es/trino/max_compute/paimon/iceberg). - HiveConnector overrides it: a HiveTableHandle uses the hive write provider; a foreign (iceberg-on-HMS) handle is delegated to getOrCreateIcebergSibling() .getWritePlanProvider(handle), passed through UNMODIFIED and NEVER cast. HUDI keeps a HiveTableHandle -> stays on hive (its write delegation is a later step). - fe-core: the 4 provider-fetch sites now pass the resolved table handle (PhysicalPlanTranslator DELETE/MERGE + INSERT, PhysicalIcebergMergeSink, PluginDrivenExternalTable.fetchSyntheticWriteColumns) by hoisting the already-adjacent getTableHandle above the provider fetch. Connector-agnostic (a per-handle overload call), no if(format)/instanceof-HMSExternal branch. The write-admission methods (supportedWriteOperations/requires*) stay connector-level (W3); beginTransaction threading is W4; the write-path TCCL pin is W6. Dormant until "hms" enters SPI_READY_TYPES. Tests: new HiveConnectorWriteProvider DivertTest (4: foreign->sibling unmodified + build-once, hive/hudi->hive no sibling, fail-loud when plugin absent); 4 fe-core mock tests stub the per-handle overload (a plain Mockito mock does not run the interface default). fe-connector-hive 213 + fe-core touched suites green, checkstyle 0 (api/hive/fe-core), import gate clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx
…per-handle write-admission) W2 landed (84890fe48d3): getWritePlanProvider(handle) overload + HiveConnector divert + 4 fe-core provider-fetch conversions; adversarial review caught + fixed a blocker (4 fe-core mock tests must stub the per-handle overload). Next = W3: the 7 write-admission per-handle overloads (default-derived from getWritePlanProvider( handle)) + fe-core admission-site handle resolution. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx
…le listing After the hms flip, a plugin-driven hive catalog lists partition data files through HiveFileListingCache.listFromFileSystem, which was unconditionally non-recursive and never read the catalog property hive.recursive_directories. Legacy fe-core (HiveExternalMetaCache.getFileCache) defaulted it to "true" and recursed into sub-directories, so a table whose data lives in sub-directories now silently loses those rows (returns top-level only) instead of all of them. Fix (connector-local, zero fe-core / SPI change): parse the flag once in the cache ctor (default true) and bake it into the production lister; when set, listFromFileSystem descends into non-hidden sub-directories. Hidden dirs (_temporary / .hive-staging) and _/.-prefixed files are skipped at every level -- exact net parity with legacy's full-path containsHiddenPath filter, since the connector filters only the leaf FileEntry.name(), so descend-all would surface staging files legacy suppresses. The failure loop is factored into a recursive collectFiles helper that throws up to the SAME systemic-vs-local classifier, so a sub-directory failure gets the same verdict as the top. All three shared-cache consumers (scan split, size estimate, ANALYZE...WITH SAMPLE stats) recurse consistently by construction (single injected instance). recursive=false stays byte-identical to today. Tests: HiveFileListingCacheTest +6 (recursive descent / non-recursive top-level-only / hidden-dir+file skip / default-true / property=false / nested-descent failure is skippable); FakeFileSystem gains tree-mode + per-location list error. fe-connector-hive 318/318, 0 checkstyle. Design: plan-doc/tasks/designs/FIX-recursive-directories-design.md Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4
…, mark done, repoint remaining Fold the 5 minor red-team folds into the design (legacy net-parity of skip-hidden-dirs via full-path containsHiddenPath, listFileSizes as 3rd consumer, subdir-failure test, FakeFileSystem invariants, RED-ability labels). HANDOFF: hive_config_test tags 2/21 DONE (2bc7ad9); remaining fe-core/SPI work narrows to query_cache + default_partition (+ ENV items). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4
…on flag via SPI
Restore genuine-NULL partition semantics for __HIVE_DEFAULT_PARTITION__ on a
non-string (INT/DATE) partition column. The shared fe-core partition-item builder
hardcoded isNull=false, so the sentinel was parsed as the column type
(IntLiteral("__HIVE_DEFAULT_PARTITION__") throws), the partition was log-skipped,
and the table mis-reported UNPARTITIONED (partition=0/0; WHERE col IS NULL empty).
Add a positional List<Boolean> partitionValueNullFlags to ConnectorPartitionInfo
(default empty = not opted in = non-null; unchanged for hudi/iceberg/maxcompute).
fe-core PluginDrivenMvccExternalTable.toListPartitionItem zips the flag into
PartitionValue(value, isNull) -> a typed NullLiteral, no parse. fe-core never
string-compares a sentinel (iron rule): hive and paimon render the identical
__HIVE_DEFAULT_PARTITION__ with opposite intent, so nullness is connector-supplied.
- hive: HiveConnectorMetadata.listPartitions marks the HMS default-partition
sentinel isNull=true (exact HIVE_DEFAULT_PARTITION.equals -> legacy
HiveExternalMetaCache:309 parity), built positionally from
HiveWriteUtils.toPartitionValues so it zips with the fe-core name re-parse.
- paimon: adopts genuine-NULL too (user decision) -> its partition.default-name
partition becomes a NullLiteral, so col IS NULL prunes to it and an MTMV refresh
materializes the null rows (was col IN ('sentinel'), which dropped them).
Realizes the connector's already-stated intent; partition-name identity kept.
- fail loud: toListPartitionItem asserts the nullFlags arity (empty or == types).
- PluginDrivenScanNode prune opt-out comment refreshed (code unchanged; the
col IS NULL example no longer applies now that paimon's null is a NullLiteral).
Tests (verified: api 6, fe-core 67, hive 9, paimon 10; 0 failures, 0 checkstyle):
ConnectorPartitionInfoTest +3, PluginDrivenMvccExternalTableTest +3 (INT/DATE
sentinel+flag -> NullLiteral; no-flag INT still drops), hive
HiveConnectorMetadataPartitionListTest +2, paimon
PaimonConnectorMetadataPartitionTest +1.
Design + red-team (5 lenses, all SOUND_WITH_FIXES, folded):
plan-doc/tasks/designs/FIX-default-partition-design.md
e2e (user): test_hive_default_partition (INT+DATE) + paimon qt_null_partition_* +
test_paimon_mtmv (golden 3->5 rows, regenerate; MV null-partition-creation gate).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4
…red-team, mark DONE Design + red-team record for the connector-supplied per-value NULL partition flag (commit 58f3e36). Records: Option A SPI shape (positional List<Boolean>) chosen over name-keyed map / Java-null-in-map; user variant B (paimon adopts genuine-NULL); 5-lens adversarial red-team (all SOUND_WITH_FIXES) with RT-F1..RT-F6 folded, incl. the major finding that test_paimon_mtmv is an impacted existing suite (golden 3->5). HANDOFF: default_partition DONE; query_cache is the sole remaining code item. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4
…edTableIf capability After the hms cutover, a flipped hive table is a PluginDrivenMvccExternalTable scanned by a PluginDrivenScanNode, not an HMSExternalTable/HiveScanNode. The Nereids SQL result cache gated eligibility, the invalidation token, and the scan-node recognition on those retired class names, so a flipped hive table silently lost result-cache eligibility. Replace the six source-name branches across four files with the connector-agnostic MTMVRelatedTableIf capability + its stable, data-tied token getNewestUpdateVersionOrTime() (hive: max transient_lastDdlTime; iceberg/paimon: monotonic snapshot version), obeying the fe-core iron rule (no instanceof HMSExternalTable / HiveScanNode). Scope = all lakehouse plugin tables that expose a valid token (hive/iceberg/paimon/hudi), gated by the existing enable_hive_sql_cache switch (default false) so behavior is unchanged unless opted in. - BindRelation: admit any ExternalTable that is MTMVRelatedTableIf. - SqlCacheContext.addUsedTable: capture getNewestUpdateVersionOrTime() as the version token; token <= 0 fails safe (mark unsupported) rather than pin a bogus constant that would serve stale results. - NereidsSqlCacheManager: admit PLUGIN_EXTERNAL_TABLE in the type gate; re-check freshness and the partition-existence skip via MTMVRelatedTableIf. - CacheAnalyzer: recognize an external cacheable scan node by its TARGET TABLE's MTMVRelatedTableIf capability (not the scan-node class) so hudi (HudiScanNode extends HiveScanNode) is not dropped and jdbc-query TVFs (FunctionGenTable) are excluded; source the freshness marker from the connector token. The dead source- specific COUNTER_QUERY_HIVE_TABLE bump is removed. The default (unstable) ExternalTable.getUpdateTime() = System.currentTimeMillis() at schema load would serve stale results within the schema-cache TTL; the connector token is data-tied. Tests (fe-core, all RED on the pre-cutover HEAD): PluginTableCacheAnalyzerTest (4), SqlCacheContextPluginTableTest (2), NereidsSqlCacheManagerPluginTableTest (2) — capability recognition, token capture + <=0 fail-safe, and invalidate-on-token-change. fe-core BUILD SUCCESS, 0 checkstyle. Design + red-team: plan-doc/tasks/designs/FIX-querycache-spi-design.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4
…k DONE Design for migrating the Nereids SQL result cache off source-name branches onto the MTMVRelatedTableIf capability + getNewestUpdateVersionOrTime() token (all four loci, the token cost accounting, blast radius, and the as-implemented tests). Folds the adversarial red-team (wf_fe484bc8-43b): capability-based scan-node recognition (hudi / jdbc-TVF), honest token-cost paragraph + registered follow-up optimization, corrected sys-table safety rationale, concrete test seams. Marks query_cache DONE in HANDOFF and repoints the next step to HIVEFS-8 / reverify backlog. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4
… listing fe-connector-hive's HiveScanPlanProvider.listAndSplitFiles unconditionally swallowed a missing-partition-location listing failure, silently dropping the legacy hive.ignore_absent_partitions=false semantics (old HiveExternalMetaCache threw "Partition location does not exist" when the flag was false). Restore it: when the listing failure is a not-found (a FileNotFoundException anywhere in the cause chain) AND hive.ignore_absent_partitions=false, rethrow a loud DorisConnectorException; the default true keeps the tolerant skip-with-warning, and transient/unreadable listing failures still skip. Add the IGNORE_ABSENT_PARTITIONS property constant. Also make hive_config_test idempotent (groovy-only): it wrote OUTFILE ORC files into fixed HDFS dirs that were never cleaned, so reruns accumulated files and inflated the recursive_directories row counts (order_qt_1/2/21). Clear the dirs the suite writes before writing (docker init loads no data into them); the country=India/Delhi absent-partition dir is intentionally left untouched so the final ignore_absent_partitions=false check still throws. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3
… SPI connector After the hms SPI cutover, hive catalogs run through fe-connector-hive, whose cache layers read only the namespaced meta.cache.hive.<entry>.ttl-second keys. The legacy catalog knobs (file.meta.cache.ttl-second, partition.cache.ttl-second, schema.cache.ttl-second) were validated but never remapped, so setting them to 0 no longer disabled the cache -- it silently kept the default 24h TTL, and newly written files/partitions/columns stayed invisible until REFRESH. fe-core did this remap via CacheSpec.applyCompatibilityMap in AbstractExternalMetaCache.initCatalog; the connector dropped it. Apply the same compatibility remap at each cache-construction site, mirroring HiveExternalMetaCache.catalogPropertyCompatibilityMap: - file.meta.cache.ttl-second -> meta.cache.hive.file.ttl-second (HiveFileListingCache) - schema.cache.ttl-second -> meta.cache.hive.table.ttl-second (CachingHmsClient) - partition.cache.ttl-second -> meta.cache.hive.partition_names.ttl-second (CachingHmsClient) partition.cache maps only to the partition-name list (fe-core's partition_values), not the per-partition objects cache, faithful to the legacy mapping. Add CacheSpec.metaCacheTtlKey(engine, entry) helper and unit tests asserting each legacy key at 0 disables the right cache (and only it). Backward-compatible: a no-op when the legacy keys are absent; the namespaced key wins if both are set. Fixes test_hive_meta_cache. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3
Both suites left state behind that broke re-runs against a persistent env: - test_hive_partition_values_tvf created hive db partition_values_db with table partition_values_all_types and never dropped it, so the section-13 `drop database if exists partition_values_db` (no FORCE) failed on re-run with "Database ... is not empty". Add FORCE so the pre-drop cleans the leftover non-empty db. This matches the connector's dropDatabase(force) cascade and is the idiom already used across the hive suites; new fe-connector-hive and legacy HiveMetadataOps.dropDbImpl behave identically here (not a regression). - test_hive_ctas_to_doris created internal tables _4/_5/_6 (and the internal scratch db) without dropping them; the _5 CTAS has no IF NOT EXISTS and no drop-before, so a re-run hit "Table 'tb_test_hive_ctas_to_doris_5' already exists". Drop the internal scratch db at the start (mirroring the partition_values_tvf pattern) to clean all leftover internal tables. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4
…littable in SPI scan The SPI hive scan (HiveScanPlanProvider) lost two LZO behaviors that legacy fe-core HiveScanNode/HiveExternalMetaCache had, because HiveFileListingCache only strips _/.-prefixed hidden files and HiveFileFormat maps LZO text to the shared TEXT format: - *.lzo.index sidecar files were scanned as data. For an LZO text table only *.lzo files are data; the Hadoop-LZO *.lzo.index sidecar (starts with neither _ nor .) was read as an extra text row -> test_hive_lzo_text_format got count 6 instead of 5. Port legacy HiveUtil.isLzoDataFile: when the table uses an LZO input format, keep only files ending in .lzo in listAndSplitFiles. - LZO files were treated as splittable. TEXT.isSplittable() is true, but a .lzo stream cannot be decompressed from an arbitrary byte offset (legacy HiveUtil.isSplittable returned false for LZO). Mask splittable with !isLzo in planScan/planScanForPartitionBatch. Latent before (the test file is < one split), but wrong for large LZO files. isLzoInputFormat/isLzoDataFile are connector-local (fe-connector-hive has no fe-core dependency). LZO-ness is table-level (the connector reads every partition with the single table format). ACID is never LZO, so the transactional branch is unchanged. Added HiveScanPlanProviderLzoFilterTest. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4
Contributor
Author
|
run buildall |
Contributor
FE UT Coverage ReportIncrement line coverage |
… Hadoop conf The FE-side Hudi metaClient built its Hadoop Configuration by copying only hadoop./fs./dfs./hive.-prefixed catalog properties, silently dropping the Doris storage-style object-store credentials (s3.access_key / s3.secret_key / s3.endpoint / use_path_style). S3AFileSystem then fell back to the default AWS provider chain and failed with NoAuthWithAWSException, so every hudi metadata read (schema load, hudi_meta timeline, partition listing) surfaced as "Hudi metadata operation failed for catalog". Mirror the iceberg/paimon connectors: overlay the catalog's bound fe-filesystem StorageProperties.toHadoopProperties().toHadoopConfigurationMap() (which translates s3.access_key -> fs.s3a.access.key, etc.) into both buildHadoopConf() sites, before the existing raw fs./dfs./hadoop. passthrough so inline user keys still win. HudiConnectorMetadata gains an overloaded constructor carrying the translated map (the 3-arg constructor delegates with an empty map, so existing same-loader unit tests are unchanged). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4
…or FE schema read
Resolving a Hudi table's Avro schema reads a data file's parquet footer:
TableSchemaResolver -> ParquetUtils.readAvroSchema -> readSchema ->
ParquetFileReader.readFooter(Configuration, Path), then converts the parquet
MessageType to Avro via parquet-avro's AvroSchemaConverter. The hudi plugin
bundles no parquet, and hive-catalog-shade carries only a RELOCATED
(org.apache.paimon.shade.*) parquet, so the un-relocated ParquetFileReader
resolved from the parent 'app' loader while ParquetUtils and Configuration came
from the child loader (org.apache.hadoop is child-first, from the bundled
hadoop-common). readFooter's Configuration parameter then split across two
loaders -> java.lang.LinkageError (loader constraint violation on
org.apache.hadoop.conf.Configuration), previously masked by the S3 credential
failure fixed in the preceding commit.
Co-locate parquet child-first (same FIX-PAIMON-HADOOP-CLASSLOADER pattern the
plugin already uses for hadoop-aws), keeping ParquetFileReader + Configuration
in one loader. parquet-hadoop transitively pulls
column/common/encoding/format-structures/jackson; parquet-avro supplies
AvroSchemaConverter (+ parquet-variant); versions are BOM-managed
(${parquet.version} = 1.17.0). Bundled via plugin-zip.xml's runtime dependencySet.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4
…E s3a:// timeline reads The hudi connector opens the table base path on the FE to read the .hoodie timeline (HudiScanPlanProvider.buildMetaClient -> HoodieTableMetaClient.getStorage -> FileSystem.get). hadoop-common is bundled child-first (org.apache.hadoop is not in the plugin's parent-first allowlist), so bundling it WITHOUT hadoop-aws left FileSystem in the child loader while S3AFileSystem resolved from the parent 'app' loader, and FileSystem.createFileSystem's cast threw "S3AFileSystem cannot be cast to FileSystem". Co-locate hadoop-aws child-first (same FIX-PAIMON-HADOOP-CLASSLOADER split iceberg/paimon carry) so the cast succeeds, plus the AWS SDK v2 modules S3A needs at runtime (s3, apache-client, s3-transfer-manager) so its ExecutionAttribute static registers against the child sdk-core instead of colliding with fe-core's and poisoning S3 for the whole JVM. Also updates the adjacent parquet comment to note hadoop-aws now sits below it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4
…E native reader Native COW / MOR-no-log reads over an s3a:// (MinIO) warehouse failed with BE "[INVALID_ARGUMENT]Invalid S3 URI: s3a://..." — BE's S3URI (s3_uri.cpp) accepts only s3/http/https. The new fe-connector hudi path shipped the raw HMS s3a:// location to TFileRangeDesc.path without the s3a->s3 rewrite that legacy HudiScanNode performed via the 2-arg LocationPath.of(path, storagePropertiesMap). Fix: normalize the NATIVE range .path() via context.normalizeStorageUri (mirrors the iceberg/paimon normalizeUri seam; fe-core does no property parsing) at ALL THREE native emitters: - collectCowSplits (snapshot COW) - buildMorRange (snapshot + incremental MOR no-log native slice) - COWIncrementalRelation.collectSplits (COW @incr; found by the design red-team) The JNI THudiFileDesc base/data/delta-log paths stay raw s3a:// (Hadoop S3AFileSystem wants the s3a scheme). Threaded as a UnaryOperator<String> through incrementalRanges so the static package-private test seams (buildMorRange / incrementalRanges / collectSplits) stay intact; a null context (offline UT) preserves the raw URI. Design: plan-doc/tasks/designs/FIX-hudi-s3a-native-scheme-design.md (red-team wf_4e4ec1d7-d4f GO_WITH_FIXES: third-site blocker folded in). Tests: HudiIncrementalPlanScanTest +2 (COW @incr + MOR no-log native path s3a->s3), existing call sites threaded with identity; fe-connector-hudi 25/25 green, 0 checkstyle violations. e2e (user-run): the 10 native-signature hudi p2 suites must stop throwing Invalid S3 URI (full green also needs FIX-hudi-s3a-jni-creds). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4
… Hudi scanner MOR-with-log hudi reads via the JNI scanner over an s3a:// (MinIO) warehouse failed with NoAuthWithAWSException "No AWS Credentials provided" (also any force_jni_scanner=true sub-query). The BE JNI reader builds the Java scanner's Hadoop Configuration from TFileScanRangeParams.properties (hadoop_conf.-prefixed, stripped back by HadoopHudiJniScanner). getScanNodeProperties emitted only (1) getBackendStorageProperties() BE-canonical AWS_* (which S3AFileSystem ignores) and (2) a raw passthrough of the catalog's s3. aliases (also ignored) — never the Hadoop-canonical fs.s3a.access.key/secret.key/endpoint. The translation (storageHadoopConfig -> StorageProperties.toHadoopProperties) existed but fed only the FE metaClient conf, not the BE scan; legacy getLocationProperties merged backendStorageProperties + the translated hadoop props, and the new path dropped the second half. Fix: emit storageHadoopConfig(context) under the location. prefix in getScanNodeProperties, after the AWS_* canonical set (1) and before the raw passthrough (2) so a user-inline fs./hadoop. key still wins (mirrors buildHadoopConf precedence). Native FILE_S3 reader is unaffected (reads AWS_* only; the extra fs.s3a.* are ignored, same as the s3. aliases already emitted). storageHadoopConfig null-guards. Design: plan-doc/tasks/designs/FIX-hudi-s3a-jni-creds-design.md (red-team wf_4e4ec1d7-d4f SOUND; linchpin — the HMS/hudi ConnectorContext storage IS bound — confirmed). Tests: HudiBackendDescriptorTest +1 (typed StorageProperties -> fs.s3a.* emitted under location., catalog carrying only s3. aliases = the real failing scenario). fe-connector-hudi full suite 175/175 green, 0 checkstyle. Also completes the task list (both hudi s3a fixes) + FIX-A summary. e2e (user-run): the 4 JNI-signature suites must stop throwing NoAuthWithAWSException. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4
…+ JNI creds), e2e pending Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4
Contributor
FE Regression Coverage ReportIncrement line coverage |
…+ emit canonical BE creds Plain-hive tables on an object-store warehouse (s3a://oss://cos://) would fail BE-side native reads — the same latent gap class that broke hudi (found via the hudi design red-team; user asked to fix hive too). Two gaps in fe-connector-hive, both unexercised locally (hive docker regression uses hdfs://; hive-s3 suites are p2/real-cloud): 1. Scheme: splitFile -> newRangeBuilder .path(filePath) shipped the raw s3a:// path to TFileRangeDesc.path; BE's native S3 reader (s3_uri.cpp S3URI::parse) accepts only s3/http/https. Legacy HiveScanNode normalized via the 2-arg LocationPath.of; hive's OWN write path already used context.normalizeStorageUri — the read path never got it. 2. Creds (a legacy regression): getScanNodeProperties emitted only the raw s3./fs. catalog aliases; BE's native FILE_S3 reader reads ONLY canonical AWS_ACCESS_KEY/SECRET/ENDPOINT (s3_util.cpp:146-148), so a private bucket 403s. Legacy getLocationProperties() returned getBackendStorageProperties() (canonical); the new path dropped it. (Hive has no JNI scanner -> no fs.s3a./JNI-creds analog.) Fix (fe-core untouched; no source-specific code): - normalizeNativeUri(raw) = context != null ? context.normalizeStorageUri(raw) : raw. Applied to the native data-file path in splitFile (covers regular + ACID data files) and to the ACID BE-facing paths (acidLocation + encodeDeleteDeltas delete-delta dir, threaded a UnaryOperator<String>). Files are still LISTED with the raw scheme (Hadoop S3AFileSystem wants s3a); only BE-facing copies are normalized. - getScanNodeProperties emits context.getBackendStorageProperties() (canonical AWS_* / resolved hadoop.*/dfs.*) under location. before the raw passthrough (inline fs./hadoop. still wins). Restores legacy behavior; for HDFS this equals what legacy sent (parity → HDFS suites preserved). Design: plan-doc/tasks/designs/FIX-hive-s3a-read-design.md. Tests: HiveScanBatchModeTest +2 (RED-able: range path s3a->s3 via planScanForPartitionBatch; canonical AWS_* emission). Full fe-connector-hive suite 328/328 green, 0 checkstyle. UNIT-VERIFIED ONLY — object-store e2e needs a real s3/oss hive table (docker is HDFS); the HDFS hive suites are the parity regression guard for the creds change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4
…onical creds), e2e pending Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4
The SPI HmsConfHelper.createHiveConf only copied raw catalog properties, so a kerberos HMS catalog that declared hadoop.security.authentication=kerberos but did not spell out hive.metastore.sasl.enabled opened a plain TSocket that the kerberized metastore dropped with TTransportException (regression test_single_hive_kerberos, test_two_hive_kerberos). Restore the legacy fe-core HMSBaseProperties.initHadoopAuthenticator behavior: inject hive.metastore.sasl.enabled=true when the metastore/hadoop auth is kerberos (hadoop.security.authentication or hive.metastore.authentication.type). Adds HmsConfHelperTest (6 cases): SASL on via either auth key incl. case-insensitive; simple/none auth leaves SASL off; arbitrary props pass through. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4
…th literal 'NULL' string A genuine-NULL list value and a literal 'NULL' string both render to the text "NULL" once PartitionKeyDesc quotes the value and generatePartitionName strips the quotes, so a partition column holding both (paimon null_partition: a real NULL row plus a 'NULL' string row) generated two partitions named p_NULL and failed the uniqueness check on CREATE MATERIALIZED VIEW (regression test_paimon_mtmv: "Duplicated named partition: p_NULL"). This surfaced once variant B (58f3e36) gave paimon/hive genuine-NULL semantics. Prefix a null-bearing MTMV partition with pn_ (read from PartitionValue.isNullPartition()) instead of p_: string values always yield a p_ name (second char '_'), so a pn_ name can never collide, and real-NULL vs 'NULL'-string stay distinct partitions. Both the base-connector item and the MV-OLAP item render pn_NULL symmetrically, so the partition-sync join is unaffected (unlike the reverted FIX-3, which used a one-sided sentinel). Renames the genuine-NULL MTMV partition p_NULL -> pn_NULL globally (hive default partition and internal OLAP null partitions too). Updates the impacted unit tests (ListPartitionItemTest + a new 'NULL'-string -> p_NULL case) and regression suites (test_hive_default_mtmv, test_null_partition_mtmv, test_null_multi_pct_mtmv, test_paimon_mtmv). hudi (isNull=false) and user-defined p_NULL partition names are unaffected. The test_paimon_mtmv order_qt_null_partition golden regenerates on the e2e run (genuine-NULL rows now materialize). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4
…equals/hashCode NPE FileSplit is Lombok @DaTa, whose generated equals()/hashCode() invoke the hand-written getSelfSplitWeight() (returns primitive long). That getter unboxed the nullable Long selfSplitWeight field, so any FileSplit that never set a size-based weight (the common case; production reads the field directly in getSplitWeight(), which is null-safe) threw NPE the moment its equals()/hashCode() ran -- e.g. the multimap comparison in FederationBackendPolicyTest.testGenerateRandomly. Return -1 ("not provided") for a null weight, matching the ConnectorScanRange SPI default, PluginDrivenSplit's weight >= 0 gating, and the sibling Iceberg/PaimonScanRange (which use primitive long fields). IcebergSplit inherits the fixed getter. Adds a deterministic regression test. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4
Contributor
Author
|
run buildall |
Contributor
FE UT Coverage ReportIncrement line coverage |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
only for testing #65473