[branch-4.1] Backport FileScannerV2 and follow-up fixes#65559
Open
Gabriel39 wants to merge 34 commits into
Open
[branch-4.1] Backport FileScannerV2 and follow-up fixes#65559Gabriel39 wants to merge 34 commits into
Gabriel39 wants to merge 34 commits into
Conversation
Problem Summary: Add the file scanner v2 reader stack for external file scans, including native readers for Parquet, CSV/TEXT, JSON, JNI-backed table readers, schema projection, column mapping, predicate handling, reader statistics, page cache support, and related BE/FE integration. This also restores affected Parquet LZO regression cases by adding Doris thirdparty Arrow LZO page decompression support for file scanner v2. Support file scanner v2 readers for external file scan paths, including LZO-compressed Parquet reads in the new Parquet reader path.
## Summary Add Parquet FileScannerV2 prefetch support and align the v2 row-group read path with the v1 Parquet reader's `MergeRangeFileReader` behavior. ## Changes - Keep the Arrow Parquet random-access wrapper, but make its underlying Doris `FileReader` switchable per row group. - Build projected Parquet leaf-column chunk ranges for the current row group, including nested projections. - Before opening each selected row group, configure `MergeRangeFileReader` when the average projected column-chunk IO is smaller than `MergeRangeFileReader::SMALL_IO`, matching the v1 small-random-IO path. - Respect the existing `merge_read_slice_size` query option when constructing `MergeRangeFileReader`. - Preserve `TracingFileReader` statistics when switching between the raw reader and merge reader. - Keep best-effort file-cache warm-up only as a fallback when merge-range reading is not active. ## Notes The important v1 parity point is that subsequent Arrow `ReadAt()` calls now go through `MergeRangeFileReader` for small projected row-group IOs. This changes the actual read path, not just background cache warming.
File scanner v2 still carried ColumnPredicate metadata pruning through FileScanRequest for Parquet row-group dictionary and bloom filters after ZoneMap pruning had moved to VExpr. This kept two pruning APIs in the same scanner path and made statistics/profile accounting harder to keep consistent. This change adds VExpr dictionary and bloom-filter evaluation interfaces, wires comparison/IN/compound predicates through those interfaces, and makes the Parquet v2 row-group/page pruning path evaluate localized VExpr conjuncts for ZoneMap, dictionary, and bloom filters. FileScanRequest no longer carries ColumnPredicate filters, and the remaining TableColumnPredicates plumbing has been removed from FileScannerV2 and table reader v2 tests. The existing STATISTICS/DICTIONARY/BLOOM_FILTER prune reasons and profile counters remain. Parquet bloom pruning now probes the file bloom filter with the Parquet physical carrier instead of the widened Doris logical type. This keeps UINT32 predicates from hashing as INT64, handles fixed-length byte-array string blooms with FLBA hashing, and avoids unsupported logical encodings such as FLOAT16. The change also removes a header-level segment_v2 namespace import from match.h that made ZoneMap ambiguous after expr zonemap headers were included broadly. This also keeps the earlier cleanup that translates Chinese comments under be/src/core/data_type_serde to English.
### What problem does this PR solve? Regenerates the expected output for `external_table_p0/tvf/test_hdfs_parquet_group4.groovy` with the latest rebuilt BE and FE binaries on the requested remote test environment.
…che#65218) FileScannerV2 only refreshed profile file-read counters in `update_realtime_counters()`, so the query IOContext counters stayed at zero for external file scans. Workload policies using `be_scan_bytes_from_remote_storage` could therefore miss remote Hive/Parquet reads and fail to stop queries. This PR publishes FileScannerV2 scan rows, scan bytes, local bytes, and remote bytes to the query IOContext and Doris metrics using deltas from cumulative reader/cache stats. The row-accounting implementation was reworked after review. The old approach tried to synchronize concrete `FileReader` private row statistics from `TableReader` after `get_block()` or `close()`. That abstraction was wrong because `FORMAT_JNI` overrides `TableReader::get_block()`, materialized readers could expose post-filter rows, and Parquet `COUNT(complex_col)` aggregate reads nested levels without going through the normal block row path. The current implementation records scan rows at the point each reader actually reads or materializes source rows, before file-local filters: - Parquet normal scan records the scheduler raw-row delta. - Parquet `COUNT(complex_col)` records the selected batch rows loaded through nested levels. - CSV/TEXT records `rows_before_filter`. - JSON, Native, and RemoteDoris record rows after materializing the file block and before applying filters. - JNI table readers record the `current_rows` returned by the Java scanner before `finalize_jni_block()`. This PR also handles `io_ctx->should_stop` in the FileScannerV2 reader path. `TableReader` short-circuits when the scanner stop flag is set, stop-time reader init/open EOF is converted to normal EOS, the Parquet v2 reader / Arrow `RandomAccessFile` path checks the stop flag before file size/read operations, and Parquet aggregate pushdown converts stop-time nested-level read EOF into normal scanner EOS instead of a scan failure.
Format V2 Paimon JNI reader now falls back to legacy split-level paimon_predicate when scan-level predicate is missing or empty. Added BE unit coverage for scan priority, legacy fallback, and missing predicate failure. Validation: build-support/clang-format.sh passed; ./run-be-ut.sh --run --filter=PaimonJniReaderTest* was attempted but local CMake is blocked by incomplete thirdparty/installed dependencies after missing Protobuf library.
Paimon JNI reads created `TableRead` without `withIOManager`, so Paimon primary-key merge reads could not spill through Paimon IOManager. This PR wires Doris catalog properties through FE and BE into the Java scanner, creates the Paimon IOManager for JNI reads, and closes it on scanner close or open failure. The same IOManager option/default temp-dir handling is implemented in both Paimon JNI scan paths: - legacy `be/src/format/table/paimon_jni_reader.cpp` - Format V2 `be/src/format_v2/jni/paimon_jni_reader.cpp`
) Problem Summary: Paimon JNI scans can create Paimon SDK internal async file readers, especially for primary-key merge reads over large ORC files. Doris scanner concurrency limits do not directly expose those internal async reader threads or Java heap pressure in the query profile. This change adds lightweight profile metrics from the Paimon JNI scanner for active scanner counts, async reader thread counts, JVM heap/non-heap usage, split and predicate diagnostic sizes, async threshold configuration visibility, and readBatch/open timing counters.
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: Creating Iceberg or Paimon external tables with
mixed-case partition columns could fail because Doris converted
top-level external column names to lower case while building external
schemas and partition specs. Reading external table schemas and
partition metadata also normalized some Paimon and Iceberg column names
to lower case, so SHOW CREATE and partition helpers could lose the
original external column spelling. This change preserves the original
top-level external field names when converting Doris columns to
Iceberg/Paimon schemas, resolves partition and primary key names
case-insensitively back to the external canonical names, and stops
schema/partition parsing paths from lowercasing external column names.
### Release note
Fix Iceberg and Paimon external table column name casing for mixed-case
partition columns.
### Check List (For Author)
- Test: Unit Test
- Maven focused FE test: MAVEN_ARGS=-o
JDK_17=/usr/local/opt/openjdk@17/libexec/openjdk.jdk/Contents/Home
JAVA_HOME=/usr/local/opt/openjdk@17/libexec/openjdk.jdk/Contents/Home
mvn test -pl fe-core -am -Dcheckstyle.skip=true -DfailIfNoTests=false
-Dmaven.build.cache.enabled=false
-Dtest=CreateIcebergTableTest,PaimonMetadataOpsTest,IcebergUtilsTest#testParseSchemaPreservesNonLowercaseColumnNames,PaimonUtilTest#testParseSchemaPreservesNonLowercaseColumnNames
- git diff --check
- A broader focused run including two existing Mockito-based
IcebergUtilsTest methods compiled successfully but those two methods
failed locally because Mockito inline Byte Buddy could not self-attach
to the Homebrew JDK 17 VM.
- Behavior changed: Yes. Iceberg and Paimon external schemas, partition
specs, and partition metadata now preserve external column name casing.
- Does this need documentation: No
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
Problem Summary: Deletion vector and position-delete cache builders allocated rows/maps with raw `new` and returned `nullptr` on read or parse failures. Those error paths leaked partially built cache values because `KVCache::get` only takes ownership when the builder returns a non-null pointer. This PR uses RAII while building `DeleteRows` and `DeleteFile`, releases ownership only after successful construction, and adds fault-injection/regression coverage for V1 Iceberg deletion-vector and position-delete failures, V1 Paimon Parquet/ORC deletion-vector failures, the shared Iceberg helper, and the format v2 TableReader path.
The Iceberg deletion vector regression test validated the NumDeleteRows profile counter with a fixed global value. In environments with multiple parallel fragment instances, each instance can report the same delete vector rows, so the merged profile counter becomes a multiple of the expected row count and the test fails even though query results are correct. The case now runs the split-cache profile query with a single fragment instance and refreshes the catalog after Spark creates and registers the format_v3 namespace before switching to it in Doris.
…pache#65370) Reading Iceberg struct fields after Spark schema evolution can materialize a column whose root nullability does not match the declared file type used to prepare the cast expression. The cast wrapper unwraps nullable inputs based on the declared input type, so a nullable declared type with a non-nullable actual column can hit an invalid unwrap path and crash BE instead of returning the cast result. This change aligns the declared cast input root nullability with the actual column shape before executing the cast. It also adds a focused BE unit test and an Iceberg regression case that creates a Spark Iceberg struct, evolves col.a from INT to BIGINT, refreshes the Doris catalog, and queries col.a/col.b/col.c with nested pruning enabled.
…pache#65369) Problem Summary: The format_v2 Parquet reader previously materialized all predicate columns before evaluating row-level predicates, so later predicate columns could not reuse rows rejected by earlier single-column predicates. This PR implements the StarRocks-style round-by-round predicate-column read path for Doris format_v2: deterministic single-column conjuncts are scheduled with their predicate columns, evaluated immediately after each column is read, and later predicate columns are read through ParquetColumnReader::select() only for surviving rows. If one predicate round filters the whole batch, unread predicate column readers skip that batch instead of materializing data. Multi-column conjuncts and delete conjuncts still run after the required predicate columns are available. The PR also adds row-level dictionary predicate filtering for dictionary-encoded Parquet columns in the v2 path. Dictionary-capable predicate children are evaluated against dictionary entries first, the data-page reader filters by dictionary id, and residual conjuncts keep the normal row-level expression path. Compound AND predicates are split so dictionary-covered children are not re-evaluated after the dictionary prefilter, while non-dictionary residual children still run on surviving rows. Profile counters and timers were added for dictionary filter candidates, selected dictionary filters, unsupported/read-failure cases, rows filtered by dictionary filters, dictionary expression rewrite time, dictionary read time, and dictionary filter build time. For correctness, volatile or non-deterministic predicates stay on the old full-batch path. Expressions such as random/rand, random_bytes, uuid, and uuid_numeric are marked non-deterministic at the vectorized expression layer; if any pushed conjunct is non-deterministic, the round-by-round schedule is disabled for that batch so stateful functions see the same full input stream as before this optimization.
…#65449) ### What problem does this PR solve? Issue Number: None Related PR: None Problem Summary: FileScannerV2 collected file-cache statistics but did not publish the FileCache profile subtree or account for cache-write bytes in the query resource context. It also lacked V1-compatible split pruning for late-arriving runtime filters on partition columns and always propagated missing-file errors, even when `ignore_not_found_file_in_external_table` was enabled. This PR: - publishes the shared FileCache profile counters from FileScannerV2 and records cache-write bytes; - passes the latest scanner conjunct snapshot into `TableReader::prepare_split`, where partition-only runtime filters are evaluated before a concrete reader is opened; - reports `FileScannerRuntimeFilterPartitionPruningTime` and `RuntimeFilterPartitionPrunedRangeNum`; - skips and counts `NOT_FOUND` splits when the external-table missing-file option is enabled; - resets native, JNI, Hudi hybrid, and Paimon hybrid split state before continuing; - adds focused unit coverage for profile reporting, partition pruning, and split cleanup after `NOT_FOUND`. ### Release note FileScannerV2 profiles now expose FileCache statistics, runtime-filter partition range pruning statistics, and ignored missing-file counts. FileScannerV2 can prune partition ranges before opening a reader and can continue after missing external files when the existing ignore option is enabled. ### Check List (For Author) - Test: Manual test / Unit Test attempted - clang-format 16 dry-run passed for all changed C++ files - `git diff --check` passed - targeted BE unit tests were started, but CMake configuration was blocked because `thirdparty/installed` is missing Protobuf; tests did not reach compilation - Behavior changed: Yes. FileScannerV2 publishes additional profile counters, prunes partition ranges using runtime filters, and skips configured ignorable `NOT_FOUND` splits. - Does this need documentation: Yes. The existing FileScanner V1/V2 profile document is updated in place.
### What problem does this PR solve? Issue Number: None Related PR: None Problem Summary: Code reviews under `be/src/format_v2` need persistent, directory-scoped guidance for the FileScannerV2 architecture and external-data correctness boundaries. This PR adds a local `AGENTS.md` and repository-local design/review references covering: - TableReader, TableColumnMapper, and FileReader responsibility boundaries - external lake/file format and writer compatibility - common FileReader index, predicate, cache, and virtual-column review checks - Parquet Row Group/Page/Row pruning, indexes, lazy materialization, and I/O behavior - ORC predicate-to-SARG conversion, Stripe/row-index/Bloom usage, and fallback correctness - focused correctness, interoperability, differential, and performance-test expectations Detailed checklists are stored under `docs/` and loaded on demand from the directory-scoped instructions to stay below the Codex instruction-size limit. ### Release note None ### Check List (For Author) - Test: No need to test (documentation-only change; Markdown whitespace, local references, Mermaid fences, and English-only content were verified) - Behavior changed: No - Does this need documentation: No
### What problem does this PR solve?
Issue Number: N/A
Related PR: N/A
Problem Summary:
Parquet COUNT pushdown uses one representative leaf to count the
top-level null state of a complex column. When a STRUCT contains an
ARRAY or MAP descendant, the representative levels are repeated. The
previous code compared repetition levels with the STRUCT root repetition
level, which is zero, skipped every level entry, and triggered a
`DORIS_CHECK` that aborted the BE. It also derived the non-null
threshold from the repeated leaf definition level and could count a NULL
struct as non-null.
This change identifies top-level rows with repetition level zero and
uses the root schema's nullable definition level to determine whether
the top-level complex value is non-null. Empty and NULL collections
inside a non-null STRUCT remain valid STRUCT rows.
### Release note
Fix BE aborts and incorrect COUNT results for Parquet STRUCT columns
containing ARRAY or MAP descendants.
### Check List (For Author)
- Test: Unit Test
-
`NewParquetReaderTest.CountStructWithRepeatedChildUsesTopLevelRowBoundaries`
- Existing STRUCT, LIST, and MAP COUNT pushdown tests
- Behavior changed: Yes. COUNT pushdown uses top-level row/null
semantics for repeated descendants.
- Does this need documentation: No
Validation on the designated Linux build host:
- `build-support/check-format.sh`
- Targeted BE unit tests: 4 tests passed
- `run-clang-tidy.sh`: test file has no warnings; production analysis
reports pre-existing full-file diagnostics and `jni-util.h` toolchain
static assertions unrelated to this change.
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: External file condition-cache entries stored only their
survivor bitmap. On a
later cache hit, the physical reader recomputed the bitmap base from the
current metadata-pruned
scan plan. If pruning selected a different first row group, bitmap
indexes shifted and valid rows
could be silently skipped.
This change stores the bitmap's base granule with the cache value. The
V2 table reader restores the
stored base on cache hits, while V2 Parquet and ORC readers derive a
base from their current plan
only when producing a new cache entry. V1 reader behavior is unchanged.
### Release note
Fix incorrect row filtering when a V2 external file condition-cache hit
uses a different pruned
scan plan from the cache-producing scan.
### Check List (For Author)
- Test: Unit Test
-
`ParquetScanConditionCacheTest.HitKeepsCachedBaseWhenCurrentPlanStartsLater`
- `TableReaderTest.ConditionCacheMissPublishesBitmapAfterReaderEof`
- `NewOrcReaderTest.ConditionCacheHitUsesSplitBaseGranule`
- Behavior changed: Yes. V2 condition-cache hits use the bitmap
coordinate base stored with the
cache entry instead of recomputing it from the current scan plan.
- Does this need documentation: No
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: FileScanner V2 used an independent CSV field-splitting
state machine that diverged from the line reader for bare quotes,
escapes outside enclosed fields, and configurations where escape equals
enclose. It also removed double quotes before nullable string
conversion, causing quoted null markers to be treated as NULL. This
change reuses the line reader's separator positions, preserves
quoted-string provenance through null matching, and adjusts separator
positions when a UTF-8 BOM is removed.
### Release note
Fix CSV FileScanner V2 parsing for enclosed fields and quoted null
literals.
### Check List (For Author)
- Test: Unit Test and Regression test
- `CsvV2ReaderTest`: 30 tests passed
- `test_local_tvf_csv_enclose_consistency`: passed
- Behavior changed: Yes. CSV V2 now matches the established
enclosed-field parser and preserves quoted null literals as strings.
- Does this need documentation: No
…rquet reads (apache#65500) Problem Summary: FileScannerV2 and its Parquet/ORC readers had several correctness and resource-control gaps: - late or pending runtime filters were not consistently reflected in split readers and could race with irreversible table/file COUNT and MIN/MAX shortcuts; - slotless or unsafe conjuncts could be omitted while split pruning, file-level predicate localization, or aggregate pushdown continued past them; - direct and nested VARBINARY filters, lossy schema-evolution casts, and TIMESTAMPTZ scale mismatches could be evaluated below the scanner with semantics different from the table expression; - missing, inverted, NaN, timezone-non-monotonic, truncated binary, or incorrectly cached Parquet/ORC statistics could drive unsafe pruning or synthetic aggregate results; - LIST, MAP, and STRUCT skips materialized discarded ranges and decoded unused leaf payloads, while nested binary builders could retain stale state between level batches; - direct slot-ref projections were treated as unsafe conversions, unnecessarily disabling valid MIN/MAX pushdown. This PR consolidates the fixes into one external-reader hardening change. It refreshes predicates for every split, waits for runtime-filter completion before aggregate shortcuts, disables aggregate pushdown whenever original conjuncts remain, preserves only safe conjunct prefixes, and keeps VARBINARY and TIMESTAMPTZ scale-mismatch filters above the file reader. It also validates all decoded ORC statistics bounds, makes invalid or inexact Parquet/ORC statistics fall back conservatively, scopes Bloom filters by row group and column, permits MIN/MAX only for order-preserving trivial mappings, and processes nested Parquet skips in bounded levels-only batches with correct binary-builder reset.
ORC file scanner v2 used a thin ORC input stream that issued reads directly through the underlying file reader. It did not implement the v1 ORC stripe-level stream collection and small-range merge path, so remote object storage scans could regress to many small stream reads for wide ORC files, multi-stripe files, and lazy or predicate-driven scans. This change adds a v2-native ORC input stream that implements beforeReadStripe(), builds selected stream ranges, merges adjacent small streams with Doris PrefetchRange policy, and serves arbitrary repeated or backward stream reads from an immutable merged-range cache. Large streams and unmerged single streams continue to read directly. The reader is wired into ORC v2 without changing v1 or falling back to v1.
### What problem does this PR solve? JNI-backed file scans had several lifecycle, compatibility, and adaptive-batching gaps: - The scan-level FileScannerV2 selector cannot inspect split-level JNI metadata. It could therefore route legacy Paimon JNI splits—especially old-FE splits without `reader_type`—to a scanner that cannot dispatch them. - A cancelled query could continue requesting Java batches when every row in the current batches was filtered out. - JNI open or close failures could discard cleanup state or mark the outer scanner closed too early, preventing retained Java resources from being cleaned up on a retry. - Adaptive batch-size changes did not consistently reach already-open JNI scanners and Paimon/Hudi hybrid readers. Paimon's physical reader cannot be resized after open, so accepting a later logical update could make the reported batch size diverge from the actual reader. - End-of-split state could be lost after the JNI scanner closed, making repeated reads after EOF non-idempotent. - Paimon options and Hadoop configuration supplied at split level were not retained as mixed-version fallback values. This PR hardens those paths by: - conservatively keeping scan-level JNI compatibility shapes on the legacy scanner and rejecting unsupported Paimon C++ ranges in FileScannerV2; - checking cancellation before fetching another Java batch; - preserving JNI cleanup state after failures, retaining each Paimon Java resource until its cleanup succeeds, and allowing scanner-level close to retry table-reader cleanup; - seeding the adaptive probe before eager JNI open, forwarding supported batch-size updates to open JNI and hybrid readers, and preserving Paimon's initial physical batch size after open; - preserving EOF state across scanner cleanup; and - applying explicit precedence between current scan-level Paimon settings and split-level rolling-upgrade fallbacks. ### Release note Harden JNI table-reader lifecycle handling, adaptive batching, and rolling-upgrade compatibility for Paimon and Hudi scans. ### Check List - [x] Added comments for scanner-selection, lifecycle, adaptive-batch, and compatibility invariants. - [x] Added focused unit tests covering cancellation, EOF idempotence, cleanup failure and retry, adaptive batching, scanner selection, and Paimon fallback precedence. - [x] Passed 30 focused ASAN BE unit tests from `FileScannerV2Test.*`, `JniTableReaderTest.*`, and `PaimonJniReaderTest.*` after addressing the latest review. - [x] Passed all 10 `PaimonJniScannerTest` Java tests; checkstyle reported 0 violations. - [x] Passed `build-support/check-format.sh` and `git diff --check`. - [x] Ran changed-file clang-tidy; analysis is blocked by the existing toolchain missing `stddef.h` and pre-existing repository diagnostics.
Adapt the requested FileScannerV2 and format_v2 backports to branch-4.1 interfaces. Keep the legacy V1 scanner path and apply the required V1 semantics manually without cherry-picking apache#62306.
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
Contributor
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
FE UT Coverage ReportIncrement line coverage |
Contributor
FE Regression Coverage ReportIncrement line coverage |
Issue Number: None Related PR: apache#65502 Problem Summary: Iceberg equality-delete handling could bind stale field ids in BY_NAME mode, lose binary initial-default bytes, treat missing V1 keys as physical columns, select id-less fields by stale ids, derive binary defaults from the latest schema instead of the selected snapshot, and read delete-file TIMESTAMPTZ keys without the data reader mapping option. Complex schema rematerialization could also trust a nested descriptor that omitted nullability, producing Struct(String) under a Struct(Nullable(String)) output type. This change uses shared name mapping, carries lossless binary defaults from the selected snapshot schema, materializes absent V1 keys as full columns, propagates TIMESTAMPTZ mapping, and treats the parent Struct DataType as the authoritative child-type contract while preserving Array and Map wrapper semantics. Fix Iceberg equality deletes for evolved schemas, binary defaults, missing V1 keys, TIMESTAMPTZ keys, and nullable nested fields. - Test: Unit Test - 17 focused BE ASAN tests covering Iceberg readers, the nullable renamed struct reproduction, and rebase compatibility - 5 existing complex projection/materialization BE ASAN tests - 175 ColumnMapper and TableReader BE ASAN tests - 17 Iceberg DDL/DML planning FE tests - 29 FE tests in ExternalUtilTest, IcebergUtilsTest, and IcebergScanNodeTest - Targeted clang-format 16 check - Behavior changed: Yes, equality deletes and evolved complex columns now resolve and materialize using the correct mapping, snapshot schema, and authoritative table types - Does this need documentation: No
Contributor
Author
|
run buildall |
Contributor
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
FE UT Coverage ReportIncrement line coverage |
Contributor
Author
|
run buildall |
Contributor
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
FE UT Coverage ReportIncrement line coverage |
Contributor
Author
|
run buildall |
Contributor
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
FE UT Coverage ReportIncrement line coverage |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Backport the FileScannerV2 / format_v2 series to
branch-4.1in the requested order.Source PRs
#65046, #65130, #65151, #65175, #65194, #65218, #65281, #65326, #65328, #65332, #65354, #65094, #65351, #65359, #65370, #65183, #65369, #65449, #65437, #65451, #65475, #65495, #65496, #65501, #65500, #65478, #65503, #65502.
Branch-specific fixes
RuntimeState, so V1 CSV, JSON, Native, ORC, and Parquet readers now fall back to defaultTQueryOptionsinstead of dereferencing a null state.RuntimeStateregression coverage for CSV and Native schema discovery.Validation
doris_be_testbuild compiled and linked successfully; the post-[fix](be) Fix Iceberg row-level deletes across schema evolution #65502 incremental verification completed 1,091/1,091 build targets.PaimonJniScannerTesttests passed.BUILD SUCCESS; checkstyle reported 0 violations.build-support/check-format.shpassed across 4,136 files.