[improvement](be) Add disk cache for external file metadata#63376
Open
xylaaaaa wants to merge 14 commits into
Open
[improvement](be) Add disk cache for external file metadata#63376xylaaaaa wants to merge 14 commits into
xylaaaaa wants to merge 14 commits into
Conversation
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
xylaaaaa
added a commit
to xylaaaaa/doris
that referenced
this pull request
May 20, 2026
### What problem does this PR solve? Issue Number: None Related PR: apache#63376 Problem Summary: Stabilize file meta disk cache related BE unit tests by isolating disk resource limit configuration, initializing FD cache for direct FS storage, and ensuring TTL cache test data carries an expiration time. ### Release note None ### Check List (For Author) - Test: Unit Test - ./run-be-ut.sh --run --filter=FileMetaDiskCacheTest.* -j 8 - ./run-be-ut.sh --run --filter=BlockFileCacheTest.file_cache_path_storage_parse:BlockFileCacheTest.file_meta_disk_cache_initializes_without_data_file_cache:BlockFileCacheTest.meta_queue_can_evict_data_cache_first:BlockFileCacheTest.normal_queue_does_not_evict_meta_cache:BlockFileCacheTest.read_if_cached_returns_downloaded_meta_block -j 8 - Behavior changed: No - Does this need documentation: No
xylaaaaa
added a commit
to xylaaaaa/doris
that referenced
this pull request
May 20, 2026
### What problem does this PR solve? Issue Number: None Related PR: apache#63376 Problem Summary: Large external scans could skip the reader meta cache entirely because the existing file meta cache policy disabled metadata caching when the scan range count exceeded the memory-cache threshold. This prevented the disk-backed file metadata cache from being used for high-file-count workloads. This change separates the L1 memory-cache policy from reader and disk-cache enablement, so large scans can skip the memory cache while still reading from and writing to the disk metadata cache. It also carries the memory-cache policy through row-id file mappings and Iceberg equality readers. ### Release note Improve external Parquet/ORC metadata disk cache behavior for large scans. ### Check List (For Author) - Test: Unit Test - `PATH=/mnt/disk1/chenjunwei/doris_tools/ldb_toolchain_llvm16/bin:$PATH build-support/check-format.sh` - `./run-be-ut.sh --run --filter=FileMetaCacheTest.*:FileMetaDiskCacheTest.*:BlockFileCacheTest.file_meta_disk_cache_initializes_without_data_file_cache:IdFileMapTest.*:ParquetReadLinesTest.*:OrcReadLinesTest.*` - Manual test: `plaud_current.DWD.DWD_POSTHOG_EVENTS_DI` CTE cold/hot profiles with external file meta disk cache enabled and disabled. - Behavior changed: Yes. Large external scans can use disk-backed file metadata cache even when L1 memory metadata cache is skipped. - Does this need documentation: No
Contributor
Author
|
run buildall |
xylaaaaa
added a commit
to xylaaaaa/doris
that referenced
this pull request
May 21, 2026
### What problem does this PR solve? Issue Number: None Related PR: apache#63376 Problem Summary: Restore the existing ExternalFileMappingInfo comment style so the file meta disk cache PR does not include unrelated comment formatting noise. ### Release note None ### Check List (For Author) - Test: No need to test (comment-only change) - `git diff --check` - `PATH=/mnt/disk1/chenjunwei/doris_tools/ldb_toolchain_llvm16/bin:$PATH build-support/check-format.sh` - Behavior changed: No - Does this need documentation: No
Contributor
Author
|
run buildall |
Contributor
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
FE UT Coverage ReportIncrement line coverage `` 🎉 |
xylaaaaa
added a commit
to xylaaaaa/doris
that referenced
this pull request
May 21, 2026
### What problem does this PR solve? Issue Number: None Related PR: apache#63376 Problem Summary: External Parquet/ORC file metadata disk cache profiles exposed hit/miss/write counts but did not report the local disk cache read and write cost. This adds FileFooterReadDiskCacheTime for successful disk-cache reads and FileFooterWriteDiskCacheTime for successful disk-cache writes so cold and hot query profiles can separate local cache overhead from remote footer read and parse time. ### Release note Add profile timers for external Parquet/ORC file metadata disk cache read and write cost. ### Check List (For Author) - Test: Unit Test - PATH=/mnt/disk1/chenjunwei/doris_tools/ldb_toolchain_llvm16/bin:$PATH build-support/check-format.sh - ./run-be-ut.sh --run --filter=FileMetaCacheTest.*:FileMetaDiskCacheTest.*:BlockFileCacheTest.file_meta_disk_cache_initializes_without_data_file_cache:IdFileMapTest.*:ParquetReaderTest.file_footer_disk_cache_time_counters_exist:OrcReaderTest.file_footer_disk_cache_time_counters_exist:ParquetReadLinesTest.*:OrcReadLinesTest.* - Behavior changed: Yes. Query profiles now include external file metadata disk cache read and write timers. - Does this need documentation: No
Contributor
TPC-H: Total hot run time: 31487 ms |
Contributor
TPC-DS: Total hot run time: 169344 ms |
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
Author
|
run buildall |
Contributor
FE UT Coverage ReportIncrement line coverage `` 🎉 |
Contributor
TPC-H: Total hot run time: 31341 ms |
Contributor
TPC-DS: Total hot run time: 170014 ms |
Contributor
TPC-H: Total hot run time: 31410 ms |
Contributor
TPC-DS: Total hot run time: 170252 ms |
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Gabriel39
reviewed
May 25, 2026
| } | ||
|
|
||
| bool FileMetaCache::should_enable_for_reader(int64_t num_scan_ranges) const { | ||
| return should_enable_memory_cache(num_scan_ranges) || |
Contributor
There was a problem hiding this comment.
这个条件只用config::enable_external_file_meta_disk_cache就行
| _current_range, _should_enable_file_meta_cache())); | ||
| auto file_id = id_file_map->get_file_mapping_id(std::make_shared<FileMapping>( | ||
| ((FileScanLocalState*)_local_state)->parent_id(), _current_range, | ||
| _should_enable_file_meta_cache(), _should_enable_file_meta_memory_cache())); |
Contributor
There was a problem hiding this comment.
这里不需要两个开关，我们只需要一个是否开启cache的开关，cache里内存的占比应该是有cache内部机制通过一个比例来控制的
| auto* file_meta_cache_ptr = _should_enable_file_meta_cache() | ||
| ? ExecEnv::GetInstance()->file_meta_cache() | ||
| : nullptr; | ||
| const bool enable_file_meta_memory_cache = _should_enable_file_meta_memory_cache(); |
Gabriel39
reviewed
May 25, 2026
Gabriel39
left a comment
Gabriel39
left a comment
Contributor
There was a problem hiding this comment.
scanner::enable_cache -> 改成通过一个配置项
cache： memory-》memory+disk
xylaaaaa
added a commit
to xylaaaaa/doris
that referenced
this pull request
May 25, 2026
### What problem does this PR solve? Issue Number: None Related PR: apache#63376 Problem Summary: File meta disk cache reader wiring still carried a separate memory-cache switch, so scan-range admission policy leaked into ORC/Parquet reader setup and could also disable disk-cache use. This change uses enable_external_file_meta_disk_cache as the only reader switch and lets FileMetaCache internals handle memory cache admission and eviction. ### Release note External file meta disk cache reader wiring is now controlled by enable_external_file_meta_disk_cache. ### Check List (For Author) - Test: Unit Test - ./run-be-ut.sh --run --filter=FileMetaCacheTest.*:FileMetaDiskCacheTest.* -j 8 - ./run-be-ut.sh --run --filter=ParquetReaderTest.file_footer_disk_cache_time_counters_exist:OrcReaderTest.file_footer_disk_cache_time_counters_exist -j 8 - Behavior changed: Yes. The reader file meta cache path no longer has a separate scanner-level memory-cache switch. - Does this need documentation: No
Gabriel39
reviewed
May 25, 2026
| // Local variables can be required because setSerializedFileTail is an assignment operation, not a reference. | ||
| ObjLRUCache::CacheHandle _meta_cache_handle; | ||
| if (_meta_cache->lookup(file_meta_cache_key, &_meta_cache_handle)) { | ||
| const bool use_memory_cache = _meta_cache->enabled(); |
Contributor
There was a problem hiding this comment.
这些接口都不需要改，我们只是替换了一下cache的实现
Contributor
TPC-H: Total hot run time: 29581 ms |
Contributor
TPC-DS: Total hot run time: 177270 ms |
Contributor
ClickBench: Total hot run time: 25.16 s |
Contributor
Author
|
run buildall |
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary: External file metadata disk cache should be built behind the FileMetaCache boundary instead of exposing a generic complete-value put/get API from BlockFileCache. This change adds a narrow FileMetaCache persistent cache interface, lookup/insert result types, profile counter plumbing, and config gates so upper-layer metadata cache code can depend on FileMetaCache APIs while the storage implementation remains replaceable.
### Release note
None
### Check List (For Author)
- Test: Unit Test
- Added focused FileMetaCache interface tests. Attempted targeted BE UT; initial run hit stale PCH after branch switch, clean run was stopped because it triggered a large full rebuild before reaching the target tests.
- Behavior changed: No
- Does this need documentation: No
Issue Number: None Related PR: apache#64961 Problem Summary: External Parquet and ORC readers only kept file metadata in the in-memory FileMetaCache. This change adds a FileMetaDiskCache implementation behind the FileMetaCache persistent-cache interface, stores serialized Parquet footers and ORC file tails in the existing BlockFileCache through its block APIs, and wires readers to read from disk cache, refill memory cache, and write new persistent entries on misses. It also wires the v2 Parquet reader path through FileScannerV2/TableReader so current Parquet scans use the file meta cache, and enables external file meta disk cache by default. Enable disk cache for external Parquet/ORC file metadata by default. - Test: Regression test / Unit Test / Manual test - python3 build-support/run_clang_format.py --clang-format-executable clang-format --style file --inplace true -j 1 <changed C++ files> - git diff --check -- <changed C++ files> - ./run-be-ut.sh --run --filter=NewParquetReaderTest.UsesFileMetaCacheForFooterMetadata -j 4 - ./run-be-ut.sh --run --filter=FileMetaCache* -j 4 - ./build.sh --be -j 4 - Manual e2e with output FE/BE: local Parquet TVF returned 30000 before and after BE restart, local ORC TVF returned 10, BE runtime config showed enable_external_file_meta_disk_cache=true, and the post-restart Parquet query added no new file meta disk cache miss log. - Behavior changed: Yes. External Parquet/ORC metadata disk cache is enabled by default and can persist file metadata through the file cache. - Does this need documentation: No
Issue Number: None Related PR: apache#64961 Problem Summary: The external file metadata disk cache integration needed stronger correctness boundaries before review. Memory cache keys did not include the metadata format, so legacy Parquet and v2 Parquet could collide while storing incompatible in-memory objects. Persistent entries could also be attempted for files without a stable modification time, and missing disk-cache ranges were treated the same as corrupt entries. This change scopes memory keys by format, adds a PARQUET_V2 persistent format, builds persistent cache identities from path, modification time, and file size, skips persistent cache when the file identity is unstable, handles missing ranges without invalidating concurrent writers, falls back to file reads on invalid persistent payloads, and separates memory-cache gating from persistent-cache gating in file scanners. None - Test: Unit Test / Manual test - git diff --check - git diff --cached --check - ./run-be-ut.sh --run --filter='*FileMeta*' -j 2 - Manual e2e with output FE/BE: local Parquet TVF returned count 25 twice and second profile showed FileFooterHitMemoryCache=1; local ORC TVF returned count 25 twice and second profile showed FileFooterHitMemoryCache=1. Local TVF does not provide modification_time, so persistent disk cache is intentionally skipped on that path and covered by FileMetaCacheDiskTest. - Behavior changed: Yes. File metadata persistent cache now requires a stable modification time, uses format-scoped identities, and avoids invalidating incomplete in-flight cache writes. - Does this need documentation: No
### What problem does this PR solve? Issue Number: None Related PR: apache#63376 Problem Summary: BE UT crashed in OrcReadLinesTest.test0 because read_lines_from_range can initialize readers without an ExecEnv file meta cache object. The scanner memory-cache toggle still dereferenced ExecEnv::file_meta_cache()->enabled() directly. Use the actual FileMetaCache pointer selected for the reader when enabling memory cache and require an initialized cache object before enabling memory or persistent file meta cache. ### Release note None ### Check List (For Author) - Test: - Unit Test: ./run-be-ut.sh --run --filter='OrcReadLinesTest.test0' -j 1 - Unit Test: ./run-be-ut.sh --run --filter='OrcReadLinesTest.*:ParquetReadLinesTest.*:*FileMeta*' -j 1 - Manual test: build-support/clang-format.sh be/src/exec/scan/file_scanner.h be/src/exec/scan/file_scanner.cpp be/src/exec/scan/file_scanner_v2.h be/src/exec/scan/file_scanner_v2.cpp - Manual test: build-support/check-format.sh - Manual test: git diff --check - Manual test: git diff --cached --check - Behavior changed: No - Does this need documentation: No
Issue Number: N/A Related PR: apache#64961 Problem Summary: Row-id fetch scanners read a mapped range without a split source, but Parquet and ORC reader initialization evaluated the normal scan-range memory file metadata cache admission rule and dereferenced the missing split source. This crashed external, P0, and cloud regression jobs. The unknown-mtime Parquet unit test also accidentally used a synthetic nonzero mtime, enabling page cache while asserting it remained disabled. Disable memory file metadata cache for row-id fetch reader initialization and make the unit test pass an explicit unknown mtime. None - Test: Unit Test / Code style check - `./run-be-ut.sh --run --filter='OrcReadLinesTest.*:ParquetReadLinesTest.*:NewParquetReaderTest.RewriteSameLocalPathDoesNotReuseUnknownMtimePageCache' -j 1` - `build-support/check-format.sh` - `git diff --check` - `git diff --cached --check` - Behavior changed: No - Does this need documentation: No
### What problem does this PR solve? Issue Number: close #xxx Related PR: apache#63376 Problem Summary: File meta disk cache already supports external file metadata through the FileMetaCache boundary, but format v2 ORC did not participate in the cache. This change adds an ORC_V2 file meta cache namespace, wires ordinary format v2 ORC readers through FileMetaCache, reuses ORC serialized file tail payloads for memory and persistent cache hits, and refreshes invalid persistent payloads after a successful source-file fallback. Legacy ORC keeps using the ORC namespace so both reader implementations do not share incompatible cached values. ### Release note None ### Check List (For Author) - Test: Unit Test / Manual test - ./build.sh --be -j 10 - ./run-be-ut.sh --run --filter='FileMetaCacheDiskTest.OrcFormatsUseIndependentKeys:NewOrcReaderTest.*FileMetaCache*:NewOrcReaderTest.OversizedPersistentPayloadRemainsMemoryCacheEligible' -j 10 - git diff --check - Behavior changed: Yes. Format v2 ORC file metadata can use the existing external file meta disk cache when enabled. - Does this need documentation: No
### What problem does this PR solve? Issue Number: None Related PR: apache#64961 Problem Summary: Runtime activation could enable persistent metadata lookups after startup without constructing the required cache objects. Nested Iceberg and transactional Hive delete readers also ignored the parent memory-cache policy, and a capacity rejection during a multi-block write could leave downloaded payload blocks behind. Separate startup configuration from the runtime size gate, propagate the reader policy, preclaim complete entries before writing, and cover real disk reinitialization. ### Release note None ### Check List (For Author) - Test: Unit Tests added; modified production and test objects compiled with ASAN and -Werror; full BE UT execution pending the local rebuild - Behavior changed: Yes, runtime cache-size activation is supported and failed writes no longer leave partial entries - Does this need documentation: No
### What problem does this PR solve? Issue Number: None Related PR: apache#63376 Problem Summary: The file metadata persistent cache interface had no direct unit coverage. Add boundary-level tests for successful round trips, invalidation, profile counters, and read/write failure fallback without depending on BlockFileCache internals. ### Release note None ### Check List (For Author) - Test: Unit Test object compiled with ASAN and -Werror; full execution pending the local BE rebuild - Behavior changed: No - Does this need documentation: No
### What problem does this PR solve? Issue Number: None Related PR: apache#64961 Problem Summary: Iceberg format-v2 data readers received the shared file metadata cache and memory-cache policy, but format-v2 position and equality delete readers were constructed without either value. Their Parquet and ORC footer metadata therefore bypassed the persistent cache entirely. Pass the cache, memory policy, and timestamp mapping option through the delete-reader construction boundary and add enabled/disabled behavior coverage. ### Release note None ### Check List (For Author) - Test: Unit Test added; modified v2 production and test objects compiled with ASAN and -Werror; full execution pending the local BE rebuild - Behavior changed: Yes, Iceberg v2 delete-file metadata now uses the configured file metadata cache - Does this need documentation: No
### What problem does this PR solve? Issue Number: None Related PR: apache#64961 Problem Summary: The disk-cache restart test exercised FSFileCacheStorage without initializing ExecEnv's FDCache, so the restored read dereferenced a null cache under ASAN. Initialize and release the FDCache only when this test owns it, and shorten the TTL worker intervals so destruction does not wait six minutes. ### Release note None ### Check List (For Author) - Test: Unit Test - 20 focused BE tests passed under ASAN - Behavior changed: No - Does this need documentation: No
### What problem does this PR solve? Issue Number: None Related PR: apache#64961 Problem Summary: FileMetaCache context lookups could leave a reused memory-cache handle pinned after a persistent hit or miss. Persistent reads also accepted entries larger than the current runtime size limit. Reset the output handle before every context lookup, invalidate oversized persistent entries, and add regression coverage for both cases. ### Release note None ### Check List (For Author) - Test: Unit Test - ContextLookupClearsReusedMemoryHandle and OversizedPersistentReadFallsBackToMiss (ASAN) - Behavior changed: Yes. Non-memory lookup results now return an invalid handle, and oversized persistent entries are treated as cache misses. - Does this need documentation: No
### What problem does this PR solve? Issue Number: N/A Related PR: apache#64961 Problem Summary: The external file metadata disk cache used a hand-written entry header and could retain incomplete or corrupted multi-block entries after write failures, partial eviction, or invalid payloads. File identities also needed to distinguish HDFS nameservices and S3 endpoints, and the ORC v2 invalid-payload fallback could serialize an oversized tail that was ineligible for persistence. This change uses a protobuf header, revalidates and removes invalid entries as a complete key, serializes same-key replacement in the FileMetaDiskCache wrapper, guards cache restoration, aligns file-system identity across readers, and avoids the oversized ORC v2 rewrite. BlockFileCache internals and interfaces are unchanged. ### Release note Improve the robustness and isolation of the default-enabled external Parquet/ORC metadata disk cache. ### Check List (For Author) - Test: Code style check - build-support/clang-format.sh - build-support/check-format.sh - git diff --check - Full BE build and focused BE UT are pending CI - Behavior changed: Yes. Invalid or incomplete persisted metadata entries are removed and rebuilt, and cache keys include the effective file-system identity. - Does this need documentation: Yes. A usage document will describe capacity, configuration, profile counters, and fallback behavior.
xylaaaaa
force-pushed
the
codex/file-meta-cache-disk-pr
branch
from
6bba4f0 to
f31f23a
Compare
Contributor
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 29631 ms |
### What problem does this PR solve? Issue Number: N/A Related PR: apache#64961 Problem Summary: The Parquet v2 test reader helper added an is_immutable argument before the FileMetaCache pointer. Several new persistent-cache tests omitted that boolean, so the BE UT build converted the cache pointer to bool and then attempted to pass bool as FileMetaCache*. Pass the explicit false value at all affected call sites so the tests compile with the intended immutable-file setting. ### Release note None ### Check List (For Author) - Test: Manual test - Compiled test/CMakeFiles/doris_be_test.dir/format_v2/parquet/parquet_reader_test.cpp.o with the local ASAN_UT build - build-support/check-format.sh - git diff --check - Behavior changed: No - Does this need documentation: No
Contributor
TPC-DS: Total hot run time: 178547 ms |
Contributor
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 29413 ms |
Contributor
TPC-DS: Total hot run time: 177684 ms |
Contributor
ClickBench: Total hot run time: 25.58 s |
### What problem does this PR solve? Issue Number: N/A Related PR: apache#63376 Problem Summary: The Parquet v2 reader test helper defaults mtime to 123 for file metadata cache coverage. UnknownMtimeSkipsPageCacheForMutableFile inherited that stable mtime after the branches were combined, so the page cache was enabled and CI observed PageReadCount=2. Explicitly pass mtime=0 so the test exercises an unknown-version mutable file as intended. ### Release note None ### Check List (For Author) - Test: Code style check - build-support/check-format.sh - git diff --check - Targeted BE UT pending CI - Behavior changed: No. This is a test-only correction. - Does this need documentation: No
Contributor
Author
|
run buildall |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: N/A
Related PR: #64961
Problem Summary:
External Parquet/ORC footer metadata was only cached in BE memory. After memory-cache eviction or a BE restart, repeated scans had to fetch and parse the same footer from remote storage again.
This PR builds on the
FileMetaCachepersistent-cache interface from #64961 and adds a disk-backed implementation for external file metadata. The feature switch is enabled by default. The implementation stays behind theFileMetaCacheboundary and reuses existingBlockFileCacheAPIs and its INDEX queue; this PR does not modifyBlockFileCacheinternals or interfaces.Lookup and write flow:
ObjLRUCachefirst.FileMetaCacheasksFileMetaDiskCachefor the serialized footer stored in the existing local file cache.Main changes:
FileMetaDiskCache, implementingFileMetaPersistentCacheentirely aboveBlockFileCache.enable_external_file_meta_disk_cache(defaulttrue, startup-only) andexternal_file_meta_disk_cache_max_entry_bytes(default 64 MiB).file_cache_pathand the existing INDEX queue. Effective disk caching requires a positive file-cachetotal_sizeand nonzero INDEX capacity; no independent metadata-cache quota or directory is added.Release note
Add a default-enabled disk-backed cache for eligible external Parquet/ORC metadata. It reuses the existing local file cache INDEX queue and is controlled by
enable_external_file_meta_disk_cacheandexternal_file_meta_disk_cache_max_entry_bytes.Check List (For Author)
./run-be-ut.sh --run --filter='*FileMeta*' -j 2./build.sh --be -j 4build-support/clang-format.shbuild-support/check-format.shgit diff --checkandgit diff --cached --checkrun buildalloutput/FE/BE: Parquet and ORC local TVFs returned identical results on repeated scans and showed memory-footer hits. Local TVF ranges do not provide a positive modification time, so L2 is intentionally skipped on that path.