[improvement](be) Support dynamic file cache capacity#65477
Open
gavinchou wants to merge 1 commit into
Open
Conversation
### What problem does this PR solve?
Issue Number: N/A
Related PR: N/A
Problem Summary: File cache paths configured with zero capacity only resolved the disk size at startup. Runtime disk expansion or shrink did not update the effective capacity or queue limits, manual reset returned free-form text, and synchronous shrink could hold the cache lock while deleting many blocks. This change adds explicit AUTO/MANUAL capacity policy, a shared disk snapshot and resize planner, atomic BlockFileCache quota publication, generation checks for automatic refresh, batched asynchronous shrink, on-demand Factory aggregation, and structured reset/info HTTP responses.
### Release note
Support automatic runtime file cache capacity updates for disk-backed paths configured with zero capacity. The file cache reset API now accepts zero to restore AUTO mode and returns structured capacity details, and op=info exposes per-path capacity, disk, queue, and resize state.
### Check List (For Author)
- Test:
- Build: source custom_env.sh && sh build.sh --be -j 16
- Unit Test: BlockFileCacheTest.* and FileCacheActionTest.* (141 tests, 138 passed, 3 disabled)
- New Unit Test smoke: capacity policy, SyncPoint disk resize, Factory preflight, and FileCacheActionTest.* (16 passed)
- Full BE Unit Test: 9495 tests, 9438 passed, 57 skipped
- Format: build-support/check-format.sh
- Behavior changed: Yes. Disk cache capacity 0 is a persistent AUTO mode, reset capacity 0 restores AUTO mode, queue quotas update with capacity, and shrink converges asynchronously.
- Does this need documentation: Yes. File Cache operations documentation should describe AUTO mode, structured reset output, and op=info.
gavinchou
requested review from
Gabriel39,
liaoxin01,
luwei16 and
morningman
as code owners
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
Contributor
Author
|
/review |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Reviewed PR #65477 for dynamic file cache capacity. I did not find a blocking correctness issue or any substantiated inline finding to raise.
Critical checkpoint conclusions:
- Goal and scope: The PR implements AUTO/MANUAL file cache capacity policy, runtime disk refresh, structured reset/info responses, and asynchronous over-capacity shrink. The implementation is focused on BE file-cache capacity handling.
- Data correctness: Capacity, policy, queue quotas, generation, and pending-eviction state are published under the cache mutex; stale auto-refresh plans are guarded by generation checks. I did not find a path where a stale auto refresh overwrites a later manual reset.
- Concurrency and lifecycle: Factory reset/reload/create are serialized with
_reset_mtx, per-cache mutation usesBlockFileCache::_mutex, and shrink work is handled by the existing background eviction thread. No concrete lock-order or publication-order bug was substantiated. - Configuration and compatibility:
CachePath::init_settings()preserves zero disk capacity as AUTO mode, manual disk capacities are clamped by observed disk size, and memory AUTO is rejected. I did not find persisted-format or FE/BE protocol compatibility concerns. - Parallel paths: Startup create, reload, HTTP reset, background monitor refresh, info reporting, and multi-cache aggregation were checked. Multi-cache reset preflights plans before applying them.
- Tests: New BE unit and HTTP action tests cover capacity policy building, auto refresh, stale generation skipping, factory aggregation/preflight, and structured HTTP reset/info output. I could not run BE tests in this runner because
.worktree_initializedandthirdparty/installedare absent, so validation was static-only;git diff --checkpassed. - Observability: The PR adds runtime info fields and bvar metrics for capacity mode, disk capacity, pending eviction, and auto-resize success/failure; I did not find a missing-observability issue for this change.
- User focus: No additional user-provided review focus was supplied.
Subagent conclusions: optimizer-rewrite appended no candidate findings, and tests-session-config appended no candidate findings. In convergence round 1, both subagents reviewed the same ledger and empty proposed final comment set and returned NO_NEW_VALUABLE_FINDINGS; no inline comments were submitted.
Contributor
TPC-H: Total hot run time: 30019 ms |
Contributor
TPC-DS: Total hot run time: 180442 ms |
Contributor
ClickBench: Total hot run time: 24.97 s |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: N/A
Related PR: N/A
Problem Summary:
File cache paths configured with zero capacity only resolved the disk size at startup. Runtime disk expansion or shrink did not update the effective capacity or the four queue limits. Manual reset returned free-form text, and shrinking synchronously could hold the cache lock while deleting many blocks.
This PR adds:
op=infoendpoint.Release note
Support automatic runtime file cache capacity updates for disk-backed paths configured with zero capacity. The file cache reset API now accepts zero to restore AUTO mode and returns structured capacity details.
op=infoexposes per-path capacity, disk, queue, and resize state.Check List (For Author)
Test
BlockFileCacheTest.*:FileCacheActionTest.*: 141 tests, 138 passed, 3 disabled.source custom_env.sh && sh build.sh --be -j 16.build-support/check-format.sh.Behavior changed:
Does this need documentation?
op=info.Check List (For Reviewer who merge this PR)