Skip to content

Add cache for single measurement validation#17899

Merged
jt2594838 merged 9 commits into
masterfrom
add_measurement_cache
Jun 11, 2026
Merged

Add cache for single measurement validation#17899
jt2594838 merged 9 commits into
masterfrom
add_measurement_cache

Conversation

@jt2594838

@jt2594838 jt2594838 commented Jun 10, 2026

Copy link
Copy Markdown
Contributor

Description

This PR adds a configurable cache for PathUtils#checkAndReturnSingleMeasurement so repeated measurement validation can reuse previous check results. The cache stores both valid and invalid validation results and can be disabled by setting the cache size to 0.

It also adds configuration plumbing for integration tests and an ignored performance IT that can be run manually to compare end-to-end write performance with the cache disabled and enabled.

Test Report

Local verification:

mvn spotless:apply -pl integration-test -P with-integration-tests
mvn spotless:apply -pl iotdb-core/datanode
mvn test-compile -DskipUTs -Drat.skip=true -pl integration-test -am -P with-integration-tests

Performance averages, 3 repeats per group, 80,000 rows per experiment:

measurementCount cacheSize Relation Cache Disabled Cache Enabled Enabled/Disabled
1000 500 cacheSize < measurementCount 4794 ms 4384 ms 0.914
2000 2000 cacheSize = measurementCount 9676 ms 8255 ms 0.853
4000 8000 cacheSize > measurementCount 19980 ms 17223 ms 0.862

@codecov

codecov Bot commented Jun 10, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 52.63158% with 18 lines in your changes missing coverage. Please review.
✅ Project coverage is 40.83%. Comparing base (1d8b23a) to head (db813af).
⚠️ Report is 32 commits behind head on master.

Files with missing lines Patch % Lines
...java/org/apache/iotdb/commons/utils/PathUtils.java 21.73% 18 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##             master   #17899      +/-   ##
============================================
+ Coverage     40.80%   40.83%   +0.02%     
+ Complexity     2612      318    -2294     
============================================
  Files          5200     5245      +45     
  Lines        353687   363239    +9552     
  Branches      45252    46782    +1530     
============================================
+ Hits         144326   148332    +4006     
- Misses       209361   214907    +5546     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@Caideyipi

Copy link
Copy Markdown
Collaborator

I think this part has a correctness issue: checkTsFileResourceExists currently warns for every .tsfile because it only checks the filename suffix and never checks whether the corresponding resource file exists. For a normal snapshot containing both xxx.tsfile and xxx.tsfile.resource, this still logs The associated resource file ... is not found in the snapshot, which is a false warning. Please check
ew File(file.getAbsolutePath() + TsFileResource.RESOURCE_SUFFIX).exists() before warning, or remove this unrelated snapshot change from this PR.

@sonarqubecloud

Copy link
Copy Markdown

@jt2594838 jt2594838 merged commit 1a3548e into master Jun 11, 2026
44 of 46 checks passed
@jt2594838 jt2594838 deleted the add_measurement_cache branch June 11, 2026 11:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants