Skip to content

[SYSTEMDS-3891] OOC k-Means Support#2442

Open
janniklinde wants to merge 1 commit intoapache:mainfrom
janniklinde:ooc/kMeans
Open

[SYSTEMDS-3891] OOC k-Means Support#2442
janniklinde wants to merge 1 commit intoapache:mainfrom
janniklinde:ooc/kMeans

Conversation

@janniklinde
Copy link
Contributor

This patch adds support for the k-means builtin script in the out-of-core backend by introducing new instructions, handling parfor.

Additionally, this patch improves performance of the OOCLRUCacheScheduler under memory pressure by debouncing cleanup runs, which previously caused stalls due to long synchronization barriers.

Finally, this patch provides bugfixes for various instructions and stream lifecycle handling.

Performance Improvements
Bugfixes
@codecov
Copy link

codecov bot commented Mar 3, 2026

Codecov Report

❌ Patch coverage is 58.14932% with 398 lines in your changes missing coverage. Please review.
✅ Project coverage is 71.78%. Comparing base (743c902) to head (6e7439f).
⚠️ Report is 3 commits behind head on main.

Files with missing lines Patch % Lines
...instructions/ooc/MatrixIndexingOOCInstruction.java 60.96% 63 Missing and 26 partials ⚠️
.../sysds/runtime/ooc/cache/OOCLRUCacheScheduler.java 64.32% 36 Missing and 25 partials ⚠️
...he/sysds/runtime/instructions/ooc/OOCWatchdog.java 0.00% 56 Missing ⚠️
...sysds/runtime/instructions/ooc/OOCInstruction.java 58.47% 33 Missing and 16 partials ⚠️
...rg/apache/sysds/runtime/ooc/stats/OOCEventLog.java 0.00% 37 Missing ⚠️
.../sysds/runtime/instructions/ooc/CachingStream.java 68.42% 20 Missing and 10 partials ⚠️
.../runtime/instructions/ooc/UnaryOOCInstruction.java 78.57% 7 Missing and 8 partials ⚠️
...runtime/instructions/ooc/BinaryOOCInstruction.java 0.00% 11 Missing and 3 partials ⚠️
...sds/runtime/controlprogram/ParForProgramBlock.java 72.97% 4 Missing and 6 partials ⚠️
...ds/runtime/instructions/ooc/TeeOOCInstruction.java 0.00% 6 Missing ⚠️
... and 15 more
Additional details and impacted files
@@             Coverage Diff              @@
##               main    #2442      +/-   ##
============================================
+ Coverage     71.72%   71.78%   +0.06%     
- Complexity    48011    48219     +208     
============================================
  Files          1554     1554              
  Lines        184671   185352     +681     
  Branches      36248    36381     +133     
============================================
+ Hits         132453   133058     +605     
- Misses        41964    41998      +34     
- Partials      10254    10296      +42     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: In Progress

Development

Successfully merging this pull request may close these issues.

1 participant