Repository navigation
Conversation
Index.data called DataCloud.get_remote once per output, and each call builds a new filesystem. With many directory outputs and an empty local cache (fresh clone, CI), `dvc data status --not-in-remote` instantiated one cloud client per output and memory grew with the number of outputs. Resolve each remote name once per index build and share the Remote between the outputs that use it. Fixes treeverse#11121
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #11124 +/- ##
==========================================
+ Coverage 90.68% 90.98% +0.29%
==========================================
Files 504 505 +1
Lines 39795 41158 +1363
Branches 3141 3265 +124
==========================================
+ Hits 36087 37446 +1359
- Misses 3042 3072 +30
+ Partials 666 640 -26 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
❗ I have followed the Contributing to DVC checklist.
📖 If this PR requires documentation updates, I have created a separate PR (or issue, at least) in dvc.org and linked it here. (No docs change needed.)
Fixes #11121
Index.datacalledDataCloud.get_remote(out.remote)once per output, and every call builds a new filesystem andRemote. EachRemotestays referenced from the index storage map, so with an empty local cachedata status --not-in-remoteends up with one cloud client per directory output, as measured in the issue.This change resolves each remote name once per index build (a small dict passed from
Index.datato_load_storage_from_out) and shares thatRemotebetween all outputs that use it. Outputs with a differentremote:still get their own.DataCloud.get_remoteitself is unchanged, so other callers (push/pull/gc with a--remote) behave as before.I chose local memoisation over caching inside
DataCloud.get_remotebecauseget_remoteis also used with config overrides and by long-livedRepoobjects, where a cache would need invalidation on config changes. Happy to move it there if you prefer.Test:
tests/func/test_repo_index.py::test_data_index_reuses_remote_per_namebuilds 4 directory outputs, 3 on the default remote and 1 on a second remote, and checks thatget_remoteis called once per name and that the 3 default-remote outputs share one ODB. It fails without the change (['', '', '', 'second'] == ['', 'second']) and passes with it;test_repo_index.py+test_data_status.pypass locally.