feat: version registry that keeps release lists between runs by roxblnfk · Pull Request #119 · php-internal/dload

roxblnfk · 2026-09-12T20:01:19Z

🔍 What was changed

DLoad now keeps a local version registry: a provider-neutral database of the releases and assets of every known repository, one readable directory per repository under the per-user cache directory: an index with the metadata and the tags, plus the releases in segments of 100 that are read only when the run reaches them and rewritten only when they change (cache-dir / DLOAD_CACHE_DIR to move it, cache-ttl=0 to disable).

Versions never expire. Only the last check of a repository has a TTL (cache-ttl, default 600 s); within it dload get makes no API request at all.
Release pages stay lazy: the first run fetches only the pages the requested version needs, older releases are loaded on demand later, and a stale check fetches only the pages with releases newer than the stored ones.
New dload get --refresh ignores the TTL once; new dload cache:clear [software...] drops the repositories a software package is served from, or the whole registry after a confirmation in a terminal (--force or a non-interactive run skips it).
A release deleted upstream after it was stored no longer breaks the run: its assets answer with a clear AssetNotFoundException instead of "repository not found, check your token", the release is dropped from the registry, and the list is fetched again in the same run before giving up.
A release deleted from the middle of the stored list is dropped on the next check instead of lingering as a phantom entry, so the offset used to load older pages stays correct. Repository identities are normalized (lower case, no surrounding slashes) and a stored record is served only for the identity it was written for, so sanitized names and case-insensitive file systems cannot mix two repositories.
Asset records carry the sha256:<hex> digest GitHub reports. Nothing reads it yet; it is stored for a future binary cache with checksum verification.
GitHub draft releases are never served: the registry keeps only their tags as hidden placeholders, so the stored count keeps matching the API paging and no page is requested twice. A check that meets a release the API adapter cannot decode leaves the stored list untouched instead of dropping the release as deleted. A rate limit that blocks a check is reported once per run without any verbosity flag, since the stored list may lack newer releases.
DLOAD_CACHE_DIR and DLOAD_CACHE_TTL now override the dload.xml attributes, and the command line overrides both: the config inflector consults sources by a fixed rank instead of attribute declaration order.
Every releases page is now requested once with per_page=100, and destroy() no longer iterates the lazy collection, which used to load every remaining page after each download.
The registry is keyed by the path the repository reports, not by the configured URI, so a full URL in dload.xml lands on the same record as owner/repo. Saves and removals hold a per-repository lock in locks/, so two concurrent runs cannot interleave their files. The GitHub token is sent to github.com and githubusercontent.com hosts only, since asset URLs are read back from files on disk.

How it works

Check: when the last check is older than the TTL (or --refresh), the newest pages are fetched until one contains a stored release; the first page always overwrites what is stored, so assets attached later are picked up.
Serve: stored releases are yielded without a request; a failed check with stored releases falls back to them.
Extend: when the consumer runs past the stored releases and the list is not marked complete, older pages are fetched by offset and appended to the record.

Why?

Every run asked GitHub or GitLab for the release list, and every request counted against the API rate limit; unauthenticated CI matrices hit the 60 requests/hour limit quickly, and nothing was remembered between runs. Caching raw HTTP responses (#118) would have kept credentials-bound answers keyed by URL; storing release metadata keeps only tags, names and download links, so the directory can be shared between machines or a CI cache freely.

Checklist

Supersedes fix: request every releases page once and cache release listings #118
How was this tested:
- Tested manually: against dolthub/dolt (644 releases) the first run makes one API request, the second run none, dolt:~1.20.0 loads the tail on demand, and a registry entry with poisoned asset links is dropped and the next release is downloaded
- Unit tests added
- composer test, composer psalm, composer cs:diff pass locally

Documentation

README (and the ru/es/zh translations) gained a "Version Registry" section; dload.xsd documents cache-dir and cache-ttl; both state that the environment variable takes precedence over the XML attribute.

codecov · 2026-09-12T20:02:12Z

Codecov Report

❌ Patch coverage is 83.18584% with 114 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...e/Repository/Internal/GitHub/Api/RepositoryApi.php	0.00%	31 Missing ⚠️
...e/Repository/Internal/GitLab/Api/RepositoryApi.php	0.00%	30 Missing ⚠️
...dule/Common/Internal/Injection/ConfigInflector.php	0.00%	9 Missing ⚠️
...itory/Internal/GitHub/Api/Response/ReleaseInfo.php	0.00%	9 Missing ⚠️
...ository/Internal/GitHub/Api/Response/AssetInfo.php	0.00%	8 Missing ⚠️
...itory/Internal/GitLab/Api/Response/ReleaseInfo.php	0.00%	6 Missing ⚠️
...odule/Repository/Internal/GitHub/GitHubRelease.php	0.00%	5 Missing ⚠️
...odule/Repository/Internal/GitLab/GitLabRelease.php	0.00%	5 Missing ⚠️
src/Module/Registry/Internal/CacheDirectory.php	88.88%	2 Missing ⚠️
src/Module/Registry/Record/ReleasePage.php	0.00%	2 Missing ⚠️
... and 7 more

Files with missing lines	Coverage Δ
src/Bootstrap.php	`98.07% <100.00%> (ø)`
src/Command/CacheClear.php	`97.50% <100.00%> (ø)`
src/Module/Config/Schema/Cache.php	`0.00% <ø> (ø)`
src/Module/Downloader/Downloader.php	`99.57% <100.00%> (ø)`
src/Module/Downloader/Exception/ReleaseGone.php	`0.00% <ø> (ø)`
src/Module/Downloader/Internal/DownloadContext.php	`0.00% <ø> (ø)`
...c/Module/Registry/Internal/FileRegistryStorage.php	`99.01% <100.00%> (ø)`
src/Module/Registry/Record/AssetRecord.php	`95.00% <100.00%> (ø)`
src/Module/Registry/Record/RepositoryRecord.php	`99.35% <100.00%> (ø)`
...le/Repository/Exception/AssetNotFoundException.php	`0.00% <ø> (ø)`
... and 27 more

... and 69 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

fix: request every releases page once with `per_page=100` fix: stop `destroy()` from loading the remaining release pages Every run asked GitHub or GitLab for the release list and spent the API rate limit on it. Releases are now kept in a local version registry: a provider-neutral database of the releases and assets of every known repository, one JSON file per repository, on by default in the per-user cache directory. Versions never expire; only the last check of a repository has a TTL (`cache-ttl`, default 600 s), within which `dload get` costs no API request. A stale check fetches only the newest pages until a stored release is reached. Pages stay lazy: the first run loads what the requested version needs, older releases are fetched on demand and appended. A failed check falls back to the stored releases. `dload get --refresh` ignores the TTL once, `dload cache:clear [software...]` drops records. The page loader used to build a paginator per page and probe the next one, so every page but the first was requested twice; `destroy()` iterated the whole lazy collection and loaded every remaining page after each download. Co-Authored-By: Dmitriy Derepko <xepozz@list.ru> Assisted-By: Claude Fable 5.1 <noreply@anthropic.com>

…them A 404 for a release asset was reported as a missing repository, with advice about tokens and addresses, and the deleted release stayed in the registry until a later check happened to overwrite it. Asset URLs now raise `AssetNotFoundException`; when every matching asset of a release is gone, the downloader drops the release from the registry, marks the repository for a check, and fetches the list once more before giving up. Assisted-By: Claude Fable 5.1 <noreply@anthropic.com>

…the head A release deleted from the middle of the stored list survived the check as a phantom at the end of the list. It inflated the release count the registry uses as the offset for loading older pages, so one real release was skipped and never loaded until the cache was cleared. The repository identity is normalized to lower case without surrounding slashes, as GitHub and GitLab resolve paths case-insensitively, and the file storage rejects a record whose identity differs from the requested one: sanitized names and case-insensitive file systems can map two identities onto one file. Windows device names such as `nul` or `com1` get a prefix, because Windows opens the device whatever the extension and the write silently failed. Assisted-By: Claude Fable 5.1 <noreply@anthropic.com>

docs: state that XML cache attributes take precedence over the environment Without arguments the command wiped the registry immediately, unlike `init`, which confirms destructive actions in a terminal. A non-interactive run and `--force` still go ahead, so CI scripts need no change. Assisted-By: Claude Fable 5.1 <noreply@anthropic.com>

The psalm job of the pull request failed on level 1 findings: mixed assignments while decoding stored records, flow analysis across `goto` in the downloader, and generator return types. Assisted-By: Claude Fable 5.1 <noreply@anthropic.com>

The stale-on-error test used a complete record, so the tail loading path was never entered; the downloader test with a working asset never reached the branch where every asset fails for a reason other than a missing file. Tail loading from inside a page and the GitLab tail path had no test at all. Assisted-By: Claude Fable 5.1 <noreply@anthropic.com>

GitHub returns a `sha256:<hex>` digest for release assets. Nothing reads it yet: the field is stored now so that a future binary cache can verify a downloaded file without re-fetching the release lists. The field is optional, so existing registry files stay readable. Assisted-By: Claude Fable 5.1 <noreply@anthropic.com>

fix(registry): keep GitHub draft releases out of the shared registry feat(registry): report an API rate limit once per run in plain sight refactor(registry): pass Path between services and keep the temp fallback per user docs: note draft releases and mid-list insertions as registry limits One JSON file per repository was read and rewritten whole on every run, which grows to megabytes for repositories with thousands of releases and assets. The record now holds an index with the tags of every segment, so counting and membership never touch the releases, and each segment of at most 100 releases is a file of its own that is read when the iteration reaches it and written only when it changed. The partial segment sits at the head, where every check adds its releases, so the full segments behind it are never rewritten. Drafts are visible to the token holder only, yet the registry is meant to be shared and cached in CI, so they are not stored. The `attach()` argument order now matches `forget()`, and the registry types are `@internal` like the rest of the tool. Assisted-By: Claude Fable 5.1 <noreply@anthropic.com>

… line override both The inflector took the first attribute in declaration order, so `cache-ttl` in a committed `dload.xml` silenced `DLOAD_CACHE_TTL=0` in CI. Sources are now consulted by a fixed rank: command line, environment, php.ini, then the configuration file. Assisted-By: Claude Fable 5.1 <noreply@anthropic.com>

Copilot

🟡 Changes recommended

Repository identity mismatches, unsafe cached asset URLs, pagination duplication, and non-transactional concurrent writes affect correctness and security.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Adds a persistent, provider-neutral version registry to reduce repeated GitHub and GitLab release API requests.

Changes:

Persists lazy, segmented release metadata with TTL and refresh support.
Adds cache clearing and recovery from deleted releases/assets.
Adds configuration, documentation, provider integration, and extensive tests.

File summaries

File	Description
`tests/Unit/Module/Repository/Internal/GitLab/Stub/PagedClientStub.php`	Adds paginated GitLab API stub.
`tests/Unit/Module/Repository/Internal/GitLab/GitLabRepositoryTest.php`	Tests GitLab pagination and caching.
`tests/Unit/Module/Repository/Internal/GitLab/FactoryTest.php`	Injects registry into factory tests.
`tests/Unit/Module/Repository/Internal/GitLab/Api/ResponseValidatorTest.php`	Tests missing GitLab assets.
`tests/Unit/Module/Repository/Internal/GitHub/Stub/PagedClientStub.php`	Adds paginated GitHub API stub.
`tests/Unit/Module/Repository/Internal/GitHub/GitHubRepositoryTest.php`	Tests GitHub pagination and caching.
`tests/Unit/Module/Repository/Internal/GitHub/Api/ResponseValidatorTest.php`	Tests missing GitHub assets.
`tests/Unit/Module/Registry/Stub/RecordingRegistry.php`	Records registry interactions.
`tests/Unit/Module/Registry/Stub/InMemoryRegistryStorage.php`	Adds in-memory registry storage.
`tests/Unit/Module/Registry/Stub/ArrayReleaseSource.php`	Models paginated release changes.
`tests/Unit/Module/Registry/StoredVersionRegistryTest.php`	Covers registry lifecycle behavior.
`tests/Unit/Module/Registry/RepositoryRecordTest.php`	Covers records and segmentation.
`tests/Unit/Module/Registry/PassThroughRegistryTest.php`	Tests disabled-registry behavior.
`tests/Unit/Module/Registry/FileRegistryStorageTest.php`	Covers filesystem persistence.
`tests/Unit/Module/Registry/CacheDirectoryTest.php`	Tests platform cache resolution.
`tests/Unit/Module/Downloader/Stub/SequenceRepositoryFactoryStub.php`	Models changing repository listings.
`tests/Unit/Module/Downloader/Stub/GoneAssetStub.php`	Models deleted assets.
`tests/Unit/Module/Downloader/DownloaderTest.php`	Tests deleted-release recovery.
`tests/Integration/Module/Registry/VersionRegistryBindingTest.php`	Verifies container bindings.
`tests/Integration/Command/CacheClearTest.php`	Tests cache clearing command.
`tests/Acceptance/DLoadTest.php`	Isolates acceptance-test cache.
`src/Module/Repository/Internal/ResponseValidator.php`	Distinguishes missing assets.
`src/Module/Repository/Internal/GitLab/GitLabRepository.php`	Integrates GitLab with registry.
`src/Module/Repository/Internal/GitLab/GitLabReleaseSource.php`	Adapts GitLab pagination.
`src/Module/Repository/Internal/GitLab/GitLabRelease.php`	Restores releases from records.
`src/Module/Repository/Internal/GitLab/GitLabAsset.php`	Restores assets from records.
`src/Module/Repository/Internal/GitLab/Factory.php`	Injects registry into repositories.
`src/Module/Repository/Internal/GitLab/Api/ResponseValidator.php`	Detects GitLab asset URLs.
`src/Module/Repository/Internal/GitLab/Api/Response/ReleaseInfo.php`	Maps releases to records.
`src/Module/Repository/Internal/GitLab/Api/Response/AssetInfo.php`	Maps assets to records.
`src/Module/Repository/Internal/GitLab/Api/RepositoryApi.php`	Adds lazy 100-item pages.
`src/Module/Repository/Internal/GitHub/GitHubRepository.php`	Integrates GitHub with registry.
`src/Module/Repository/Internal/GitHub/GitHubReleaseSource.php`	Adapts GitHub pagination.
`src/Module/Repository/Internal/GitHub/GitHubRelease.php`	Restores releases from records.
`src/Module/Repository/Internal/GitHub/GitHubAsset.php`	Restores assets from records.
`src/Module/Repository/Internal/GitHub/Factory.php`	Injects registry into repositories.
`src/Module/Repository/Internal/GitHub/Api/ResponseValidator.php`	Detects GitHub asset URLs.
`src/Module/Repository/Internal/GitHub/Api/Response/ReleaseInfo.php`	Maps releases to records.
`src/Module/Repository/Internal/GitHub/Api/Response/AssetInfo.php`	Stores asset digests.
`src/Module/Repository/Internal/GitHub/Api/RepositoryApi.php`	Adds lazy filtered pages.
`src/Module/Repository/Internal/Collection.php`	Exposes loaded items safely.
`src/Module/Repository/Internal/CachedGenerator.php`	Exposes generated cache contents.
`src/Module/Repository/Exception/AssetNotFoundException.php`	Adds missing-asset exception.
`src/Module/Registry/VersionRegistry.php`	Defines registry operations.
`src/Module/Registry/RepositoryId.php`	Defines normalized identities.
`src/Module/Registry/ReleaseSource.php`	Defines lazy release sources.
`src/Module/Registry/RegistryStorage.php`	Defines persistence operations.
`src/Module/Registry/Record/RepositoryRecord.php`	Implements repository state.
`src/Module/Registry/Record/ReleaseSegment.php`	Implements lazy segments.
`src/Module/Registry/Record/ReleaseRecord.php`	Defines release metadata.
`src/Module/Registry/Record/ReleasePage.php`	Defines source pages.
`src/Module/Registry/Record/AssetRecord.php`	Defines asset metadata.
`src/Module/Registry/Internal/StoredVersionRegistry.php`	Implements cached listings.
`src/Module/Registry/Internal/PassThroughRegistry.php`	Implements disabled mode.
`src/Module/Registry/Internal/FileRegistryStorage.php`	Persists segmented records.
`src/Module/Registry/Internal/CacheDirectory.php`	Resolves default cache path.
`src/Module/Downloader/Exception/ReleaseGone.php`	Signals stale release listings.
`src/Module/Downloader/Downloader.php`	Retries after deleted releases.
`src/Module/Config/Schema/Cache.php`	Adds cache configuration.
`src/Module/Common/Internal/Injection/ConfigInflector.php`	Prioritizes config sources.
`src/Command/Get.php`	Adds `--refresh`.
`src/Command/CacheClear.php`	Adds cache clearing command.
`src/Bootstrap.php`	Registers registry services.
`README.md`	Documents the registry.
`README-zh.md`	Adds Chinese documentation.
`README-ru.md`	Adds Russian documentation.
`README-es.md`	Adds Spanish documentation.
`dload.xsd`	Defines cache XML attributes.
`bin/dload`	Registers `cache:clear`.

Review details

Files reviewed: 68/69 changed files
Comments generated: 5
Review effort level: Balanced

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

…ot the configured URI Both factories accept a full URL and reduce it to the path, so the release records were stored under the reduced path while `attach()` and `forget()` used the raw URI: `cache:clear <software>` missed the record and a deleted release was never forgotten. The identifier is now built from the repository name inside the downloader; `RepositoryId::fromConfig()` is gone so the trap cannot return. Assisted-By: Claude Fable 5.1 <noreply@anthropic.com>

…ease The API adapters skip a release they cannot decode, so the fetched head lacked it exactly like a deleted one and the check dropped it from the record until the next TTL expiry. The page now carries the number of skipped releases; a check that meets one leaves the stored list as it is and is repeated on the next run. Without a stored list the readable releases are still served, as before. Assisted-By: Claude Fable 5.1 <noreply@anthropic.com>

Asset URLs are read back from the version registry on disk, so a tampered file could point a download at any host, and the client used to attach the bearer token to every request. The token now follows requests to `github.com` and `githubusercontent.com` and their subdomains only. Assisted-By: Claude Fable 5.1 <noreply@anthropic.com>

…d count matches the API paging fix(github): accept the `null` publication date of a draft The tail is loaded from the page the stored count points at, while GitHub counts drafts in its pages: with drafts on top the count fell short, and every extension re-requested a page it already had. A draft is now stored as a hidden record with nothing but its tag, so it holds its position and is never served. Assisted-By: Claude Fable 5.1 <noreply@anthropic.com>

…d or removed Every file was renamed into place atomically, but a save is several files: a run could remove as orphans the segments another run had just written, or publish an index over segment content the other had produced under the same key. The lock files live in a `locks/` directory beside the repositories because Windows cannot remove a directory that holds an open file. The wait is bounded, so a lock left by a stuck run is reported instead of hanging the download. Assisted-By: Claude Fable 5.1 <noreply@anthropic.com>

…e suite left untested Line coverage of the files changed on this branch goes from 89% to 97% overall; the downloader from 69% to 100%. The remaining gaps are the `posix_geteuid()` branch, unreachable without the extension, and the closing lines of the page-loader closures, which never fall through. Assisted-By: Claude Fable 5.1 <noreply@anthropic.com>

… on POSIX too The test pinned the target directory with an open file, which only Windows refuses to delete; on Linux the rename went through and the test failed. A directory without write permission keeps its file on POSIX; the test is skipped for root, who ignores permissions. Assisted-By: Claude Fable 5.1 <noreply@anthropic.com>

The flag is the one a user types by hand most often, and `-r` is free in `get`. The option tables of the READMEs did not list `--refresh` at all; the row is added alongside `--force`. Assisted-By: Claude Fable 5.1 <noreply@anthropic.com>

github-actions Bot added enhancement New feature or request tests labels Sep 12, 2026

roxblnfk mentioned this pull request Sep 12, 2026

fix: request every releases page once and cache release listings #118

Closed

roxblnfk and others added 2 commits September 13, 2026 00:16

roxblnfk force-pushed the feat/version-registry branch from e85e96b to ea36d7e Compare September 12, 2026 20:17

roxblnfk added 4 commits September 14, 2026 19:34

roxblnfk force-pushed the feat/version-registry branch from d754a00 to 86d6b00 Compare September 14, 2026 15:36

roxblnfk force-pushed the feat/version-registry branch from 05f39d1 to 794f0ad Compare September 14, 2026 16:27

roxblnfk added 2 commits September 14, 2026 20:28

roxblnfk force-pushed the feat/version-registry branch from 794f0ad to 90ffe6e Compare September 14, 2026 16:28

roxblnfk requested a balanced review from Copilot September 14, 2026 17:58

Copilot started reviewing on behalf of roxblnfk September 14, 2026 17:58 View session

Copilot AI reviewed Sep 14, 2026

View reviewed changes

roxblnfk added 8 commits September 15, 2026 10:33

feat(get): add -r as the short form of --refresh

b74ae4d

The flag is the one a user types by hand most often, and `-r` is free in `get`. The option tables of the READMEs did not list `--refresh` at all; the row is added alongside `--force`. Assisted-By: Claude Fable 5.1 <noreply@anthropic.com>

roxblnfk merged commit c1d5df9 into 1.x Sep 15, 2026
22 of 23 checks passed

roxblnfk deleted the feat/version-registry branch September 15, 2026 08:51

github-actions Bot mentioned this pull request Sep 15, 2026

chore(1.x): release 1.16.0 #120

Merged

Navigation Menu

Uh oh!

feat: version registry that keeps release lists between runs - #119

feat: version registry that keeps release lists between runs#119
roxblnfk merged 17 commits into
1.xfrom
feat/version-registry

roxblnfk commented Sep 12, 2026 •

edited

Loading

Uh oh!

codecov Bot commented Sep 12, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

roxblnfk commented Sep 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔍 What was changed

How it works

Why?

Checklist

Documentation

Uh oh!

codecov Bot commented Sep 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

roxblnfk commented Sep 12, 2026 •

edited

Loading

codecov Bot commented Sep 12, 2026 •

edited

Loading