Add a separate action for removing old wheels

[agriyakhetarpal]

Description

This PR closes #62; by converting the .github/workflows/remove-wheels.yml workflow to a composite action in remove-wheels/action.yml that users can trigger in their workflows separately to delete binaries uploaded to the https://anaconda.org/scientific-python-nightly-wheels/ index or other Anaconda.org indices.

This way, users should be able to add, for example:

- name: Remove old wheels
  uses: scientific-python/upload-nightly-action/remove-wheels@vx.y.z # or, better, via the commit hash
  with:
    anaconda_nightly_token: ${{ secrets.ANACONDA_NIGHTLY_UPLOAD_TOKEN }} # same as that for uploads (convenience?)
    n_latest_uploads_to_keep: '5'
    anaconda_nightly_upload_organization: 'my-custom-org'
    package_name: 'my-package'

or similar variations, to delete all but the last five wheel uploads, which means that this repository does not need to do so and can delegate this responsibility to the workflow files for the packages listed under the https://anaconda.org/scientific-python-nightly-wheels/repo page.

Changes made

• A new action in ./remove-wheels/action.yml that accepts an organisation name, a package name, and the number of latest uploads to keep
• The wheel removal logic extracted into a nicer Python script at remove-wheels/remove_wheels.py (based on discussions I had with @Cadair)
• Documentation updates in the README

Checklist

• Resolve the to-do items (some of them have been marked as inline comments in the code for now)
• Add relevant documentation, and adjust headings in the README
• If .github/workflows/remove-wheels.yml stays, figure out how to not unintentionally delete wheels without the existence of packages-ignore-from-cleanup.txt

[agriyakhetarpal]

Okay, I've just read some discussion(s) on keeping the removal of old wheels "centralised" in #4 (comment) and in the comments after it. Those made sense to me, so, this PR could be considered in conjunction with that policy – to let SPNW clean up old wheels itself, and let these changes be a "refactor" of the wheel removal logic in a sense, i.e., from being embedded in a workflow to a separate action that gets used in it. This would also ease the removal process around those who want to take up the responsibility of removing their wheels that have been uploaded outside SPNW-managed Anaconda.org indices. It is to be noted that that discussion predates the ones linked in #62, so it is likely the case that requests for easier removals have arisen in more contexts by now.

[bsipocz]

Yes, having the cleanup parts moved to a new action and then using it sounds to be the right way to go, that way we expose the functionality for easy reuse while we also keep using it.

[matthewfeickert]

I am in transit and won't have time to review or give feedback on this for a few days. My original intent in Issue #62 was to make a new repository and have that action be totally independent of this repository. Though, we can certainly get this in here and then just move it elsewhere later if people agree that's a good idea.

[agriyakhetarpal]

That sounds great! Thanks for triggering the workflow for me – I shall revisit this either later in the day or early next week. That said, I don't think one can run the workflow from forks, though... I have an account on Anaconda.org already since I'm helping manage the PyWavelets wheels (#75), should I add a dummy package for testing (in my account, that is)?

[agriyakhetarpal]

to make a new repository and have that action be totally independent of this repository. Though, we can certainly get this in here and then just move it elsewhere later if people agree that's a good idea.

Personally, I would say that related actions like these ones could be grouped together just fine, since one repository is easier to manage, rather than two – easier permissions to deal with, unified documentation in the README, unified dependency management (using pixi), etc., so this could fit in the same repository very well. For example, https://github.com/github/codeql-action contains six actions, with each managed in its own folder.

Of course, this is just my opinion, so I'm happy to implement whatever works best within the community.

[agriyakhetarpal]

Thanks for the documentation suggestions, addressed them in 2f62d85 – that commit should resolve the conversations from the review above.

[agriyakhetarpal]

@Cadair and I have modified the action from this repository's workflows to run in a way that deletes packages one at a time (through a dynamically created matrix based on the available wheels in the specified index), rather than deleting multiple packages at once.

[bsipocz]

ping @matthewfeickert, @tupui and possibly @nabobalis on behalf of the tools interest group.

I've also put this on my list to get back to, but it's pretty far down so don't wait for me if the remaining questions/issues can be sorted.

[matthewfeickert]

which means that this repository does not need to do so and can delegate this responsibility to the workflow files for the twenty-six packages listed under the https://anaconda.org/scientific-python-nightly-wheels/repo page.

I'm very time constrained this week given multiple deadlines, so I'm not going to have time to do a useful review until next week.

The one thing I will say now though is that if

which means that this repository does not need to do so and can delegate this responsibility to the workflow files for the twenty-six packages listed under the https://anaconda.org/scientific-python-nightly-wheels/repo page.

is still the plan/goal of this PR I think that should be avoided. It is a good idea and fine to give projects the ability to clean up their own wheels easily, but that ability should not be removed from the org by fully delegating all abilities.

Though if

The older wheel removal script moved to its dedicated file in remove_wheels.sh

is still correct, then that ability still remains which is good. 👍

The PR body could probably stand for a cleanup to make review easier, as well as a review done by the authors to lead reviewers through what they did.

[bsipocz]

is still the plan/goal of this PR I think that should be avoided. It is a good idea and fine to give projects the ability to clean up their own wheels easily, but that ability should not be removed from the org by fully delegating all abilities.

Fully agree, we should still be able to curate the packages, too, I mean the ones we host (though I believe Cadair's use case is to use all these machinery to upload to elsewhere -- and their the curation would fall on us)

[Cadair]

When I was hacking on this with @agriyakhetarpal the plan was that the cleanup script in this repo would be a github workflow which ran the action for all the packages that need to be cleaned up. So either users explicitly cleaning up or this repo doing it would be using the same code.

[stefanv]

Is this package list generation still necessary?

[agriyakhetarpal]

get-pkgs generates the input matrix, so yes

[agriyakhetarpal]

(CI fails because this PR is from my fork)

[bsipocz]

(CI fails because this PR is from my fork)

Could you rebase to resolve the conflicts?

[agriyakhetarpal]

Thanks! Done. I also renamed n_latest_uploads to n_latest_uploads_to_keep as that sounds clearer to me.

[agriyakhetarpal]

Turns out jq isn't on Windows yet (conda-forge/jq-feedstock#3, conda-forge/jq-feedstock#24), which is surprising – this kind of explains why you were wanting me to rewrite the wheel removal script in Python instead, @Cadair 😅

[agriyakhetarpal]

I added this in the last commit in 7f6204c as a means to test the action on PRs (otherwise it runs only on a daily schedule and on manual triggers). Please let me know in case you think this is not ideal – I'll revert. Thanks!