WIP Migrate to pixi

[crusaderky]

Follow-up to dask/dask#12389

• drastically simplify deployment for developers, particularly when juggling multiple environments
• drastically improve reproducibility between CI and desktop
• prevent CI from failing overnight - with the exception of the upstream workflows, which retain the old behaviour by design

Functional changes

• The Tests workflow is still triggered on schedule in order to detect test flakiness, but is no longer affected by upstream changes.

• ⚠️ The Tests workflow no longer automatically tracks the dask/dask git tip. ⚠️ If a PR in dask/dask breaks something in dask/distributed CI, you will still notice it in the Upstream workflow.

• Added Upstream workflow, which features two runs:

  • nightly (nightly wheels and git tip of key dependencies)
  • python 3.14 (all latest releases of all dependencies).

  Upstream / py314 differs from Tests / py314 as Tests uses the lockfile, meaning dependency versions don't change on their own, whereas Upstream throws out the lockfile, getting all latest and greatest packages as soon as they are released. In other words, Upstream / py314 replicates the behaviour of the old Tests workflow.

• Repaired the Conda build workflow, which has been broken for some time. Now Conda build runs on all PRs (takes under 5 minutes).

• Tweaked the triggers of some CI jobs for sanity

Follow-ups

• Migrate test report to pixi (depends on dask/distributed)
• Add pixi.toml to dask/partd and dask/zict (short of a full migration to pixi) and reintroduce their git tips to the nightly environment. This is low priority as these packages barely see any movement nowadays.
• Run a new pixi task in CI to build and validate pypi wheels
• Use pixi to manage linters
• Use pixi to automate docs building locally
• Use pixi to run with cupy (no CI; only local)

Known issues

• pixi currently has poor support for SCM versioning. There are a few workarounds in this PR.
• github does not offer a way to hide line counts in lockfiles. This annoyingly causes bugous line counts (e.g. 51k lines in this PR!) whenever pixi.lock changes. For clarity, this is an issue shared by all package managers for all languages which use a lock file.

[github-actions]

Unit Test Results

See test report for an extended history of previous test failures. This is useful for diagnosing flaky tests.

0 files   -     31  0 suites   - 31   0s ⏱️ - 10h 4m 50s
0 tests  -  4 076  0 ✅  -  3 967  0 💤  -   106  0 ❌  - 3 
0 runs   - 57 728  0 ✅  - 55 327  0 💤  - 2 398  0 ❌  - 3 

Results for commit 362e99e. ± Comparison against base commit d8fabe2.