Skip to content

ci(windows): add Windows MSVC CI jobs - #2738

Open
pimlock wants to merge 25 commits into
mainfrom
fix/windows-msvc-cache/pimlock
Open

ci(windows): add Windows MSVC CI jobs#2738
pimlock wants to merge 25 commits into
mainfrom
fix/windows-msvc-cache/pimlock

Conversation

@pimlock

@pimlock pimlock commented Aug 13, 2026

Copy link
Copy Markdown
Collaborator

Summary

Run the supported Windows MSVC lint and test workload for pull requests and merge queues. Keep its Cargo and compiler caches warm on main, then build the x64 release binaries without publishing them.

Related Issue

No issue required: localized CI infrastructure work following #2496.

Changes

  • run rust:lint and test:rust on windows-2025 for pull-request mirrors and merge queues
  • run the same workload on main and manual dispatches to seed the shared Cargo target and sccache namespaces
  • build openshell-gateway.exe and openshell.exe after the cache-seed job succeeds on main or a manual dispatch
  • use the official prebuilt Z3 release instead of compiling Z3 from source on Windows
  • cache Cargo target artifacts and use the GitHub Actions sccache backend
  • keep ARM64 validation in the existing local tasks until a native hosted workflow is enabled
  • update the Windows build architecture notes and maintenance skill
  • use enterprise-approved pinned revisions of jdx/mise-action and dtolnay/rust-toolchain

Testing

  • mise run pre-commit
  • cargo check -p openshell-prover --features prebuilt-z3
  • cargo check -p openshell-server --features prebuilt-z3
  • workflow YAML parse and git diff --check
  • GitHub Actionlint
  • no Zizmor findings for .github/workflows/windows-msvc.yml
  • enterprise policy accepted both pinned actions and started the x64 Windows job
  • post-rebase lint and test completion on the hosted Windows x64 runner
  • native Windows ARM64 validation, local-only by design
  • E2E tests added/updated (not applicable; CI-only change)

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)
  • Architecture docs updated

@copy-pr-bot

copy-pr-bot Bot commented Aug 13, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@pimlock
pimlock marked this pull request as ready for review August 13, 2026 23:31
@pimlock

pimlock commented Aug 13, 2026

Copy link
Copy Markdown
Collaborator Author

Windows MSVC cache experiments

I ran the hosted windows-2025 workflow repeatedly while changing one major cache/build variable at a time. Times below are end-to-end workflow/job times and can vary slightly with runner setup and GitHub cache transfer latency.

Configuration Cache state Runtime Notes
Original workflow / bundled Z3 effectively cold ~75m48s–78m22s Baseline; Z3 compiled from source and the Rust target tree was not providing a useful warm reuse path. run, repeat
Cargo target cache / bundled Z3 cold ~75m16s Populated the target archive. run
Cargo target cache / bundled Z3 warm ~18m39s Large improvement; target artifacts were the most effective cache layer. run
sccache only / bundled Z3 cold ~80m47s Target cache disabled to isolate sccache. Remote writes made the cold run slower. run
sccache only / bundled Z3 warm ~67m55s 1,157 hits / 122 misses (90.46% hit rate), but linking, non-cacheable crate types, Z3 work, and remote reads left total time high. run
Cargo target cache + prebuilt Z3 cold ~37m28s z3-sys downloaded the pinned official prebuilt Z3 archive instead of compiling Z3 via CMake/MSBuild. run
Cargo target cache + prebuilt Z3 warm ~16m24s Check 1m14s, release build 6m36s, tests 4m46s, focused test 2m12s; target restore ~58s. run
Target cache + prebuilt Z3 + sccache target warm, sccache cold ~17m09s 62 misses, 62 successful writes, 0 errors; cache writes took 32.6s. run
Target cache + prebuilt Z3 + sccache both warm ~15m45s 62/62 sccache hits, 0 misses/errors; check 1m07s, release build 5m19s, tests 5m06s, focused test 2m25s. run
Warm hybrid cache, redundant focused test removed both warm ~12m33s Check 1m01s, release build 5m07s, full tests 4m58s. The full suite already includes the Windows unsupported-driver contract test. run

Conclusions

  • Cargo target caching was the largest win: roughly 75 minutes cold to roughly 19 minutes warm with bundled Z3.
  • Prebuilt Z3 materially improved cold builds: the cold target-cache run dropped from roughly 75 minutes to roughly 37 minutes.
  • sccache alone was not competitive, despite a high hit rate. It does not cache final linking or several Rust crate types, and GitHub remote reads still cost time.
  • sccache is useful as a secondary layer on top of the target cache: it reduced the warm release build from roughly 6m36s to 5m19s and produced a clean 100% hit rate for the 62 cacheable compiler requests in the measured run.
  • The best measured stack is Cargo target cache + prebuilt Z3 + sccache, with the redundant focused test removed.
  • The PR/main split is intended to keep PR validation focused on Windows Clippy + Rust tests, run that same workload on main to warm the exact PR cache namespaces, and build (but not upload/publish) release binaries after the seed succeeds.

Current validation note

The earlier single-job hybrid workflow is proven by the successful runs above. The new multi-job split currently receives an opaque GitHub startup_failure before any job is created, on both manual dispatch and the PR mirror. actionlint reports the workflow as valid, and a temporary ref at the last known-good commit starts normally, so I am continuing to isolate the workflow-graph difference before treating the split as ready.

@pimlock

pimlock commented Aug 13, 2026

Copy link
Copy Markdown
Collaborator Author

Validation update: the opaque pre-job startup_failure was isolated to jdx/mise-action v4.2.4 on windows-2025. Restoring the previously measured v4.2.0 pin fixes workflow startup. The PR mirror is now running PR lint and test (x64), and a manual dispatch is independently running Seed cache (x64) followed by Build binaries (x64). The newer Swatinem/rust-cache v2.9.2 pin remains in place.

@pimlock pimlock changed the title perf(ci): accelerate Windows MSVC validation ci(windows): add Windows MSVC CI jobs Aug 14, 2026
@pimlock
pimlock marked this pull request as draft August 14, 2026 21:10
@pimlock
pimlock force-pushed the fix/windows-msvc-cache/pimlock branch from 91fa94f to 2fdb6ec Compare August 18, 2026 23:31
@pimlock

pimlock commented Aug 19, 2026

Copy link
Copy Markdown
Collaborator Author

Pausing this PR until the compute-driver refactor stack lands:

#2786 overlaps 8 of the 18 files changed here and establishes a cleaner Windows boundary by keeping first-party driver dependencies out of openshell-core and openshell-server. The current strict Windows Clippy fixes add 65 cfg attributes, many of which are likely to become unnecessary after that refactor. Continuing now would create avoidable code and rebase churn.

After the stack merges, the plan is to:

  1. Rebase this PR again.
  2. Keep the Windows PR/main/cache-seed job split, target cache, sccache, and prebuilt Z3 work.
  3. Update lint/build coverage for the new openshell-gateway composition crate.
  4. Remove obsolete platform-scoping changes and handle any remaining Kubernetes or UDS boundaries at module level where possible.
  5. Rerun the hosted Windows timing and cache validation.

The branch is currently rebased onto main; local mise run pre-commit passes.

@pimlock

pimlock commented Aug 20, 2026

Copy link
Copy Markdown
Collaborator Author

Update to the earlier waiting note: #2786 has merged, but the stack was split before merge. The merged #2786 is only the compute-driver registry pre-factor and intentionally keeps first-party driver composition in openshell-server.

The remaining relevant stack is now:

#2823 is the architectural change we were originally waiting for. It overlaps 8 of this PR’s 18 files, including the core/server configuration and Kubernetes-auth files where most of the Windows cfg changes currently live. This PR is also now conflicting with main, so rebasing and simplifying it before #2823 would create another round of avoidable churn.

Plan: continue holding this PR until #2822 and #2823 land, then rebase, retain the Windows workflow/cache/prebuilt-Z3 work, update lint coverage for openshell-gateway, and recreate only the platform fixes still required by strict hosted Windows Clippy.

@pimlock
pimlock force-pushed the fix/windows-msvc-cache/pimlock branch from e5f0d01 to 5894ebb Compare August 26, 2026 23:02
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
@pimlock
pimlock force-pushed the fix/windows-msvc-cache/pimlock branch from 5894ebb to c80a5ad Compare September 1, 2026 23:03
Comment thread .github/workflows/windows-msvc.yml Fixed
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
@pimlock

pimlock commented Sep 1, 2026

Copy link
Copy Markdown
Collaborator Author

/ok to test 5dd3496

@pimlock
pimlock marked this pull request as ready for review September 1, 2026 23:10
@pimlock
pimlock requested a review from sjenning as a code owner September 1, 2026 23:10
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Sep 2, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@pimlock

pimlock commented Sep 2, 2026

Copy link
Copy Markdown
Collaborator Author

/ok to test 5ece42d

@pimlock

pimlock commented Sep 2, 2026

Copy link
Copy Markdown
Collaborator Author

Final Windows validation

Head 5ece42d8 is green locally and in hosted CI.

  • local mise run --skip-tools rust:lint: passed on Windows x64 / Rust 1.95
  • local mise run --skip-tools test:rust: passed (2,279.4s cold test build)
  • hosted Windows lint + test: passed
  • Cargo target cache: restored successfully, 610 MB (639366429 bytes)
  • identical-head cache rerun: passed in 13m35s
  • sccache on identical-head rerun: 734 hits / 31 misses (95.95%), 0 read errors
  • Branch Checks, Helm, DCO, Actionlint, Zizmor, and required aggregate gates: passed

The first post-fix hosted run populated the current compiler inputs; the identical-head rerun verified that both the Cargo target archive and GHA sccache namespace are reusable.

@NVIDIA NVIDIA deleted a comment from copy-pr-bot Bot Sep 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants