test(e2e): drive the Lemonade recovery scenario with a planted runtime

[rominf]

Depends on #342 — this rewrites the contract test #342 adds, so it stays in draft until that merges. Closes #349.

Problem

e2e-test-hooks is a cargo feature that compiles two scripted-failure seams into rocm, and it exists to serve exactly one scenario: @id:serve-lemonade-preparation-recovery (serve-18).

• engines/lemonade/src/lib.rs — makes the llama.cpp backend install fail on demand
• apps/rocm/src/main.rs — waives rocm serve's no-GPU pre-flight, so the scenario can reach the install phase on a GPU-less host

The cost is spread across the repo: 2 feature declarations, 3 cfg sites, 10 --features lines across nightly.yml and e2e-selfhosted.yml, xtask's own build, and 3 contract tests.

Two consequences follow from the feature existing at all.

Binaries under test are not the binaries shipped. Every lane that pre-builds with the feature installs a binary that differs from a release build. #342 fixed this for the Windows lifecycle lane; it was still true on Linux, where the release build happens inside cargo xtask e2e — with the feature — so the Linux lifecycle lane packaged and installed a hook-carrying binary through the real installer.

The contract tests encoded the asymmetry instead of removing it. One test asserted the Windows lane must be hook-free, two asserted self-hosted and nightly must be hook-ful, and ci.yml's Linux lane was asserted by neither — the same blind spot that hid the duplicate-build drift #342 fixed, one lane over.

Fix

Delete the feature and drive the failure from a planted runtime instead.

tests/e2e-cucumber/src/bin/fake-lemonade.rs is a fixture binary copied into an isolated runtime directory under both lemonade and lemond (it takes its role from argv[0]), alongside a hand-written install manifest. It reproduces the three calls the engine makes: status exits 0, backends prints a table reporting llamacpp:rocm as installable, and backends install exits non-zero.

The CLI then fails where a genuinely broken install fails — at a non-zero child process — and rocm stays byte-identical to what ships.

A bin target of e2e-cucumber rather than an xtask-built artifact, so the suite reaches it through CARGO_BIN_EXE_fake-lemonade with no path to keep in step. It has to be a real executable, not a script: the engine spawns it by exact path and Windows CreateProcess needs a PE image.

Why this doesn't weaken verification

resolve_runtime accepts the planted manifest because it checks only that the manifest parses and that lemond is a file. No checksum, version, or signature is involved on this path — the SHA-pinned archive download lives in prepare_embeddable, which the serve path does not go through. Nothing here bypasses a check the product performs.

Approaches ruled out are recorded in #349 (mocking the request — the boundary is a subprocess, not HTTP; prewarming — ephemeral on hosted lanes, already installed on prewarmed ones; dropping the network; mocking verify_sha256 — a fail-open seam inside supply-chain verification).

The tradeoff, explicitly

Seam #2 was what let serve-18 run without a GPU. Without it, rocm serve reaches its GPU-required pre-flight first, so the scenario is retagged @requires-gpu and moves off the blocking every-PR lane onto a gated one.

That is a real loss of coverage cadence and it is the reviewer's call, not a cleanup side effect. The counter-argument is that what it covered on the every-PR lane was a code path that only existed in test builds.

To limit what that move gives up, a unit test in engines/lemonade pins the coupling a GPU-less lane would otherwise silently lose: the fixture's backends table must still parse into a llamacpp:rocm install attempt. If a parser change stopped accepting it, the scenario would quietly stop reaching the failure it asserts and only a GPU lane would notice. That test runs everywhere.

Contract tests

assert_prebuilt_e2e_lanes_enable_test_hooks is repurposed, not deleted: "match the feature xtask uses" becomes "match the invocation xtask uses", which is the invariant that actually prevents the duplicate release build. It asserts whole-line, so a trailing --features … cannot satisfy it — verified by re-adding the flag and watching it fail.

#342's !contains("e2e-test-hooks") assertion is dropped; its reuse assertions stay.

Test plan

Verified locally:

• The fixture's four behaviours, driven directly: backends table, status → 0, backends install → non-zero, lemond stays alive until killed.
• The new lemonade unit test, and the reworked contract tests (24/24). Re-adding --features rocm/e2e-test-hooks to a lane makes nightly_prebuilt_e2e_lanes_match_xtask_build fail.
• cargo check --workspace --all-targets, plus cargo check -p e2e-cucumber --test e2e explicitly (the e2e target is test = false, so --all-targets skips it).
• cargo clippy --workspace --all-targets -- -D warnings and cargo fmt --check clean.
• cargo test --workspace --all-targets: two failures in rocm-dash-tui (chatgpt_oauth_client_*, "cannot create token cache dir: Permission denied"). Confirmed pre-existing — they fail identically on origin/main in this environment, and this diff does not touch that crate.

Not verified locally, stated per AGENTS.md §8: serve-18 itself. This host has no usable AMD GPU, so rocm serve stops at the pre-flight — which is exactly why the scenario is now @requires-gpu. I confirmed the CLI reaches and reports that pre-flight against a planted runtime, but the retry-count and recovery-guidance assertions past it need a GPU host. The self-hosted GPU lane is what will exercise them, and this PR should not merge until that lane has run it green.