Skip to content

ci(e2e): add an MI350P self-hosted E2E lane #341

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
tomastola merged 5 commits into from
Sep 10, 2026
Merged

ci(e2e): add an MI350P self-hosted E2E lane #341

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
888baab
fix(e2e): report gfx950 as its own platform, not as MI300X
tomastola Sep 3, 2026
417a528
ci(e2e): add an MI350P self-hosted E2E lane
tomastola Sep 3, 2026
5cf4f0f
docs(e2e): derive the README lane lists from the workflows
tomastola Sep 8, 2026
c446667
docs(e2e-report): stop enumerating platforms in the report legend
tomastola Sep 8, 2026
fd0ceaf
test(e2e): cover every gfx form that reaches the slug mapping
tomastola Sep 8, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
129 changes: 128 additions & 1 deletion .github/workflows/e2e-selfhosted.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ on:
type: choice
default: all
# Each value also selectable on its own; `all` covers every lane below.
options: [all, app-dev-gpu, strix-ubuntu, strix-windows, strix-wsl, rad3]
options: [all, app-dev-gpu, strix-ubuntu, strix-windows, strix-wsl, rad3, mi350p]
name_filter:
description: "Scenario-name regex (cucumber --name); empty = full suite"
type: string
Expand Down Expand Up @@ -958,6 +958,132 @@ jobs:
name: e2e-gpu-rad3-report
path: tests/e2e-cucumber/results/

e2e-gpu-mi350p:
name: E2E tests (MI350P)
# 90min: matches the other Instinct lane — same collapsed suite.
timeout-minutes: 90
# `mi350p` names the hardware rather than reusing the generic `amd-gpu`:
# GitHub matches a job to any runner whose labels are a superset, so a shared
# label would let this card pick up MI300X-targeted work and vice versa.
runs-on: [self-hosted, linux, mi350p]
needs: [changes]
if: >-
always()
&& needs.changes.result == 'success'
&& (
(github.event_name != 'workflow_dispatch'
&& needs.changes.outputs.serve == 'true')
|| (github.event_name == 'workflow_dispatch'
&& (inputs.platform == 'all' || inputs.platform == 'mi350p'))
)
continue-on-error: true
env:
E2E_SERVE_TIMEOUT_SECS: "300"
E2E_INCLUDE_NIGHTLY: "${{ inputs.include_nightly && '1' || '' }}"
# Full serve matrix at the merge-queue gate, like every other lane (see e2e-gpu).
E2E_MERGE_QUEUE: "${{ github.event_name == 'merge_group' && '1' || '' }}"
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1

# The pod is usually fresh (the runner scales to zero between jobs), but the
# work volume is not, and two jobs can share one pod's life. Same reclaim as
# e2e-gpu, scoped to e2e leftovers only.
- name: Reclaim GPU from stray E2E processes
run: |
pkill -f '/tmp/rocm-e2e.*llama-server' 2>/dev/null || true
pkill -f '/tmp/rocm-e2e.*vllm serve' 2>/dev/null || true
pkill -f 'e2e-shared.*llama-server' 2>/dev/null || true
pkill -f '__engine-serve-http.*rocm-e2e' 2>/dev/null || true
pkill -f 'e2e-target/release/rocm daemon' 2>/dev/null || true
rm -rf /tmp/rocm-e2e-* 2>/dev/null || true
echo "reclaimed"

# Bounded wait, as in e2e-gpu. The pod is admitted for exactly one card, so
# rocm-smi sees that card alone. The floor matches the other Instinct lane
# rather than scaling with the 144GB capacity: it only has to be high enough
# to catch a leftover serve still holding the card, and low enough that a
# clean pod passes on the first poll.
- name: GPU preflight (bounded wait for an available GPU)
run: |
MIN_FREE_GIB="${GPU_PREFLIGHT_MIN_FREE_GIB:-16}"
CEILING_SECS="${GPU_PREFLIGHT_CEILING_SECS:-90}"
min_free=$(( MIN_FREE_GIB * 1024 * 1024 * 1024 ))
deadline=$(( SECONDS + CEILING_SECS ))
reason="rocm-smi never returned within its timeout (driver wedged or GPU absent)"
while [ "$SECONDS" -lt "$deadline" ]; do
out=$(timeout 15 rocm-smi --showmeminfo vram 2>/dev/null) || { sleep 5; continue; }
total=$(printf '%s\n' "$out" | grep -i 'VRAM Total Memory' | sed 's/.*: *//' | grep -oE '[0-9]+' | tail -1)
used=$(printf '%s\n' "$out" | grep -i 'VRAM Total Used Memory' | sed 's/.*: *//' | grep -oE '[0-9]+' | tail -1)
if [ -z "$total" ] || [ -z "$used" ]; then
reason="rocm-smi returned no VRAM figures (no AMD GPU detected)"
sleep 5; continue
fi
free=$(( total - used ))
if [ "$free" -ge "$min_free" ]; then
echo "GPU ready: $(( free / 1024 / 1024 / 1024 )) GiB free (>= ${MIN_FREE_GIB} GiB)."
exit 0
fi
reason="VRAM never dropped below the floor: only $(( free / 1024 / 1024 / 1024 )) GiB free (< ${MIN_FREE_GIB} GiB) — a serve is likely still holding the GPU"
echo "waiting: $(( free / 1024 / 1024 / 1024 )) GiB free (< ${MIN_FREE_GIB} GiB)…"
sleep 5
done
echo "::error::GPU preflight failed after ${CEILING_SECS}s: ${reason}"
exit 1

# cache: false for the same reason as e2e-gpu — the build cache is kept in
# CARGO_TARGET_DIR on the work volume, not in GitHub's cache service.
- uses: actions-rust-lang/setup-rust-toolchain@166cdcfd11aee3cb47222f9ddb555ce30ddb9659 # v1.17.0
with:
cache: false

- name: Run E2E tests on MI350P
run: |
# $RUNNER_WORKSPACE is on a volume that survives the runner being scaled
# away, so the cargo target dir and the shared caches are still warm on
# the next job even though the container is not.
export CARGO_TARGET_DIR="$RUNNER_WORKSPACE/e2e-target"
export E2E_SHARED_CACHE_DIR="$RUNNER_WORKSPACE/e2e-shared"
# A SEPARATE volume is mounted at exactly this path by the runner's
# deployment. Keeping the ~23GB uv cache off the work volume is
# deliberate; if you change this path, change that deployment too.
export E2E_SHARED_UV_CACHE_DIR="/var/tmp/rocm-e2e-uv-cache"
# Pre-warm one shared runtime in place before the suite. Not an
# optimization — `install sdk` bakes absolute paths into the runtime
# manifest, so installing anywhere temporary breaks every later serve.
# See e2e-gpu for the full rationale.
prewarm="$RUNNER_WORKSPACE/e2e-prewarm-multi-arch-v2"
export E2E_SHARED_RUNTIMES_DIR="$prewarm/data/runtimes"

# See the e2e-gpu lane for why the e2e-test-hooks feature must match
# what `cargo xtask e2e` would build.
cargo build --release -p rocm -p rocmd --features rocm/e2e-test-hooks
export ROCM_CLI_BINARY="$CARGO_TARGET_DIR/release/rocm"
export ROCM_CLI_ROCMD_BINARY="$CARGO_TARGET_DIR/release/rocmd"

# Pre-warm once, serially, in place (no mv/symlink), and refresh it when
# the channel index has moved on — the tree is a cache, not a one-shot.
# See the e2e-gpu lane and `xtask e2e-prewarm`.
HF_HOME="$E2E_SHARED_CACHE_DIR/huggingface" \
UV_CACHE_DIR="$E2E_SHARED_UV_CACHE_DIR" \
cargo xtask e2e-prewarm --channel release --prewarm-dir "$prewarm"

# Optional scenario-name filter for a scoped manual dispatch — a cheap
# way to exercise this lane against a single scenario without the full suite.
NAME_FILTER="${{ github.event.inputs.name_filter }}"
if [ -n "$NAME_FILTER" ]; then
echo "name filter active: $NAME_FILTER"
cargo xtask e2e -- --name "$NAME_FILTER"
else
cargo xtask e2e
fi

- name: Upload E2E report
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: e2e-gpu-mi350p-report
path: tests/e2e-cucumber/results/

# Consolidate this workflow's self-hosted platform reports into one GPU-side
# cross-platform grid (Summary + merged HTML). Distinct name from ci.yml's
# required `E2E consolidated report` so it does NOT collide with that required
Expand All @@ -975,6 +1101,7 @@ jobs:
- e2e-gpu-strix-windows
- e2e-wsl
- e2e-gpu-rad3
- e2e-gpu-mi350p
# Gate on `serve`: every lane this report consolidates (the GPU jobs) is now
# serve-gated, so a serve-only change runs them and their report must still be
# produced. On dispatch `serve` is unset, so also run when the trigger was
Expand Down
90 changes: 90 additions & 0 deletions .github/workflows/nightly.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -549,6 +549,95 @@ jobs:
name: e2e-gpu-rad3-report
path: tests/e2e-cucumber/results/

e2e-gpu-nightly-mi350p:

@rominf rominf Sep 8, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adding this nightly lane makes a sentence in tests/e2e-cucumber/README.md wrong, and the test whose whole job is to prevent that can't see it.

README line 185 still reads:

The nightly workflow runs non-blocking jobs — MI300X, Radeon R9700, and Strix Halo on Ubuntu, Windows, and WSL2 — with E2E_INCLUDE_NIGHTLY=1

With e2e-gpu-nightly-mi350p in nightly.yml, that list is now a platform short.

The guard is the final assert! in xtask/src/workflow_contract.rs's docs-coverage test, and it asserts the README contains that exact literal, copied into the test source. So it stays green while its own failure message — "E2E README must identify every nightly job platform" — has stopped being true. The four sibling assertions immediately above it are derived from the YAML (declared_ids, backticked_list_between), which is exactly why docs/ci-hardware-testing.md got updated correctly and this didn't. It's the one hand-copied hole in an otherwise derived net.

Minimum fix: add MI350P to the README sentence and to the literal in the test. Better fix, if you're up for it: derive it from self_hosted_e2e_jobs(&nightly) the way the assertions just above do, so the next lane can't repeat this.

Separately, the CI job table a few lines earlier in the same README (~line 168) lists e2e, e2e-gpu, e2e-gpu-strix-ubuntu, e2e-gpu-strix-windows, e2e-wsl — it's already missing e2e-gpu-rad3, and would now be missing e2e-gpu-mi350p too. Nothing guards that table at all; the derived one only reads docs/ci-hardware-testing.md. Worth adding both rows while you're in the file.

@tomastola tomastola Sep 8, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 0d3d644, taking the "better fix" — the sentence is derived now, not
hand-copied.

The README nightly sentence names the nightly job ids in backticks, and the guard
compares them against self_hosted_e2e_jobs(&nightly), exactly like the four
assertions above it. The literal copy in the test source is gone, so the failure mode
you describe — the guard staying green while its own message stops being true — can't
recur for this sentence.

You were right that the CI job table was worse: no guard at all, and already missing
e2e-gpu-rad3 before this lane. Both rows added, and that table is derived too, against
e2e-selfhosted.yml — the mock row is matched by workflow so only the self-hosted rows
have to track the YAML, plus a check that none of them is documented as blocking.

I verified both new assertions actually bite rather than trusting a green run: removing
e2e-gpu-nightly-mi350p from the sentence fails with

the E2E README must name every nightly self-hosted lane, in workflow order
  left:  [..., "e2e-gpu-nightly-strix", ...]
  right: [..., "e2e-gpu-nightly-mi350p", "e2e-gpu-nightly-strix", ...]

and removing the e2e-gpu-mi350p table row fails the table assertion the same way.

name: E2E tests (MI350P, incl. nightly-only)
timeout-minutes: 90
runs-on: [self-hosted, linux, mi350p]
continue-on-error: true
env:
E2E_INCLUDE_NIGHTLY: "1"
E2E_SERVE_TIMEOUT_SECS: "300"
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1

- name: Reclaim GPU from stray E2E processes
run: |
pkill -f '/tmp/rocm-e2e.*llama-server' 2>/dev/null || true
pkill -f '/tmp/rocm-e2e.*vllm serve' 2>/dev/null || true
pkill -f 'e2e-shared.*llama-server' 2>/dev/null || true
pkill -f '__engine-serve-http.*rocm-e2e' 2>/dev/null || true
pkill -f 'e2e-target/release/rocm daemon' 2>/dev/null || true
rm -rf /tmp/rocm-e2e-* 2>/dev/null || true
echo "reclaimed"

# Bounded wait for a free GPU. Floor matches the per-PR MI350P lane and the
# other Instinct lane — high enough to catch a leftover serve, low enough
# that a clean pod passes on the first poll.
- name: GPU preflight (bounded wait for an available GPU)
run: |
MIN_FREE_GIB="${GPU_PREFLIGHT_MIN_FREE_GIB:-16}"
CEILING_SECS="${GPU_PREFLIGHT_CEILING_SECS:-90}"
min_free=$(( MIN_FREE_GIB * 1024 * 1024 * 1024 ))
deadline=$(( SECONDS + CEILING_SECS ))
reason="rocm-smi never returned within its timeout (driver wedged or GPU absent)"
while [ "$SECONDS" -lt "$deadline" ]; do
out=$(timeout 15 rocm-smi --showmeminfo vram 2>/dev/null) || { sleep 5; continue; }
total=$(printf '%s\n' "$out" | grep -i 'VRAM Total Memory' | sed 's/.*: *//' | grep -oE '[0-9]+' | tail -1)
used=$(printf '%s\n' "$out" | grep -i 'VRAM Total Used Memory' | sed 's/.*: *//' | grep -oE '[0-9]+' | tail -1)
if [ -z "$total" ] || [ -z "$used" ]; then
reason="rocm-smi returned no VRAM figures (no AMD GPU detected)"
sleep 5; continue
fi
free=$(( total - used ))
if [ "$free" -ge "$min_free" ]; then
echo "GPU ready: $(( free / 1024 / 1024 / 1024 )) GiB free (>= ${MIN_FREE_GIB} GiB)."
exit 0
fi
reason="VRAM never dropped below the floor: only $(( free / 1024 / 1024 / 1024 )) GiB free (< ${MIN_FREE_GIB} GiB) — a serve is likely still holding the GPU"
echo "waiting: $(( free / 1024 / 1024 / 1024 )) GiB free (< ${MIN_FREE_GIB} GiB)…"
sleep 5
done
echo "::error::GPU preflight failed after ${CEILING_SECS}s: ${reason}"
exit 1

- uses: actions-rust-lang/setup-rust-toolchain@166cdcfd11aee3cb47222f9ddb555ce30ddb9659 # v1.17.0
with:
cache: false

- name: Run full E2E on MI350P (incl. nightly-only)
run: |
# $RUNNER_WORKSPACE is on a volume that survives the runner being scaled
# away, so the cargo target dir and shared caches stay warm.
export CARGO_TARGET_DIR="$RUNNER_WORKSPACE/e2e-target"
export E2E_SHARED_CACHE_DIR="$RUNNER_WORKSPACE/e2e-shared"
# Separate volume mounted at exactly this path by the runner's
# deployment; if you change this path, change that deployment too.
export E2E_SHARED_UV_CACHE_DIR="/var/tmp/rocm-e2e-uv-cache"
prewarm="$RUNNER_WORKSPACE/e2e-prewarm-multi-arch-v2"
export E2E_SHARED_RUNTIMES_DIR="$prewarm/data/runtimes"

# See the e2e-gpu lane in e2e-selfhosted.yml for why the
# e2e-test-hooks feature must match what `cargo xtask e2e` would build.
cargo build --release -p rocm -p rocmd --features rocm/e2e-test-hooks
export ROCM_CLI_BINARY="$CARGO_TARGET_DIR/release/rocm"
export ROCM_CLI_ROCMD_BINARY="$CARGO_TARGET_DIR/release/rocmd"

# Pre-warm once, serially, in place (no mv/symlink), and refresh it when
# the channel index has published a newer runtime — the tree is a cache,
# not a one-shot. See `xtask e2e-prewarm`.
HF_HOME="$E2E_SHARED_CACHE_DIR/huggingface" \
UV_CACHE_DIR="$E2E_SHARED_UV_CACHE_DIR" \
cargo xtask e2e-prewarm --channel release --prewarm-dir "$prewarm"

cargo xtask e2e

- name: Upload E2E report
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: e2e-gpu-mi350p-report
path: tests/e2e-cucumber/results/

# Strix Halo counterpart to e2e-gpu-nightly. The same @nightly large-model
# scenario serves the platform-specific Lemonade GGUF on this host. Keep the
# runner's storage and runtime setup aligned with ci.yml's proven Strix Ubuntu job.
Expand Down Expand Up @@ -910,6 +999,7 @@ jobs:
needs:
- e2e-gpu-nightly
- e2e-gpu-nightly-rad3
- e2e-gpu-nightly-mi350p
- e2e-gpu-nightly-strix
- e2e-gpu-nightly-strix-windows
- e2e-wsl-nightly
Expand Down
10 changes: 7 additions & 3 deletions crates/e2e-report/src/lib.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -413,6 +413,7 @@ fn parse_descriptor(name: &str) -> Descriptor {
"" | "report" => ("Mock", "Linux"),
"gpu" => ("MI300X", "Linux"),
"gpu-rad3" => ("R9700", "Linux"),
"gpu-mi350p" => ("MI350P", "Linux"),

@rominf rominf Sep 8, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking, and it predates this PR — but the new lane makes it a third omission, and the text is rendered by the very report the new column now shows up in.

The legend enumerates the hardware platforms by hand in two places in this file: the Markdown step summary ("MI300X / Strix Halo run on real self-hosted GPU hardware with real engines") and the HTML legend (b { "MI300X / Strix Halo" }). R9700 was already absent; MI350P makes three lanes the legend implicitly disclaims while the grid displays their columns.

Since it's hand-maintained prose with no guard on it, the durable fix is to stop enumerating — something like "the non-Mock platforms run on real self-hosted GPU hardware; non-blocking while proven out" — rather than adding a name every time a lane lands. Happy either way on whether that belongs here or in a follow-up.

@tomastola tomastola Sep 8, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, and done here rather than as a follow-up — it's two lines and the alternative is
remembering to add a name every time a lane lands, which is the habit that produced the
omission in the first place. daaa985.

Both places now say "Every other platform — real self-hosted GPU hardware; non-blocking
while proven out" and enumerate nothing.

One wrinkle worth noting since it changed the wording I first wrote: the two surfaces
have opposite orientations. The HTML grid puts platforms in columns (Grid::build
dedups columns by slug), but the Markdown step summary puts them in rows
(| Platform | OS | Total | ... |). So "every other column" would have been wrong in the
Markdown summary. "Platform" is correct on both, which is a small argument for the
non-enumerating phrasing beyond just staleness.

"gpu-strix-ubuntu" => ("Strix Halo", "Ubuntu"),
"gpu-strix-windows" => ("Strix Halo", "Windows"),
// Same silicon again, third host boundary: an Ubuntu distro under WSL2 on
Expand Down Expand Up @@ -1298,8 +1299,10 @@ pub fn consolidated_summary_markdown(inputs: &[(String, PathBuf)]) -> String {
forwarding — with no GPU, no model download, and no engine process, so it \
runs on a GitHub-hosted runner. It **gates the PR**: it runs on every push, \
and if it fails the PR's required check goes red and the PR cannot merge. \
**MI300X / Strix Halo** run on real self-hosted GPU hardware with real \
engines. They are **non-blocking**: they still run and are reported here, but \
**Every other platform** is a real self-hosted GPU host running real \
engines — the rows below are the list, so naming them here would only go \
stale as lanes are added. They are **non-blocking**: they still run and \
are reported here, but \
a failure does NOT block the PR from merging (the hardware/runners are still \
being proven out, so their results are informational rather than a merge \
gate).\n\n\
Expand Down Expand Up @@ -1935,7 +1938,7 @@ fn legend() -> Markup {
"and gates the PR."
}
li {
b { "MI300X / Strix Halo" }
b { "Every other platform" }
" — real self-hosted GPU hardware; non-blocking while proven out."
}
li {
Expand Down Expand Up @@ -2219,6 +2222,7 @@ mod tests {
("e2e-report", "Mock", "Linux"),
("e2e-gpu-report", "MI300X", "Linux"),
("e2e-gpu-rad3-report", "R9700", "Linux"),
("e2e-gpu-mi350p-report", "MI350P", "Linux"),
("e2e-gpu-strix-ubuntu-report", "Strix Halo", "Ubuntu"),
("e2e-gpu-strix-windows-report", "Strix Halo", "Windows"),
// Must not fall through to `fallback_descriptor`, which would render
Expand Down
33 changes: 18 additions & 15 deletions docs/ci-hardware-testing.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,7 @@ separate tier flag or tag filter to maintain.
| `e2e-gpu-strix-windows` | `e2e-selfhosted.yml` | Strix Halo (gfx1151) on native Windows 11 | self-hosted `[self-hosted, windows, strix-halo, native]` |
| `e2e-wsl` | `e2e-selfhosted.yml` | Strix Halo (gfx1151) on Ubuntu under WSL2 | self-hosted `[self-hosted, linux, strix-halo, wsl]` |
| `e2e-gpu-rad3` | `e2e-selfhosted.yml` | Radeon AI PRO R9700 (gfx1201) on Linux | self-hosted `[self-hosted, linux, r9700]` |
| `e2e-gpu-mi350p` | `e2e-selfhosted.yml` | MI350P (AMD Instinct, gfx950) on Linux | self-hosted `[self-hosted, linux, mi350p]` |

The Strix Halo lanes pin the extra `native` label because two Linux runners
share the `strix-halo` label (a native host and a WSL host) and the jobs'
Expand All @@ -55,10 +56,10 @@ named for.
resolve to skip here, and known bugs resolve to xfail from
`expectations.toml`. It is a required check and must stay green.

The self-hosted jobs (`e2e-gpu`, `e2e-gpu-strix-ubuntu`,
`e2e-gpu-strix-windows`, `e2e-wsl`, and `e2e-gpu-rad3`) run on AMD GPU systems, so they exercise
host/GPU detection, engine `detect`/`capabilities`, and live serving scenarios
that the mock job cannot. GPU availability is advisory in the WSL lane, as
The self-hosted jobs (`e2e-gpu`, `e2e-gpu-strix-ubuntu`, `e2e-gpu-strix-windows`,
`e2e-wsl`, `e2e-gpu-rad3`, and `e2e-gpu-mi350p`) run on AMD GPU systems, so they
exercise host/GPU detection, engine `detect`/`capabilities`, and live serving
scenarios that the mock job cannot. GPU availability is advisory in the WSL lane, as
described below.

`e2e-wsl` runs on an Ubuntu distro hosted in WSL2 on the Strix Halo Windows box
Expand Down Expand Up @@ -100,10 +101,11 @@ reports — including partial or failed runs — by scenario id into one HTML re
and GitHub step summary.

The lane artifacts are named canonically (`e2e-report`, `e2e-gpu-report`,
`e2e-gpu-rad3-report`, `e2e-gpu-strix-ubuntu-report`, `e2e-gpu-strix-windows-report`,
`e2e-gpu-strix-wsl-report`) in every workflow, because the report derives each
platform's name and OS from the artifact name. An unrecognised name renders as a
guessed platform on Linux, which would report a Windows lane as Linux; `xtask`'s
`e2e-gpu-rad3-report`, `e2e-gpu-mi350p-report`, `e2e-gpu-strix-ubuntu-report`,
`e2e-gpu-strix-windows-report`, `e2e-gpu-strix-wsl-report`) in every workflow,
because the report derives each platform's name and OS from the artifact name.
An unrecognised name renders as a guessed platform on Linux, which would report
a Windows lane as Linux; `xtask`'s
`every_uploaded_e2e_artifact_has_a_name_the_report_can_label` guards against it.

## Triggers
Expand All @@ -128,11 +130,11 @@ They can also be triggered manually via `e2e-selfhosted.yml`'s
`workflow_dispatch`, independent of the `serve` gate, with these inputs:

- `platform` (choice: `all`, `app-dev-gpu`, `strix-ubuntu`, `strix-windows`,
`strix-wsl`, `rad3`) — which self-hosted job(s) to run. `app-dev-gpu` maps to
`e2e-gpu`, `strix-ubuntu` to `e2e-gpu-strix-ubuntu`, `strix-windows` to
`e2e-gpu-strix-windows`, `strix-wsl` to `e2e-wsl`, and `rad3` to
`e2e-gpu-rad3`. (The mock lane has its own `platform` input on `ci.yml`; it is
not part of this workflow.)
`strix-wsl`, `rad3`, `mi350p`) — which self-hosted job(s) to run. `app-dev-gpu`
maps to `e2e-gpu`, `strix-ubuntu` to `e2e-gpu-strix-ubuntu`, `strix-windows` to
`e2e-gpu-strix-windows`, `strix-wsl` to `e2e-wsl`, `rad3` to
`e2e-gpu-rad3`, and `mi350p` to `e2e-gpu-mi350p`. (The mock lane has its own
`platform` input on `ci.yml`; it is not part of this workflow.)
- `name_filter` (string) — a scenario-name regex forwarded to the cucumber
harness (`cargo xtask e2e -- --name <regex>`) so a dispatch can run a
single scenario instead of the full suite. Empty runs everything applicable
Expand Down Expand Up @@ -214,8 +216,9 @@ the pre-warm block is duplicated across multiple jobs in two shells;
## Blocking vs. non-blocking

The self-hosted jobs — `e2e-gpu`, `e2e-gpu-strix-ubuntu`,
`e2e-gpu-strix-windows`, `e2e-wsl`, and `e2e-gpu-rad3` — all run with `continue-on-error: true`, so a
hardware failure that RUNS never gates a PR merge. Their results still surface
`e2e-gpu-strix-windows`, `e2e-wsl`, `e2e-gpu-rad3`, and `e2e-gpu-mi350p` — all run with
`continue-on-error: true`, so a hardware failure that RUNS never gates a PR
merge. Their results still surface
in the self-hosted consolidated report for visibility.

### Timeouts on the shared Strix box
Expand Down
Loading
Loading