Skip to content

feat(remote): serve a model on a tailnet GPU machine #333

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
volen-silo wants to merge 3 commits into
base: main
Choose a base branch
from
Open

feat(remote): serve a model on a tailnet GPU machine #333

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
549a611
feat(remote): serve a model on a tailnet GPU machine
volen-silo Sep 14, 2026
00add7c
test(remote): cover the remote control channel against a real SSH server
volen-silo Sep 14, 2026
b2c5dd9
docs(remote): document the remote commands and how to exercise them
volen-silo Sep 14, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
60 changes: 60 additions & 0 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -122,6 +122,9 @@ jobs:
- '**/*.ps1'
- '**/*.feature'
- 'tests/e2e-cucumber/**'
# The containerised control-channel lane; its Dockerfile matches
# none of the extension globs above.
- 'tests/remote-ssh/**'
- 'install*'
# Pinned-key consistency compares docs/keys/* against the installers,
# so a canonical-key-only change must trigger the heavy job that runs
Expand Down Expand Up @@ -644,6 +647,55 @@ jobs:
prek-version: 0.4.5
extra-args: --all-files --no-group local-tools

remote-ssh:
# `rocm remote` drives ssh, scp, and the remote's service/serve tooling. Its
# unit tests use a scripted stand-in, which proves the control flow but
# assumes the real tools behave a certain way. This job checks that
# assumption against a real OpenSSH server in a container: argument
# handling, exit-code propagation, stdin delivery of a credential, file
# copy, batch-mode refusal, the shape Tailscale Funnel takes in the serve
# config (including that the fake refuses the ports the real daemon
# refuses), and — the one that matters most — that withdrawing a published
# endpoint actually removes it. A publish is configuration rather than a
# process, so it survives reboots, and a withdrawal that silently does
# nothing leaves a GPU endpoint exposed with nothing tracking it.
#
# No GPU, no ROCm, no tailnet: the remote's `rocm` and `tailscale` are
# stand-ins that answer in the shapes the real tools do.
name: remote control channel (containerised)
runs-on: ubuntu-latest

@juhovainio juhovainio Sep 15, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor: every other job gated on heavy in this workflow (build-and-test, test, windows-build-and-test) has needs: [changes, clippy, prek], specifically so a fast lint failure short-circuits before spending build time. This new job only has needs: changes, so it'll run its full 30-minute container+cargo cycle even when clippy or prek would have failed instantly.

@volen-silo volen-silo Sep 15, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not addressed yet. Confirmed: every other heavy-gated job carries needs: [changes, clippy, prek] and this one has only needs: changes, so it burns the full container and cargo cycle on runs a lint failure would have ended in seconds.

This is also a prerequisite for the other open question on this lane — whether it should be a required check — because the gating has to be right before anything depends on it.

timeout-minutes: 30

@juhovainio juhovainio Sep 15, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor: the other heavy-gated jobs in this file also carry if: github.event_name != 'workflow_dispatch', so a manual dispatch stays a fast loop. This job doesn't have that guard, so a manual dispatch will now also pull in this 30-minute containerized job.

@volen-silo volen-silo Sep 15, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not addressed yet, and it belongs with the needs: change above rather than separately — both are about this job not matching the conventions of its siblings in the same file.

needs: changes
if: needs.changes.outputs.heavy == 'true'
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1

# Docker and the OpenSSH client are both present on ubuntu-latest; jq is
# the only extra, and only for asserting on JSON contracts.
- name: Install jq
run: sudo apt-get update && sudo apt-get install -y jq

- name: Check the control-channel contracts
run: tests/remote-ssh/run.sh

# The pinned toolchain, and the rust-cache that comes with it, as every
# other cargo job in this workflow uses. Without it this lane builds the
# whole workspace cold against a different compiler from the rest of CI,
# inside a 30-minute timeout.
- uses: actions-rust-lang/setup-rust-toolchain@166cdcfd11aee3cb47222f9ddb555ce30ddb9659 # v1.17.0

# The same container, driven by the real binary this time: discover a
# target, probe it, serve, publish, reconcile status, re-publish after an
# out-of-band withdrawal, tear down, and refuse to publish over a
# Funnel-exposed port. Catches the orchestration mistakes a scripted
# stand-in cannot — an argument the CLI builds wrongly, a session record
# it fails to clean up.
- name: Build the CLI for the end-to-end run
run: cargo build -p rocm

- name: Drive rocm remote end to end
run: tests/remote-ssh/run-e2e.sh

clippy:
runs-on: ubuntu-latest
timeout-minutes: 20
Expand Down Expand Up @@ -1053,6 +1105,14 @@ jobs:

- name: Run E2E tests
if: steps.gate.outputs.run == 'true'
# E2E_INCLUDE_DOCKER turns on the `rocm remote` scenarios that need a
# second machine, stood up as a container. This is the only lane that
# opts in: GitHub-hosted runners can build the fixture image, while the
# self-hosted GPU runners have a daemon but no route to the package
# mirror it installs from, so they skip rather than fail on an image
# they could never build.
env:
E2E_INCLUDE_DOCKER: "1"
run: cargo xtask e2e

- name: Upload E2E report
Expand Down
43 changes: 42 additions & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -221,6 +221,7 @@ form works depends on the engine your GPU selects.
| `rocm install sdk` | Install TheRock ROCm wheels into a managed Python environment |
| `rocm install driver` | Install the AMD kernel driver on Linux |
| `rocm serve <model>` | Start a local OpenAI-compatible model server |
| `rocm remote serve <machine> <model>` | Serve a model on another GPU machine over your private network |
| `rocm dash` | Open the full-screen telemetry dashboard |
| `rocm setup status` | Show first-time setup state |
| `rocm version` | Print the rocm-cli version |
Expand Down Expand Up @@ -415,12 +416,52 @@ model across multiple GPUs is not supported. Because selection uses the
Manage background servers started with `--managed`:

```
rocm services list [--all]
rocm services list [--all] [--json]
rocm services logs <service-id>
rocm services stop <service-id> [--yes]
rocm services restart <service-id> [--yes]
```

`--json` prints the service records verbatim, for scripting and for the remote
orchestration below.

### Remote machines (preview)

Run a model on a different GPU machine and reach it from your own. Both machines
join a [Tailscale](https://tailscale.com) network; the GPU machine serves the
model on its own loopback address and publishes that port onto the network, so
the endpoint keeps working after the command exits and answers from any of your
machines rather than only the one that started it.

```console
rocm remote targets [--tag <tag>]
rocm remote doctor <machine> [--symptom <text>]
rocm remote serve <machine> <model> [--engine <engine>] [--gpu <index>]
[--tailnet-port <port>] [--install-rocm]
rocm remote status [<session>]
rocm remote attach <session>
rocm remote stop <session> [--force]
```

`targets` lists machines on your network — it does not check whether they can
actually serve, which is what `doctor` is for. `serve` prepares the machine
(installing the CLI if it is missing), starts the model, publishes the endpoint
and prints the address together with an API key.

**The endpoint is reachable by every machine on your network that your network's
access rules allow**, not just yours. That API key is what stops anyone else
using it, so `rocm remote` always sets one — unlike local serving, which is
credential-free because only your own machine can reach it.

`status` reports the model and the endpoint separately, because either can fail
alone: a healthy model with no endpoint needs `attach`, not a restart. `stop`
withdraws the endpoint and stops the model, and keeps the session listed if it
cannot confirm both — use `--force` to forget one whose machine is gone.

Communication with the machine uses your existing `ssh` setup. Set
`ROCM_REMOTE_SSH_CONFIG` to point at a configuration file other than the
default.

### Dashboard

```
Expand Down
Loading
Loading