feat: CDI-based NVIDIA GPU passthrough into gVisor actor containers by eliranw · Pull Request #502 · agent-substrate/substrate

eliranw · 2026-07-23T14:07:24Z

Summary

atecontroller propagates a pool's nvidia.com/gpu request onto the ateom container and mounts the host NVIDIA toolkit read-only (path overridable via ATE_NVIDIA_TOOLKIT_HOST_PATH)
ateom-gvisor generates a CDI spec with nvidia-ctk and injects the device nodes, driver-library mounts, and env into each actor container's OCI spec
runs the CDI createContainer hooks except update-ldcache which needs a privileged ateom, staging the SONAME symlinks it would create from each library's ELF DT_SONAME
enables runsc --nvproxy at sandbox creation

Requesting nvidia.com/gpu on the pool is the only configuration needed; a pool that requests N GPUs makes all N usable.

Two details of the CDI spec are worth calling out, because getting either wrong fails at runtime rather than at parse time. nvidia-ctk leaves major/minor unset — CDI delegates that to the OCI runtime — so each device node is resolved by stat-ing the host; without it the actor gets 0,0 char devices and NVML reports it cannot communicate with the driver. And it emits per-index, per-UUID, and all devices that repeat the same nodes, so only all is applied. The spec is plain JSON, so encoding/json suffices and no CDI library is vendored.

update-ldcache is the one hook that cannot run here: its ldconfig unshares a mount namespace and mounts a private /proc, which mount_too_revealing() rejects under the pod's masked /proc. Permitting it would need procMount: Unmasked, which Kubernetes only allows with hostUsers: false, and that user namespace breaks the per-actor cgroup delegation from #496. Skipping it avoids the whole chain, so a GPU worker keeps the same posture as any other unprivileged gVisor worker. create-symlinks and enable-cuda-compat still run unmodified.

--nvproxy must be set when the sandbox is created — the pause container, which holds no GPU devices — so runsc's auto-detection never fires on its own; without the flag the GPU subcontainer crashes the sentry on start. GPU detection matches any device index rather than assuming /dev/nvidia0, since a worker sharing a multi-GPU node can be assigned /dev/nvidia2 and /dev/nvidia3.

GPU pools must set spec.ateomImage to a glibc build (KO_DEFAULTBASEIMAGE=debian:stable-slim ko build ./cmd/ateom-gvisor) because the distroless default cannot exec nvidia-ctk; the default base is unchanged for every other pool. atelet also has to run on the GPU nodes to restore actors there, so its DaemonSet needs a toleration for whatever taint they carry. Both are documented in the API guide rather than defaulted.

Testing

make test
env -u NO_COLOR make verify
Real GPU, Tesla T4 / driver 580.65.06, through the full actor flow: nvidia-smi, vectorAdd, nbody at 3.77 TFLOP/s, PyTorch matmul via cuBLAS at 3.5 TFLOP/s (T4 peak FP32 is ~8.1, so no measurable sandbox penalty)
Actor whose entrypoint runs the CUDA sample directly exits 0, confirming the injected env reaches the workload without help from the test harness
Two workers holding two GPUs each on one 4-GPU node see disjoint device sets

Snapshot and restore work when the workload holds no CUDA context. A live CUDA context cannot be checkpointed - gVisor fails with can't save with live nvproxy clients and the failed checkpoint terminates the sandbox so a GPU actor can only be suspended between CUDA workloads. Documented as a known limitation in the API
guide; a follow-up issue will track lifting it via cuda-checkpoint.

Fixes #627

Tests pass
Appropriate changes to documentation are included in the PR

google-cla · 2026-07-23T14:07:35Z

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

Davanum Srinivas (dims) · 2026-07-23T14:18:36Z

eliranw is there a way to do this without introducing all the new vendored components? (21k lines of code is too much!)

eliranw · 2026-07-23T15:27:26Z

It seems like we decided to commit vendor packages in this repo. I think it’s best to use the cdi package instead of reimplementing it. As far as I understand, the main downside of committing the vendor folder (instead of gitignoring it) is having a large diff. Do you have a different solution?

To make it easier to review, I moved the vendor changes into a separate commit. I noticed many other PRs do the same thing - does this approach make sense to you?

Davanum Srinivas (dims) · 2026-07-23T15:51:36Z

eliranw it's not whether the vendor-ed stuff is a separate commit or not. just that we should avoid adding stuff if not really required.

see for example - main...dims:substrate:gpu-nocdi

Benjamin Elder (BenTheElder) · 2026-07-23T16:19:24Z

thanks for working on this

I'd also like to keep imported deps minimal.

I started looking at uVM without privileged and we may have a bigger problem there ... ideally I don't want to require a plugin on the host but we need permission to talk to KVM.

I may have an answer to that wrinkle but broadly speaking I'm also hoping that even if not implemented yet we can figure out how we'll extend this to uVM later, as I'm also working on closing the existing gap (durable directory support) at higher priority ...

eliranw · 2026-07-23T18:37:03Z

Thanks Davanum Srinivas (@dims) and Benjamin Elder (@BenTheElder). Dropped the vendored CDI as you both suggested and re-tested on a GPU node.

I’ve also gated the whole thing to gVisor-only for now so the NVIDIA toolkit mount only lands on gVisor pools, meaning a micro-VM pool that requests a GPU won’t pick up the nvidia toolkit.

On uVM - I'll start looking into that

eliranw · 2026-07-30T19:45:04Z

Benjamin Elder (@BenTheElder) Davanum Srinivas (@dims) Omer Yahud (@omeryahud) - I've updated this PR to reflect #496 (unprivileged ateom).

I had a few issues getting the GPU CDI hooks working, but right now GPU pools keep exactly the same posture as any other gVisor worker - no user namespace, no procMount: Unmasked, and per-actor cgroup delegation still working. The issue was that only one CDI hook was failing: update-ldcache, which needs the private /proc; skipping just that hook (and writing the SONAME symlinks from DT_SONAME instead) avoids the issue I had. The other CDI hooks run unmodified. I also needed a debian:stable-slim gvisor image to run the cdi generation tool.
I also dropped the vendored CDI library per your earlier comment.

I did a full test end to end on a cluster with a T4 node workerpool on GKE with different NVIDIA tooling to check compatibility: nvidia-smi, vectorAdd, nbody, and a PyTorch cuBLAS matmul, plus two workers with two GPUs each on one 4-GPU node to check isolation betweek workers on a single node. I've added details in the updated description.

Omer Yahud (omeryahud) · 2026-08-03T07:16:16Z

+	// (see nvproxyGlobalArgs). Set once here at boot to avoid a data race on the
+	// concurrent RunWorkload path.
+	if gpuPresent() {
+		runscNvproxy = true


I don't like passing arguments in a back channel, let's pass it as a function argument to maybeInjectGPU.

You can leave the check here for logging purposes

instead of plumbing it I dropped the global entirely and had that call gpuPresent() directly, kept the check

Omer Yahud (omeryahud) · 2026-08-03T08:03:27Z

+// maybeApplyGPUPodShape shapes a gVisor worker pod that requests a GPU so ateom
+// can inject the GPU into actors via CDI. It mounts the host NVIDIA toolkit
+// (version-matched to the node) — the glibc-based ateom image runs those toolkit
+// binaries directly — and applies the user-namespace + unmasked-/proc posture the


Is this correct? A left-over from a previous implementation?

Yes, the host path was defaulting to the container path, so they’re split now (/usr/local/nvidia/toolkit vs /opt/nvidia-toolkit, which has to sit outside the device plugin’s read-only tree), and I fixed a stale comment beside it still claiming the function applies a userns + unmasked /proc.

Davanum Srinivas (dims) · 2026-08-04T13:51:14Z

Nice work. The pause container getting --nvproxy at create, the cgroup device allow rules, and the NVIDIA_VISIBLE_DEVICES strip are all subtle and correct.

One question before the details: what happens to the golden snapshot when the workload holds a live CUDA context at suspend time? Doesn't gVisor refuses to save a sandbox with open GPU state? can we cross check if the test plan accounts for this?

Davanum Srinivas (dims) · 2026-08-04T13:52:31Z

+		spec.Hooks = &specs.Hooks{}
+	}
+	for _, h := range edits.Hooks {
+		if h.HookName != "createContainer" || len(h.Args) < 2 || h.Args[1] == "update-ldcache" {


Should this be an allowlist and not a deny list? Newer toolkits could emit more hooks

Sounds very reasonable, so I switched to an allowlist with logging of skipped hooks so we see them on newer toolkits as/when they come.

Davanum Srinivas (dims) · 2026-08-04T13:53:21Z

+only to the container that requests it. If per-container resource limits are added
+later, GPU assignment should follow them.
+
+**Requirements**


Do we need to say anything about LD_LIBRARY_PATH?

Yes, added.
For context, CDI doesn’t emit LD_LIBRARY_PATH at all, the stock mechanism is the update-ldcache hook, which we skip for the unprivileged posture, so nothing puts the driver dir on the loader path. Every image I’d tested was an nvcr.io/nvidia/* one that sets it itself; override it and CUDA sees 0 devices even though the GPU is fully injected. Now prepending the driver library directory, with a unit test and a docs note.

Davanum Srinivas (dims) · 2026-08-04T13:54:23Z

+// SONAME symlinks (libcuda.so.1 -> libcuda.so.580.x) into the actor rootfs, which is
+// what lets the GPU worker keep the plain unprivileged posture (no user namespace, no
+// unmasked /proc). The CDI spec is plain JSON, so no CDI library is needed.
+func injectGPUIntoBundle(ctx context.Context, bundleDir, cdiSpecDir string) error {


Does atelet always rewrite the bundle before each Run/Restore attempt? If not, a second call here doubles every device, mount, env, and hook. A guard that skips injection when an nvidia device is already in spec.Linux.Devices would make it safe either way.

It does, but I added a guard so we don't depend on the atelet, plus a regression test, and confirmed the test fails without it.

Davanum Srinivas (dims) · 2026-08-04T13:56:33Z

-		containerName, // Name of the container
-	)
+	}
+	startArgs = append(startArgs, nvproxyGlobalArgs()...)


Is --nvproxy needed on start? do we want to be consistent where all we add --nvproxy?

I thought so, but I tested removing it and it worked, kept it on create and restore, which actually boot the sentry process. start acts on a sandbox that already exists, so the flag has no effect there.

Davanum Srinivas (dims) · 2026-08-04T13:58:00Z

+const (
+	// driverLibDir is where the CDI/GKE device plugin injects the driver user-mode
+	// libraries into a GPU pod; nvidia-ctk needs it on its library search path.
+	driverLibDir = "/usr/local/nvidia/lib64"


do we need this to be overridable?

Yes, both of the variables you commented on now derive from one ATE_NVIDIA_DRIVER_ROOT (default /usr/local/nvidia).

Davanum Srinivas (dims) · 2026-08-04T13:58:09Z

+	// driverBinDir holds the driver binaries (nvidia-smi, ...). nvidia-ctk discovers
+	// them via PATH, so we prepend this when generating the CDI spec — otherwise the
+	// spec carries only libraries and the actor has no nvidia-smi.
+	driverBinDir = "/usr/local/nvidia/bin"


Yes, both of the variables you commented on now derive from one ATE_NVIDIA_DRIVER_ROOT (default /usr/local/nvidia).

Davanum Srinivas (dims) · 2026-08-04T13:59:24Z

+// directly. The pod keeps the same security posture as any other gVisor worker.
+// No-op for non-GPU pools and non-gVisor classes; an empty class defaults to
+// gVisor (WorkerPoolSpec kubebuilder default).
+func maybeApplyGPUPodShape(


Hmm, A microvm pool that requests a GPU silently gets none, should this fail harder? earlier?

So to mitigate microVM pools with GPUs until/when actors there get support, I added a CEL rule on WorkerPoolSpec + tests - https://github.com/agent-substrate/substrate/pull/502/changes#diff-1ede08f8c093df5328ddbd446e4621241be469d4970f73b824426b601b35bab3R54

Signed-off-by: Eliran Wolff <eliranw@nvidia.com>

Generate a per-pod CDI spec with nvidia-ctk, parse it in-tree, and inject the device nodes (major/minor stat-resolved) of the "all" CDI device, the driver- library mounts, and env into each actor's OCI spec. Run the CDI createContainer hooks from the mounted toolkit except update-ldcache, whose ldconfig needs a private /proc mount; stage the SONAME symlinks it would create from each library's ELF DT_SONAME instead. That keeps the GPU worker on the same unprivileged posture as any other gVisor worker (no user namespace, no procMount:Unmasked, cgroup delegation intact). Enable runsc --nvproxy at sandbox creation. Detect GPUs by device-node glob so any assigned index works (multi-GPU). The distroless ateom cannot exec nvidia-ctk, so GPU pools must run a glibc ateom build (WorkerPool.spec.ateomImage). Signed-off-by: Eliran Wolff <eliranw@nvidia.com>

Signed-off-by: Eliran Wolff <eliranw@nvidia.com>

GPU passthrough is implemented only for the gVisor runtime, but the pod template's resources are applied before the sandbox-class check, so a micro-VM pool requesting nvidia.com/gpu still got the request on its worker pod. The pod then scheduled onto a GPU node and held a device that no actor could use. Reject the combination at apply time with a CEL rule on WorkerPoolSpec, matching the cross-field rules ActorTemplate already carries. The rule keys off limits or requests, mirroring the pod-shaping check, and is written positively so a future sandbox class must opt in rather than silently inherit GPU support. Signed-off-by: Eliran Wolff <eliranw@nvidia.com>

eliranw · 2026-08-04T19:28:17Z

One question before the details: what happens to the golden snapshot when the workload holds a live CUDA context at suspend time? Doesn't gVisor refuses to save a sandbox with open GPU state? can we cross check if the test plan accounts for this?

Davanum Srinivas (@dims) Good catch as the test plan didn’t cover it, and with a live context the checkpoint fails.
After this is merged I’d open a separate issue to track GPU suspend/resume using cuda-checkpoint, linked to #450 since utilizing that likely needs PreSuspend/PostResume lifecycle hooks. For now it’s documented as a known limitation: a GPU actor can only be suspended with no CUDA context open, otherwise the checkpoint fails and the sandbox is terminated. Do you think that’s enough for now?

Davanum Srinivas (dims) · 2026-08-04T19:36:06Z

Do you think that’s enough for now?

eliranw will leave this to the maintainers of this repo! :)

eliranw force-pushed the eliranw/gvisor-gpu-poc branch from b510ec9 to 2693528 Compare July 23, 2026 14:17

eliranw force-pushed the eliranw/gvisor-gpu-poc branch from 2693528 to 9cf40a5 Compare July 23, 2026 15:41

eliranw force-pushed the eliranw/gvisor-gpu-poc branch 3 times, most recently from f896c76 to 5320561 Compare July 23, 2026 18:31

eliranw force-pushed the eliranw/gvisor-gpu-poc branch 3 times, most recently from 37f756b to 39d9060 Compare July 30, 2026 19:27

Omer Yahud (omeryahud) reviewed Aug 1, 2026

View reviewed changes

Comment thread docs/api-guide.md Outdated

Omer Yahud (omeryahud) reviewed Aug 1, 2026

View reviewed changes

Comment thread docs/api-guide.md

Omer Yahud (omeryahud) reviewed Aug 3, 2026

View reviewed changes

eliranw force-pushed the eliranw/gvisor-gpu-poc branch 3 times, most recently from 01a3e03 to c5c0ab6 Compare August 3, 2026 14:20

eliranw requested a review from Omer Yahud (omeryahud) August 3, 2026 15:12

Davanum Srinivas (dims) reviewed Aug 4, 2026

View reviewed changes

feat(atecontroller): mount NVIDIA toolkit into GPU worker pods

9c5f8fa

Signed-off-by: Eliran Wolff <eliranw@nvidia.com>

eliranw force-pushed the eliranw/gvisor-gpu-poc branch from c5c0ab6 to 22f4b82 Compare August 4, 2026 18:08

eliranw added 3 commits August 4, 2026 21:39

docs: document GPU passthrough into gVisor actors

19ddd9e

Signed-off-by: Eliran Wolff <eliranw@nvidia.com>

eliranw force-pushed the eliranw/gvisor-gpu-poc branch from 22f4b82 to 63b24c8 Compare August 4, 2026 18:39

Uh oh!

Conversation

eliranw commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Testing

Uh oh!

google-cla Bot commented Jul 23, 2026

Uh oh!

Davanum Srinivas (dims) commented Jul 23, 2026

Uh oh!

eliranw commented Jul 23, 2026

Uh oh!

Davanum Srinivas (dims) commented Jul 23, 2026

Uh oh!

Benjamin Elder (BenTheElder) commented Jul 23, 2026

Uh oh!

eliranw commented Jul 23, 2026

Uh oh!

eliranw commented Jul 30, 2026

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Davanum Srinivas (dims) commented Aug 4, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

eliranw Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

eliranw commented Aug 4, 2026

Uh oh!

Davanum Srinivas (dims) commented Aug 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

eliranw commented Jul 23, 2026 •

edited

Loading

eliranw Aug 4, 2026 •

edited

Loading