Select devices per-container in the ActorTemplate

[eliranw]

Problem

Follow-up to #502, which landed GPU passthrough at the worker level. A pool requests N GPUs and every actor container gets all of them. The ActorTemplate containers have no say in the device layout, so device assignment is coupled to the worker.

Per Benjamin Elder (@BenTheElder) (review), device assignment should move into the ActorTemplate so it is decoupled from the worker and can control which containers in an actor receive the devices.

Use cases

Actors are pod-shaped (multiple containers, one gVisor sandbox, shared netns), so per-container resource control unlocks:

• Accelerator container plus CPU-only sidecar. The model or trainer container claims the accelerator (nvidia.com/gpu, amd.com/gpu, an FPGA); the logging, proxy, or metrics sidecar next to it claims none. Today every sidecar is handed the device nodes.
• Security isolation. An untrusted or user-supplied sidecar is denied device access as defense in depth. Today it silently receives whatever the pod holds (/dev/nvidia*, an SR-IOV VF), widening the attack surface that feat: CDI-based NVIDIA GPU passthrough into gVisor actor containers #502 otherwise works to keep narrow. The same applies to memory and CPU: a runaway sidecar can currently consume the whole pod budget and starve the container that matters.
• Devices split across containers. On a multi-device node, worker containers each claim a distinct GPU for data-parallel or shard-per-container work over the shared netns.

Proposed API

Declare resources per-container using the standard Kubernetes limits shape. The pool reserves capacity; each container claims from it.

kind: WorkerPool
spec:
  sandboxClass: gvisor
  template:
    resources:
      limits:
        nvidia.com/gpu: "2"        # pool reserves 2 GPUs on the node (unchanged; this is what schedules onto a GPU node)
    containers:
      - name: trainer
        image: pytorch@sha256:...
        resources:
          limits:
            nvidia.com/gpu: "2"    # this container claims both
            memory: 8Gi
      - name: sidecar-logger
        image: fluentbit@sha256:...
        resources:
          limits:
            memory: 256Mi          # capped, and no gpu injected (today it would wrongly get both)

One new field on Container:

// Resources for this container, including device-plugin resources such as
// nvidia.com/gpu.
// +optional
Resources *ContainerResources `json:"resources,omitempty"`

// ContainerResources are the resource limits for one actor container.
type ContainerResources struct {
	// Limits is the maximum amount of compute resources allowed, including
	// device-plugin resources such as nvidia.com/gpu.
	//
	// +optional
	Limits corev1.ResourceList `json:"limits,omitempty"`
}

The YAML matches vanilla Kubernetes per-container limits, and reusing corev1.ResourceList keeps it device-generic for free: extended resources already cover RDMA, FPGA, and others.

Limits only

ContainerResources deliberately does not reuse corev1.ResourceRequirements, which also carries requests and claims.

A restore needs at least the footprint that was captured — micro-VM guest RAM comes back from the memory snapshot, and under gVisor the saved pages would be faulted into a smaller cgroup. A soft request cannot express "must have this much to come back at all".

Adding requests later is additive and non-breaking if a use case appears. The obvious one is cpu.weight, relative priority between sibling containers under contention.

Scope

• Add resources to ActorTemplate.Container, as ContainerResources (limits only).
• CEL validation: extend the existing sandboxClass == 'gvisor' guard to the container level. The type already restricts this to limits, so there is nothing else to reject at admission.
• At assignment time, reject an actor whose container nvidia.com/gpu limits sum to more than the assigned pool's reservation. This cannot be CEL: the reservation lives on the WorkerPool, and the template selects workers by label, so one template may match several pools with different reservations.
• Plumb the per-container limits through WorkerAssignment to ateom.
• In ateom, set spec.Linux.Resources on the OCI spec from the container's limits, and partition the pod's assigned GPUs across the containers that request them, the way the device plugin does in real Kubernetes, instead of the unconditional per-container maybeInjectGPU. The partition must be deterministic, ordered by container.
• Update the API guide.

The cgroup machinery already exists: ensureContainerCgroupsPath gives every actor container its own cgroup leaf, so cpu.max, memory.max and friends are present and unset.

Risks

• Device identity is not stable across resume. CDI names are pod-local, so on restore the actor may land on different physical devices with new UUIDs and PCI IDs. Injection already re-runs on the restore path, so device paths are re-resolved. Remaining care:
  • an app that cached a device UUID or PCI ID before snapshot sees a different one after;
  • the container-to-device partition must be deterministic so a container keeps its ordinal across resume.
• Pool reservation stays the scheduling knob. Keep the pool's resources.limits.nvidia.com/gpu as the reservation and validate that the containers' sum does not exceed it, rather than deriving the pool total from containers. Smaller change, single scheduling knob.

Open questions

Can per-container CPU and memory be enforced? google/gvisor#190 suggests cgroup limits land on the sandbox rather than the container, since one sentry backs them all. Micro-VM actors have a real guest kernel, so it may work there. Worth settling before the field covers more than devices.

Out of scope

• DRA. Dynamic Resource Allocation (resource.k8s.io, GA in 1.34) is the home for physical index pinning, topology co-allocation, and device sharing such as MIG and time-slicing. If those are needed, revisit through a DRA ResourceClaim rather than extending counts.
• CPU pinning. cpuset.cpus exists on the per-container cgroup, but counts cannot express it and it belongs with topology above. Under gVisor it would also pin the sentry's host threads rather than CPUs the sandboxed app sees.
• Non-GPU device classes. The surface allows them; the implementation lands with the first such device.
• Snapshot and restore of live device state, such as a live CUDA context. Tracked separately by Snapshot/restore for GPU actors (suspend/resume with a live CUDA context) #779.

References

• GPU support for gVisor actors #627: GPU support for gVisor actors, the parent this follows up.

[BenTheElder]

Actors are pod-shaped (multiple containers, one gVisor sandbox, shared netns), so per-container device control unlocks:

It also unlocks a possible future where we are not 1:1 with actor : pod.

Declare devices per-container using the standard Kubernetes resources shape, the same corev1.ResourceRequirements type the pool already uses. The pool reserves capacity; each container claims from it.

I'm wondering how much requests vs limits makes sense and if we should allow requests at this stage. Like what would it mean if you requested 2GI for uVM and then on restore got only 1?

cc Julian Gutierrez Oschmann (@juli4n)

[eliranw]

I agree limits only fits better, restore needs at least the same amount of memory, and for devices it’s based on limits anyway. I’ll update the design.
Separately I looked at whether per-container cpu/memory is even enforceable in gVisor, saw this google/gvisor#190. Are cgroup limits in gVisor at the sandbox level only? Seems like it might be an issue