feat(vm): support corporate HTTP forward proxy egress for microVM sandboxes - #3090
feat(vm): support corporate HTTP forward proxy egress for microVM sandboxes#3090feloy wants to merge 2 commits into
Conversation
BlockedGator is blocked because PR #3090 currently has merge conflicts with Next action: @feloy, please update the branch from Gator metadata
|
johntmyers
added
the
gator:blocked
johntmyers
left a comment
johntmyers
left a comment
There was a problem hiding this comment.
gator-agent
PR Review Status
The merge-conflict blocker from the previous head is resolved. The initial code review found two concrete blockers in the new VM proxy path: CA bundle staging can consume an unbounded special file and accepts certificates the guest will reject, and the published loopback recipe is incorrect for the supported QEMU/TAP backend.
Action required: @feloy, please address GATOR-dc1218b5-01 and GATOR-dc1218b5-02, then push an update.
Blocking findings:
GATOR-dc1218b5-01: bound and fully validate the host CA bundle before staging it.GATOR-dc1218b5-02: make the host-loopback guidance accurate for both libkrun/gvproxy and QEMU/TAP.
Carried findings:
- None
Gator metadata
- Validation: Project-valid implementation of linked issue #3088 with a clear VM corporate-proxy user path, threat-boundary analysis, docs, and E2E coverage.
- Docs: Updated, but the host-loopback contract is materially incorrect for QEMU/TAP.
- Checks: Current-head Branch Checks and Helm Lint gates are pending; required branch workflows have not been dispatched.
- E2E:
test:e2eis required for VM networking and credential flow; dispatch is deferred until review blockers are resolved. - Head SHA:
dc1218b51ac14f1315c1a633cae316ff45963549 - Base SHA:
7b64c5c88ece0878592160cb86da5e0fd1f64158 - Merge base SHA:
7b64c5c88ece0878592160cb86da5e0fd1f64158 - Patch ID:
e940265736870bac2a568c5f4c0be0ba145c2786 - Gator payload:
8 - Review mode:
initial - Previous reviewed SHA: none
- Review budget exhausted: no
- Maintainer decision required: no
- Next state:
gator:in-review
| /// failure inside every sandbox. | ||
| async fn read_sandbox_proxy_ca_bundle(path: &str) -> Result<Vec<u8>, Status> { | ||
| let path_owned = path.to_string(); | ||
| let bytes = tokio::task::spawn_blocking(move || fs::read(&path_owned)) |
There was a problem hiding this comment.
gator-agent
Warning — GATOR-dc1218b5-01 · Bound and fully validate the proxy CA file
Summary: After an operator configures proxy_ca_bundle, any authorized VM creation reads that path fully into memory. A special file such as /dev/zero can OOM-kill the VM driver, while PEM-looking invalid DER passes this check and then makes every guest supervisor fail after boot.
Fix: Read only a bounded regular file, validate at least one certificate through the same rustls_pemfile and RootCertStore::add_parsable_certificates path as the supervisor, and cover oversized/non-regular input plus invalid DER.
Verify: Configure /dev/zero under a memory limit, then an AAAA PEM block. Both creates must return a host-side invalid-argument error without unbounded growth or guest boot.
Agent context
- Location:
crates/openshell-driver-vm/src/driver.rs:5378 - Ownership: This PR introduces the VM CA-file read and claims host acceptance matches guest acceptance.
There was a problem hiding this comment.
Fixed. Both halves of the finding were real: the read was unbounded, and the
PEM marker substring check accepted bundles the guest then rejects.
The read moved into openshell-core::driver_utils::read_upstream_proxy_ca_bundle_file,
which reuses the bounded-read path the credential reader already had — the two
now share one private read_regular_file_bounded helper, so a non-regular path
is rejected on the handle's fstat (nothing is read from /dev/zero at all),
the file is capped at MAX_UPSTREAM_PROXY_CA_BUNDLE_BYTES (1 MiB), and the
read stays bounded even if the file grows between stat and read. On Unix the
open is non-blocking, so a FIFO cannot hang the driver either.
Validation is now the same code the supervisor runs, not an equivalent-looking
copy: rustls_pemfile::certs followed by
RootCertStore::add_parsable_certificates, requiring at least one anchor
rustls actually accepts. openshell-supervisor-network's read_proxy_ca_bundle
delegates to the shared function, so host acceptance and guest acceptance
cannot drift — and the guest-side read is now bounded too, which it was not
before.
Error messages keep the proxy_ca_bundle '<path>' prefix and never echo file
contents; the driver still surfaces them as gRPC InvalidArgument at sandbox
create.
Tests, per the finding's verify note:
openshell-core:ca_bundle_file_accepts_a_real_certificate(rcgen-minted
CA, the positive case pinning host acceptance to guest acceptance),
ca_bundle_file_rejects_non_regular_and_oversized_paths(a directory,
/dev/zero, and a file one byte over the bound),
ca_bundle_file_missing_path_is_an_error,
ca_bundle_rejects_a_file_without_certificate_blocks, and
ca_bundle_rejects_pem_blocks_holding_invalid_der(theAQIDblock).openshell-driver-vm:proxy_ca_bundle_without_a_certificate_fails_the_sandbox
now also assertsInvalidArgumentfor/dev/zero, for an oversized file, and
for a PEM block with invalid DER.- The existing supervisor tests for the same cases still pass against the
shared implementation.
| # into the overlay upperdir on every launch, so a sandbox image cannot supply | ||
| # its own values or disable the operator's by baking a file at that path. | ||
| # | ||
| # Reachability: guest egress leaves through gvproxy, so a proxy listening on |
There was a problem hiding this comment.
gator-agent
Warning — GATOR-dc1218b5-02 · Scope the loopback recipe to the actual VM backend
Summary: A Linux GPU VM uses QEMU/TAP, where host.openshell.internal maps to the TAP gateway rather than gvproxy's 192.168.127.254 NAT. An operator following this generic recipe with a proxy bound only to 127.0.0.1 therefore loses all proxy-required egress even though configuration validation succeeds.
Fix: Either add and verify equivalent QEMU/TAP loopback forwarding, or update every new reference to limit this NAT recipe to libkrun/gvproxy and tell QEMU/TAP operators to use an address reachable through the TAP gateway.
Verify: With a loopback-only proxy, compare an allowed TLS request from libkrun and GPU-backed QEMU sandboxes; documented configurations must reach the proxy or explicitly reject unsupported loopback parity.
Agent context
- Location:
docs/reference/gateway-config.mdx:813 - Sibling docs:
docs/reference/sandbox-compute-drivers.mdx:369,crates/openshell-driver-vm/README.md:157,architecture/sandbox.md:316,.agents/skills/debug-openshell-cluster/SKILL.md:629
There was a problem hiding this comment.
Fixed, and the finding understated it slightly — worth recording what the code
actually does.
On QEMU/TAP the guest's /etc/hosts maps host.openshell.internal to
VM_NET_GW, the TAP host address (write_host_gateway_aliases in
openshell-vm-sandbox-init.sh), so gvproxy's 192.168.127.254 NAT is indeed
absent. But a proxy bound to that TAP host address does not work either:
nft_ruleset.rs installs an input chain that accepts only
tcp dport <gateway_port> from the TAP interface and drops the rest, and an
operator cannot lift that from another table — a drop in the driver's chain
is terminal at that hook. So on QEMU/TAP no proxy on the gateway host is
reachable, at any bind address, not only a loopback-bound one.
Taking the second branch of your fix suggestion, since equivalent loopback
forwarding cannot be added without widening what the guest may reach on the
host:
- Every reference now scopes the gvproxy NAT recipe to the libkrun backend and
states the QEMU/TAP contract explicitly:docs/reference/gateway-config.mdx,
docs/reference/sandbox-compute-drivers.mdx(linking its own Host Firewall
section),crates/openshell-driver-vm/README.md,architecture/sandbox.md,
.agents/skills/debug-openshell-cluster/SKILL.md, and the module docs of
e2e/rust/tests/vm_corporate_proxy.rs, which prove the libkrun rule only. - Documentation alone would still leave the silent-timeout failure you
describe, so the driver now takes the "explicitly reject unsupported loopback
parity" option:configure_qemu_launch_planfails the create with
FailedPreconditionwhenhttps_proxyaddresses the gateway host — loopback
literals,localhost,192.168.127.254, or any of the
host.openshell.internal/host.containers.internal/host.docker.internal
aliases (proxy_url_targets_gateway_host). The message names the backend
constraint and the remedy. Unit test:
qemu_backend_rejects_a_gateway_host_proxy.
The check is on the launch plan rather than on config validation because the
backend is chosen per sandbox (GPU ⇒ QEMU), so the same gateway config remains
valid for libkrun sandboxes on the same host.
johntmyers
added
gator:in-review
…dboxes The corporate forward proxy machinery from NVIDIA#1792 is driver-agnostic and already merged: openshell-supervisor-network implements CONNECT chaining, NO_PROXY matching, credentials, https:// proxies and corporate CA trust, and openshell-sandbox exposes it as six argv-only flags. Podman gained the driver half in NVIDIA#2245/NVIDIA#2512 and Kubernetes in NVIDIA#2633; the VM driver had none of it, so VM sandboxes on proxy-only networks could not reach any destination requiring the proxy even when policy allowed it. The blocking piece was not proxy logic but delivery: the VM guest init script runs as PID 1 and execs a fixed supervisor command line, and libkrun's krun_set_exec receives an empty argv, so there was no channel for driver-owned supervisor arguments. The supervisor's proxy flags deliberately have no environment fallback, and build_guest_environment merges user-supplied environment, so the guest env is not a safe transport either. Add a driver-authored argument file, mirroring the existing init.d manifest: the driver writes /opt/openshell/supervisor-args into the overlay upperdir on every launch and the guest reads it verbatim, one argument per line, appending it to every supervisor exec. It is written even when empty, which is what makes the channel unforgeable -- the upperdir always shadows the read-only image layer, so an image can neither supply its own arguments nor disable the operator's by omitting the file. Because both launch backends exec the same init script, this covers libkrun and QEMU without touching either. A microVM has no bind mounts or container secrets, so the credential and CA bundle are staged into the per-sandbox overlay the way the gateway JWT already is: credential root-only at 0600, CA at 0644, both rewritten every launch so a removed setting clears prior material, and both deleted with the sandbox state directory. This places the credential at rest in the overlay image on the gateway host, which differs from the Podman secret model and is documented as an explicit security consideration. Validation is fail-closed and shared: a new openshell_core::driver_utils::validate_upstream_proxy_settings holds the pairing rules the Podman driver established, and both the gateway and the driver call it so an invalid table names the offending key instead of surfacing as an opaque driver-readiness timeout. Guest egress leaves through gvproxy, so a proxy on the gateway host's loopback is reachable only through host.openshell.internal; the guest to gateway callback is unaffected. Closes NVIDIA#3088 Signed-off-by: Philippe Martin <phmartin@redhat.com>
Two review findings on the corporate forward proxy support for microVM sandboxes. The driver read the operator's proxy_ca_bundle with an unbounded fs::read and accepted it on a substring match for the PEM BEGIN CERTIFICATE marker. A special file such as /dev/zero therefore grew driver memory without bound on every authorized sandbox create, and a PEM block holding invalid DER passed the host check but contributes no trust anchor in the guest, so every supervisor would fail after boot with an error attributed to the sandbox rather than to the setting. Move the read into openshell-core as read_upstream_proxy_ca_bundle_file: it reuses the credential reader's bounded-read path (non-regular files rejected on fstat, size capped, read bounded even if the file grows), then requires at least one anchor that RootCertStore::add_parsable_certificates accepts. The supervisor's own reader now delegates to it, so host acceptance and guest acceptance are the same function and cannot drift. The published host-loopback recipe was written for libkrun only. gvproxy NATs host.openshell.internal to the gateway host's 127.0.0.1, but GPU sandboxes run on the QEMU/TAP backend where that name resolves to the TAP host address and the driver's own nftables input chain accepts only the gateway port from the guest — no proxy on the gateway host is reachable there at any bind address, so an operator following the generic recipe lost all proxy-required egress while configuration validation succeeded. Scope the recipe to libkrun in every reference and reject a gateway-host proxy URL when a launch plan resolves to QEMU, naming the reason, instead of booting a sandbox whose policy-approved CONNECTs all time out. Signed-off-by: Philippe Martin <phmartin@redhat.com>
2 participants
Summary
Gives the MicroVM compute driver the same operator-owned corporate forward-proxy surface the Podman and Kubernetes drivers already have, so VM sandboxes on proxy-only corporate networks can reach the destinations policy allows. The supervisor-side machinery from #1792 is reused unchanged; what this adds is the per-driver half — configuration, validation, a driver→guest argument channel, and credential/CA delivery into the guest.
Related Issue
Closes #3088
Changes
The argv channel (the one genuinely new mechanism). Podman and Kubernetes build the supervisor's command line in Rust. The VM guest init script runs as PID 1 and execs a fixed argv, and libkrun's
krun_set_execreceives an empty argument array, so there was no channel for driver-owned supervisor arguments. The supervisor's proxy flags deliberately have noenv =fallback, andbuild_guest_environmentmerges user-supplied environment, so the guest environment is not a safe transport either./opt/openshell/supervisor-argsinto the overlay upperdir on every launch; the guest init script reads it verbatim, one argument per line (no word splitting, globbing, or expansion), and appends it to every supervisorexec.init.dmanifest, which solves the same trust problem for guest init drop-ins.runtime.rsis untouched.Configuration.
[openshell.drivers.vm]acceptshttps_proxy,no_proxy,proxy_auth_file,proxy_auth_allow_insecure,proxy_connect_by_hostname, andproxy_ca_bundle, with matching--https-proxy…--proxy-ca-bundleflags andOPENSHELL_VM_*environment variables on the driver binary (needed independently, since the driver also runs as an external gRPC driver). The booleans travel as explicit values rather than presence flags so an operator's explicitfalsesurvives the gateway→driver hop and still trips the pairing checks. The keys are not inheritable from[openshell.gateway]and are rejected bytemplate.driver_config.vm.Shared validation, fail-closed. Added
openshell_core::driver_utils::validate_upstream_proxy_settingsholding the pairing rules the Podman driver established, called by both the gateway and the driver — without the gateway-side check an invalid table surfaces as an opaque driver-readiness timeout instead of an error naming the key. Podman and Kubernetes are deliberately left on their own copies to keep this PR scoped.Credential and CA delivery. A microVM has no bind mounts or container secrets, so both are staged into the per-sandbox overlay the way the gateway JWT already is — credential root-only at
0600, CA at0644, both at fixed/opt/openshellpaths, both rewritten every launch (so removing a setting clears material a previous launch staged into a preserved overlay), and both removed with the sandbox state directory.Docs.
docs/reference/gateway-config.mdx(MicroVM section),docs/reference/sandbox-compute-drivers.mdx,crates/openshell-driver-vm/README.md,architecture/sandbox.md, and a VM corporate-proxy troubleshooting section in thedebug-openshell-clusterskill.Security considerations for review
overlay.ext4on the gateway host, which differs from the Podman secret model. It is taken because the per-sandbox gateway JWT — a bearer token for the gateway — already travels this exact path, so the exposure is not new; it is documented inarchitecture/sandbox.mdand the config reference rather than left implicit. Alternatives considered and rejected as disproportionate: copying to tmpfs via an init drop-in and deleting from the overlay, or a new driver→guest vsock secret channel.host.openshell.internal(gvproxy NATs192.168.127.254to the host's127.0.0.1). Documented, and exercised by the e2e. The guest→gateway callback is unaffected and never traverses the proxy.Testing
mise run pre-commitpassesUnit — 29 new tests:
openshell-core: the shared validator (unsupported scheme, inline credentials, present-but-empty values, each auxiliary key without a proxy URL, the cleartext acknowledgement including thehttps://exemption, CA bundle valid with a plain-HTTP intercepting proxy) and the three new guest paths in the control-root test.openshell-driver-vm: argv builder emits guest paths and never the operator's host paths;Some(false)is not passed as a presence flag; the argument file renders empty when unconfigured and one-argument-per-line otherwise, preserving values containing spaces; newline/NUL values rejected; credential and CA validated with the supervisor's own validators and errors never echo file contents;build_guest_environmentemits no proxy setting and no user-supplied environment can forge an--upstream-*argument;template.driver_config.vmrejects all six keys;Debugredacts the proxy URL and credential path.openshell-server:spawnforwards the settings as driver argv and omits unset ones; invalid config is rejected before the driver starts, naming the key.E2E — new
e2e/rust/tests/vm_corporate_proxy.rsin thee2e:vmlane, the VM counterpart ofpodman_corporate_proxy.rs. Fixtures run as host processes (no container-runtime dependency for this lane) and are reached from the guest throughhost.openshell.internal, which is also what proves the documented host-loopback reachability rule. Three cases:CONNECT 192.168.127.254:<port> auth=ok, proving both the validated-IP CONNECT target and that the overlay-staged credential reached the supervisor (the proxy answers 407 without it). The policy-denied destination never appears in the proxy log at all, and a port-qualifiedno_proxydestination is reached while never touching the proxy.https://proxy trusted viaproxy_ca_bundle, with the CA minted by the fixture and recovered from its output.no_proxywithouthttps_proxystops the gateway from serving, with the offending keys named in its log.Verification performed on this branch:
mise run pre-commitcargo clippy --all-targets -- -D warnings(core, driver-vm, server, e2e)mise run test:rust(full workspace incl.openshell-server)mise run e2e:vmsmoke1,host_gateway_alias4,vm_gateway_start1,vm_corporate_proxy3smokepassing is a deliberate regression signal: every sandbox now runs the argument reader before the supervisor exec, so a mistake in the reader or in the empty-array expansion underset -uwould break ordinary VM boot, not just the proxy path.Two notes for anyone running the VM lane locally on macOS, both discovered here and neither a product bug:
brew install e2fsprogsis required — the driver shells out tomke2fson the host to build the guest rootfs. It is keg-only; the driver probes/opt/homebrew/opt/e2fsprogs/sbindirectly.TooLargepanic. E2E workloads of any size should useSandboxGuard::create(delivered viaExecSandbox) rather thancreate_keep_with_args.Checklist