Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions .agents/skills/debug-openshell-cluster/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ Common findings:

- `No active gateway`: register one with `openshell gateway add <endpoint>`.
- Connection refused: gateway process is not running, service exposure is wrong, or a port-forward/proxy is not active.
- TLS/certificate errors: the endpoint scheme or trust chain is wrong, a local mTLS bundle does not match the gateway CA, or TLS termination does not match the gateway listener.
- TLS/certificate errors: the endpoint scheme or trust chain is wrong, a CLI mTLS bundle does not match the gateway CA, a sandbox is missing the gateway CA, or TLS termination does not match the gateway listener. Sandboxes should not contain a TLS client certificate or private key.
- `Unauthenticated` from an edge or OIDC gateway: refresh stored credentials with `openshell gateway login [name]`, then retry. Use `gateway logout` only when intentionally clearing local credentials.
- A direct development endpoint with a private or self-signed certificate can be isolated with `--gateway-endpoint <url> --gateway-insecure`; do not persist or recommend insecure verification for shared gateways.

Expand Down Expand Up @@ -375,7 +375,9 @@ Less commonly, `UnknownCA` can occur if the gateway's client-verification CA
is misconfigured. The default `clientCaFromServerTlsSecret=true` is correct
for all configurations — the internal server certificate is always signed by
the chart CA (the same CA that signs the client cert), so its `ca.crt` is
the right trust anchor. Only override this if you intentionally mount a
the right trust anchor. Sandbox pods project only `ca.crt` from the copied
Secret; `tls.crt` and `tls.key` are reserved for user clients and must not be
visible in a sandbox. Only override this if you intentionally mount a
separate client CA via `server.tls.clientCaSecretName`. Verify the mounted
client CA matches the CA that signed the client certificate:

Expand Down
7 changes: 4 additions & 3 deletions .agents/skills/helm-dev-environment/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ mise run helm:skaffold:run
mise run helm:skaffold:run:sidecar
```

**Supervisor sidecar topology with TLS/mTLS enabled** (build once and leave running):
**Supervisor sidecar topology with gateway TLS and CLI mTLS enabled** (build once and leave running):
```bash
mise run helm:skaffold:run:sidecar-mtls
```
Expand All @@ -85,8 +85,9 @@ Binary-aware policy mode runs that sidecar as UID 0 with `SYS_PTRACE` and
must be at least `1000` and distinct from the workload UID. The
sidecar-mTLS profile reuses `ci/values-sidecar.yaml` and restores
`server.disableTls=false` inline for Skaffold. The `pkiInitJob` hook (a pre-install
Job that runs `openshell-gateway generate-certs`) generates mTLS secrets on first
install. The default Skaffold values export gateway and Kubernetes-driver traces to
Job that runs `openshell-gateway generate-certs`) generates the gateway and CLI
TLS secrets on first install. Sandbox pods project only `ca.crt`; they use bearer
tokens, not the CLI client certificate, for gateway authentication. The default Skaffold values export gateway and Kubernetes-driver traces to
the collector service installed by `helm:k3s:create`. Envoy Gateway opt-in; see the
Optional Add-ons section below.

Expand Down
20 changes: 10 additions & 10 deletions architecture/gateway.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,12 +41,12 @@ finalized supervisor session disconnects.
## Protocol and Auth

The gateway listens on one service port and multiplexes gRPC and HTTP traffic.
The default local single-user deployment mode is mTLS user authentication:
clients present a certificate signed by the local deployment CA, and the
gateway maps the verified certificate subject to a user principal. Kubernetes
deployments use mTLS for transport only and require OIDC or a trusted access
proxy for user authentication unless the explicit unsafe local-development
`allow_unauthenticated_users` switch is enabled.
When a client CA is configured without OIDC, mTLS user authentication defaults
on independently of the compute driver: clients present a certificate signed
by the deployment CA, and the gateway maps the verified certificate subject to
a user principal. Sandboxes do not receive that client certificate. They
authenticate the gateway with the CA and authenticate their own RPCs with
gateway-minted bearer tokens.
When that service port is bound to loopback, the listener can also accept
plaintext HTTP on the same port for sandbox service subdomains only. That local
browser path is enabled by default and disabled with
Expand Down Expand Up @@ -188,7 +188,7 @@ Supported auth modes:

| Mode | Use |
|---|---|
| mTLS user auth | Local single-user Docker, Podman, and VM gateway access. |
| mTLS user auth | Gateway user access with verified client certificates. |
| Plaintext | Local development or a trusted reverse proxy boundary. |
| Unauthenticated local users | Trusted Kubernetes dev or fully trusted proxy deployments only. |
| Cloudflare JWT | Edge-authenticated deployments where Cloudflare Access supplies identity. |
Expand Down Expand Up @@ -217,8 +217,8 @@ the capability query's admin authorization check. The CLI combines the health
and capability results so a reachable gateway with an expired or rejected
token is reported as connected but unauthenticated.

Sandbox supervisor RPCs authenticate with explicit sandbox credentials; mTLS
does not grant sandbox identity. Kubernetes deployments use the
Sandbox supervisor RPCs authenticate with explicit sandbox credentials; TLS
authenticates the gateway and does not grant sandbox identity. Kubernetes deployments use the
gateway-minted JWT bootstrap path: the supervisor starts with a projected
ServiceAccount token, exchanges it for a gateway-minted sandbox JWT, and uses
that JWT on subsequent gateway RPCs.
Expand Down Expand Up @@ -700,7 +700,7 @@ aliases, network names, and the sandbox JWT issuer.

`[openshell.gateway]` carries a small set of values (`sandbox_namespace`,
`default_image`,
`supervisor_image`, `guest_tls_ca/cert/key`, `client_tls_secret_name`,
`supervisor_image`, `guest_tls_ca`, `client_tls_secret_name`,
`host_gateway_ip`, `enable_user_namespaces`) that are inherited into each
driver's `[openshell.drivers.<name>]` table when the driver-specific table
does not override them. The allowlist is per-driver so a gateway-wide
Expand Down
2 changes: 1 addition & 1 deletion crates/openshell-bootstrap/src/pki.rs
Original file line number Diff line number Diff line change
Expand Up @@ -89,7 +89,7 @@ pub fn generate_pki(extra_sans: &[String]) -> Result<PkiBundle> {
.into_diagnostic()
.wrap_err("failed to sign server certificate")?;

// --- Client cert (shared by CLI and sandbox pods) ---
// --- User client cert (CLI only; sandboxes use bearer identity) ---
let client_key = KeyPair::generate()
.into_diagnostic()
.wrap_err("failed to generate client key")?;
Expand Down
8 changes: 0 additions & 8 deletions crates/openshell-core/src/container_paths.rs
Original file line number Diff line number Diff line change
Expand Up @@ -44,8 +44,6 @@ pub const SUPERVISOR_CONTAINER_DIR: &str = "/opt/openshell/bin";
pub const SUPERVISOR_CONTAINER_BINARY: &str = "/opt/openshell/bin/openshell-sandbox";
pub const TLS_CLIENT_DIR: &str = "/etc/openshell/tls/client";
pub const TLS_CA_MOUNT_PATH: &str = "/etc/openshell/tls/client/ca.crt";
pub const TLS_CERT_MOUNT_PATH: &str = "/etc/openshell/tls/client/tls.crt";
pub const TLS_KEY_MOUNT_PATH: &str = "/etc/openshell/tls/client/tls.key";
pub const SANDBOX_TOKEN_MOUNT_PATH: &str = "/etc/openshell/auth/sandbox.jwt";
pub const UPSTREAM_PROXY_AUTH_MOUNT_PATH: &str = "/etc/openshell/auth/upstream-proxy";
pub const CONTAINER_POLICY_PATH: &str = "/etc/openshell/policy.yaml";
Expand All @@ -60,8 +58,6 @@ pub const SUPERVISOR_CA_CERT_PATH: &str = "/etc/openshell-tls/openshell-ca.pem";
pub const SUPERVISOR_CA_BUNDLE_PATH: &str = "/etc/openshell-tls/ca-bundle.pem";

pub const VM_GUEST_TLS_CA_PATH: &str = "/opt/openshell/tls/ca.crt";
pub const VM_GUEST_TLS_CERT_PATH: &str = "/opt/openshell/tls/tls.crt";
pub const VM_GUEST_TLS_KEY_PATH: &str = "/opt/openshell/tls/tls.key";
pub const VM_GUEST_SANDBOX_TOKEN_PATH: &str = "/opt/openshell/auth/sandbox.jwt";
pub const VM_GUEST_INIT_DROPIN_DIR: &str = "/opt/openshell/init.d";
pub const VM_GUEST_INIT_DROPIN_MANIFEST: &str = "/opt/openshell/init.d.manifest";
Expand All @@ -84,8 +80,6 @@ mod tests {
SUPERVISOR_CONTAINER_BINARY,
TLS_CLIENT_DIR,
TLS_CA_MOUNT_PATH,
TLS_CERT_MOUNT_PATH,
TLS_KEY_MOUNT_PATH,
SANDBOX_TOKEN_MOUNT_PATH,
UPSTREAM_PROXY_AUTH_MOUNT_PATH,
CONTAINER_POLICY_PATH,
Expand All @@ -98,8 +92,6 @@ mod tests {
SUPERVISOR_CA_CERT_PATH,
SUPERVISOR_CA_BUNDLE_PATH,
VM_GUEST_TLS_CA_PATH,
VM_GUEST_TLS_CERT_PATH,
VM_GUEST_TLS_KEY_PATH,
VM_GUEST_SANDBOX_TOKEN_PATH,
VM_GUEST_INIT_DROPIN_DIR,
VM_GUEST_INIT_DROPIN_MANIFEST,
Expand Down
17 changes: 5 additions & 12 deletions crates/openshell-core/src/driver_utils.rs
Original file line number Diff line number Diff line change
Expand Up @@ -90,22 +90,15 @@ pub const SUPERVISOR_CONTAINER_BINARY: &str = "/opt/openshell/bin/openshell-sand
// ---------------------------------------------------------------------------
// In-container mount paths for guest TLS materials and the sandbox token.
//
// All container-based drivers (Docker, Podman, Kubernetes) mount the gateway's
// mTLS client credentials at these fixed paths inside every sandbox container.
// The supervisor reads these paths on startup to establish its gRPC-over-mTLS
// connection back to the gateway. The paths must remain stable across driver
// versions since the supervisor binary is built and packaged separately.
// Container-based drivers mount the gateway CA at this fixed path inside every
// sandbox container. The supervisor reads it on startup to authenticate the
// gateway TLS endpoint. Sandbox identity is provided separately by a bearer
// token.
// ---------------------------------------------------------------------------

/// Container-side mount path for the guest mTLS CA certificate.
/// Container-side mount path for the gateway CA certificate.
pub const TLS_CA_MOUNT_PATH: &str = "/etc/openshell/tls/client/ca.crt";

/// Container-side mount path for the guest mTLS client certificate.
pub const TLS_CERT_MOUNT_PATH: &str = "/etc/openshell/tls/client/tls.crt";

/// Container-side mount path for the guest mTLS client private key.
pub const TLS_KEY_MOUNT_PATH: &str = "/etc/openshell/tls/client/tls.key";

/// Container-side mount path for the per-sandbox JWT token.
pub const SANDBOX_TOKEN_MOUNT_PATH: &str = "/etc/openshell/auth/sandbox.jwt";

Expand Down
29 changes: 6 additions & 23 deletions crates/openshell-core/src/grpc_client.rs
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ use openshell_extension_core::{BearerTokenSlot, ExtensionCredentialStore};
use tonic::Status;
use tonic::metadata::AsciiMetadataValue;
use tonic::service::interceptor::InterceptedService;
use tonic::transport::{Certificate, Channel, ClientTlsConfig, Endpoint, Identity};
use tonic::transport::{Certificate, Channel, ClientTlsConfig, Endpoint};
use tracing::{debug, info, warn};

/// Channel type after the [`AuthInterceptor`] is applied. Aliased so the
Expand Down Expand Up @@ -125,10 +125,9 @@ impl tonic::service::Interceptor for AuthInterceptor {

/// Build the plain (un-intercepted) gRPC channel.
///
/// When the endpoint uses `https://`, mTLS is configured using these env vars:
/// When the endpoint uses `https://`, server-authenticated TLS is configured
/// using this env var:
/// - `OPENSHELL_TLS_CA` -- path to the CA certificate
/// - `OPENSHELL_TLS_CERT` -- path to the client certificate
/// - `OPENSHELL_TLS_KEY` -- path to the client private key
///
/// When the endpoint uses `http://`, a plaintext connection is used (for
/// deployments where TLS is disabled, e.g. behind a Cloudflare Tunnel).
Expand All @@ -147,34 +146,20 @@ async fn build_plain_channel(endpoint: &str) -> Result<Channel> {

let tls_enabled = endpoint.starts_with("https://");

// TODO: TLS certs are loaded once here and never re-read. The gateway
// TODO: The TLS CA is loaded once here and never re-read. The gateway
// server side supports hot-reload (ArcSwap + notify in tls.rs). The
// supervisor should do the same so that cert-manager rotations take
// effect without restarting the sandbox.
if tls_enabled {
let ca_path = std::env::var(sandbox_env::TLS_CA)
.into_diagnostic()
.wrap_err("OPENSHELL_TLS_CA is required")?;
let cert_path = std::env::var(sandbox_env::TLS_CERT)
.into_diagnostic()
.wrap_err("OPENSHELL_TLS_CERT is required")?;
let key_path = std::env::var(sandbox_env::TLS_KEY)
.into_diagnostic()
.wrap_err("OPENSHELL_TLS_KEY is required")?;

let ca_pem = std::fs::read(&ca_path)
.into_diagnostic()
.wrap_err_with(|| format!("failed to read CA cert from {ca_path}"))?;
let cert_pem = std::fs::read(&cert_path)
.into_diagnostic()
.wrap_err_with(|| format!("failed to read client cert from {cert_path}"))?;
let key_pem = std::fs::read(&key_path)
.into_diagnostic()
.wrap_err_with(|| format!("failed to read client key from {key_path}"))?;

// Trust only the configured CA — this is the chart's internal CA
// that signs both the gateway's internal server certificate and
// this client's identity certificate. The gateway uses SNI-based
// that signs the gateway's internal server certificate. The gateway uses SNI-based
// certificate selection to present this internal cert to supervisor
// connections, so no public root trust is needed here.
//
Expand All @@ -183,9 +168,7 @@ async fn build_plain_channel(endpoint: &str) -> Result<Channel> {
// (Docker/Podman drivers), and broadening the trust store would let
// an attacker who controls the image + DNS present a publicly valid
// certificate and intercept the supervisor→gateway TLS connection.
let mut tls_config = ClientTlsConfig::new()
.ca_certificate(Certificate::from_pem(ca_pem))
.identity(Identity::from_pem(cert_pem, key_pem));
let mut tls_config = ClientTlsConfig::new().ca_certificate(Certificate::from_pem(ca_pem));
if let Ok(server_name) = std::env::var(sandbox_env::GATEWAY_TLS_SERVER_NAME)
&& !server_name.is_empty()
{
Expand Down
9 changes: 5 additions & 4 deletions crates/openshell-driver-docker/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -178,12 +178,13 @@ assigned address that containers should use for callbacks; package-managed
macOS gateways should leave it unset.

For HTTPS endpoints, the server certificate must include the endpoint host as a
subject alternative name. Docker sandboxes also need the client TLS bundle
mounted into the container and exposed with:
subject alternative name. Docker sandboxes receive only the gateway CA through:

- `OPENSHELL_TLS_CA`
- `OPENSHELL_TLS_CERT`
- `OPENSHELL_TLS_KEY`

The supervisor authenticates the gateway with this CA and authenticates its
RPCs with a sandbox-scoped bearer token. Client certificates and private keys
are not mounted into sandbox containers.

HTTP endpoints reject TLS material because the supervisor would not use it.

Expand Down
58 changes: 14 additions & 44 deletions crates/openshell-driver-docker/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -80,8 +80,6 @@ const WATCH_POLL_MAX_BACKOFF: Duration = Duration::from_secs(30);

const SUPERVISOR_MOUNT_PATH: &str = openshell_core::driver_utils::SUPERVISOR_CONTAINER_BINARY;
const TLS_CA_MOUNT_PATH: &str = openshell_core::driver_utils::TLS_CA_MOUNT_PATH;
const TLS_CERT_MOUNT_PATH: &str = openshell_core::driver_utils::TLS_CERT_MOUNT_PATH;
const TLS_KEY_MOUNT_PATH: &str = openshell_core::driver_utils::TLS_KEY_MOUNT_PATH;
const SANDBOX_TOKEN_MOUNT_PATH: &str = openshell_core::driver_utils::SANDBOX_TOKEN_MOUNT_PATH;
const SUPERVISOR_PATH: &str = "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin";
const HOST_OPENSHELL_INTERNAL: &str = "host.openshell.internal";
Expand Down Expand Up @@ -139,13 +137,15 @@ pub struct DockerComputeConfig {
/// the full resolution order.
pub supervisor_image: Option<String>,

/// Host-side CA certificate for Docker sandbox mTLS.
/// Host-side CA certificate for sandbox-to-gateway TLS.
pub guest_tls_ca: Option<PathBuf>,

/// Host-side client certificate for Docker sandbox mTLS.
/// Deprecated. Sandboxes authenticate with bearer tokens and must not
/// receive a user client certificate.
pub guest_tls_cert: Option<PathBuf>,

/// Host-side private key for Docker sandbox mTLS.
/// Deprecated. Sandboxes authenticate with bearer tokens and must not
/// receive a user client private key.
pub guest_tls_key: Option<PathBuf>,

/// Docker bridge network that sandbox containers join.
Expand Down Expand Up @@ -193,8 +193,6 @@ impl Default for DockerComputeConfig {
#[derive(Debug, Clone, PartialEq, Eq)]
pub(crate) struct DockerGuestTlsPaths {
pub(crate) ca: PathBuf,
pub(crate) cert: PathBuf,
pub(crate) key: PathBuf,
}

#[derive(Debug, Clone)]
Expand Down Expand Up @@ -2667,12 +2665,6 @@ fn build_binds(
)];
if let Some(tls) = &config.guest_tls {
binds.push(format!("{}:{}:ro,z", tls.ca.display(), TLS_CA_MOUNT_PATH));
binds.push(format!(
"{}:{}:ro,z",
tls.cert.display(),
TLS_CERT_MOUNT_PATH
));
binds.push(format!("{}:{}:ro,z", tls.key.display(), TLS_KEY_MOUNT_PATH));
}
if sandbox
.spec
Expand Down Expand Up @@ -2858,14 +2850,6 @@ fn build_environment_for_oci_user(
openshell_core::sandbox_env::TLS_CA.to_string(),
TLS_CA_MOUNT_PATH.to_string(),
);
environment.insert(
openshell_core::sandbox_env::TLS_CERT.to_string(),
TLS_CERT_MOUNT_PATH.to_string(),
);
environment.insert(
openshell_core::sandbox_env::TLS_KEY.to_string(),
TLS_KEY_MOUNT_PATH.to_string(),
);
}

environment.remove(openshell_core::sandbox_env::SANDBOX_TOKEN);
Expand Down Expand Up @@ -4035,8 +4019,6 @@ fn canonicalize_existing_file(path: &Path, description: &str) -> CoreResult<Path

fn docker_guest_tls_configured(docker_config: &DockerComputeConfig) -> bool {
docker_config.guest_tls_ca.is_some()
&& docker_config.guest_tls_cert.is_some()
&& docker_config.guest_tls_key.is_some()
}

pub(crate) fn docker_guest_tls_paths(
Expand All @@ -4046,24 +4028,25 @@ pub(crate) fn docker_guest_tls_paths(
|| docker_config.guest_tls_cert.is_some()
|| docker_config.guest_tls_key.is_some();

if docker_config.guest_tls_cert.is_some() || docker_config.guest_tls_key.is_some() {
return Err(Error::config(
"guest_tls_cert and guest_tls_key are no longer supported; sandboxes authenticate to the gateway with bearer tokens",
));
}

if !docker_config.grpc_endpoint.starts_with("https://") {
if tls_flags_provided {
return Err(Error::config(format!(
"guest_tls_ca/guest_tls_cert/guest_tls_key were provided but grpc_endpoint is '{}'; TLS materials require an https:// endpoint",
"guest_tls_ca was provided but grpc_endpoint is '{}'; TLS materials require an https:// endpoint",
docker_config.grpc_endpoint,
)));
}
return Ok(None);
}

let provided = [
docker_config.guest_tls_ca.as_ref(),
docker_config.guest_tls_cert.as_ref(),
docker_config.guest_tls_key.as_ref(),
];
if provided.iter().all(Option::is_none) {
if docker_config.guest_tls_ca.is_none() {
return Err(Error::config(
"docker compute driver requires guest_tls_ca, guest_tls_cert, and guest_tls_key when grpc_endpoint uses https://",
"docker compute driver requires guest_tls_ca when grpc_endpoint uses https://",
));
}

Expand All @@ -4072,21 +4055,8 @@ pub(crate) fn docker_guest_tls_paths(
"guest_tls_ca is required when Docker sandbox TLS materials are configured",
));
};
let Some(cert) = docker_config.guest_tls_cert.clone() else {
return Err(Error::config(
"guest_tls_cert is required when Docker sandbox TLS materials are configured",
));
};
let Some(key) = docker_config.guest_tls_key.clone() else {
return Err(Error::config(
"guest_tls_key is required when Docker sandbox TLS materials are configured",
));
};

Ok(Some(DockerGuestTlsPaths {
ca: canonicalize_existing_file(&ca, "docker TLS CA certificate")?,
cert: canonicalize_existing_file(&cert, "docker TLS client certificate")?,
key: canonicalize_existing_file(&key, "docker TLS client private key")?,
}))
}

Expand Down
Loading
Loading