Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions architecture/gateway.md
Original file line number Diff line number Diff line change
Expand Up @@ -509,6 +509,11 @@ validate its config. The effective sandbox config contains only the registered
services required by that policy; supervisors invoke those services directly on
the request path.

The effective sandbox config also carries the supervisor-wide HTTP response
whole-body timeout. The gateway reads this static value from
`[openshell.supervisor]`, defaults it to 120 seconds, and distributes it as
milliseconds. A zero value from an older gateway maps to the same default.

Provider credential expiry is enforced during gateway-to-sandbox credential
resolution and again by the sandbox placeholder resolver. This keeps expired
credentials from resolving even when a running sandbox still has retained
Expand Down
10 changes: 10 additions & 0 deletions architecture/sandbox.md
Original file line number Diff line number Diff line change
Expand Up @@ -185,6 +185,16 @@ middleware registry validates implementation-owned config. The generic
registry and chain runner live in `openshell-supervisor-middleware`; first-party
implementations live in `openshell-supervisor-middleware-builtins`.

The same selected chain can inspect the matching final HTTP response before it
returns to the workload. Response stages select header-only, whole-body, or
streaming mode independently. The relay preserves upstream framing for a
header-only chain and owns normalized downstream framing only when body bytes
can change. Whole-body stages delay commitment and share one non-resetting,
supervisor-wide accumulation deadline. Body stages receive a final body result
and then one trailer exchange; trailer mutations can only change or remove
existing, non-protected names. Intentional blocks return the canonical 403
before commitment and abort delivery without injected bytes after commitment.

The supervisor installs policy and middleware registry changes as one runtime
generation and preserves the last-known-good generation if preparation fails.
Policy-only updates reuse the connected registry, so an external middleware
Expand Down
16 changes: 16 additions & 0 deletions crates/openshell-core/src/grpc_client.rs
Original file line number Diff line number Diff line change
Expand Up @@ -943,6 +943,8 @@ pub struct SettingsPollResult {
pub policy_validation_failure_mode: crate::PolicyValidationFailureMode,
/// Whether the gateway can mint authenticated extension credentials.
pub extension_authentication_enabled: bool,
/// Supervisor-wide response whole-body accumulation timeout.
pub http_response_whole_body_timeout_ms: u64,
}

fn settings_poll_result(inner: crate::proto::GetSandboxConfigResponse) -> SettingsPollResult {
Expand All @@ -963,6 +965,11 @@ fn settings_poll_result(inner: crate::proto::GetSandboxConfigResponse) -> Settin
.parse()
.unwrap_or_default(),
extension_authentication_enabled: inner.extension_authentication_enabled,
http_response_whole_body_timeout_ms: if inner.http_response_whole_body_timeout_ms == 0 {
crate::DEFAULT_HTTP_RESPONSE_WHOLE_BODY_TIMEOUT_MS
} else {
inner.http_response_whole_body_timeout_ms
},
}
}

Expand All @@ -984,6 +991,15 @@ mod settings_poll_tests {
);
}

#[test]
fn zero_whole_body_timeout_uses_compatibility_default() {
let result = settings_poll_result(GetSandboxConfigResponse::default());
assert_eq!(
result.http_response_whole_body_timeout_ms,
crate::DEFAULT_HTTP_RESPONSE_WHOLE_BODY_TIMEOUT_MS
);
}

#[test]
fn unknown_validation_failure_mode_fails_closed() {
let result = settings_poll_result(GetSandboxConfigResponse {
Expand Down
3 changes: 3 additions & 0 deletions crates/openshell-core/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,9 @@ pub const VERSION: &str = match option_env!("OPENSHELL_GIT_VERSION") {
None => env!("CARGO_PKG_VERSION"),
};

/// Default wall-clock bound for HTTP response whole-body middleware buffering.
pub const DEFAULT_HTTP_RESPONSE_WHOLE_BODY_TIMEOUT_MS: u64 = 120_000;

#[cfg(test)]
#[path = "../build_version.rs"]
mod build_version;
Expand Down
12 changes: 12 additions & 0 deletions crates/openshell-sandbox/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -2380,6 +2380,9 @@ async fn load_policy(
openshell_core::grpc_client::fetch_settings_snapshot(endpoint, id)
})
.await?;
openshell_supervisor_network::set_http_response_whole_body_timeout(Duration::from_millis(
snapshot.http_response_whole_body_timeout_ms,
));

let mut proto_policy = if let Some(p) = snapshot.policy.clone() {
p
Expand Down Expand Up @@ -3781,6 +3784,9 @@ async fn run_policy_poll_loop_with_client<C: PolicyGatewayClient>(
// reconciled below instead of being recorded as already applied.
match client.poll_settings(&ctx.sandbox_id).await {
Ok(result) => {
openshell_supervisor_network::set_http_response_whole_body_timeout(
Duration::from_millis(result.http_response_whole_body_timeout_ms),
);
let _ = ctx.workspace_tx.send(client.workspace());
match initial_poll_disposition(&ctx.loaded_policy_origin, &result) {
InitialPollDisposition::Acknowledge(candidate) => {
Expand Down Expand Up @@ -3867,6 +3873,10 @@ async fn run_policy_poll_loop_with_client<C: PolicyGatewayClient>(
}
};

openshell_supervisor_network::set_http_response_whole_body_timeout(Duration::from_millis(
result.http_response_whole_body_timeout_ms,
));

// Reuse installed per-service credentials, rotating only when one is
// missing or due. Rotation happens on the existing gateway channel and
// updates slots in place, so it is independent of config revision and
Expand Down Expand Up @@ -4963,6 +4973,8 @@ network_policies:
workspace: String::new(),
policy_validation_failure_mode: PolicyValidationFailureMode::default(),
extension_authentication_enabled: false,
http_response_whole_body_timeout_ms:
openshell_core::DEFAULT_HTTP_RESPONSE_WHOLE_BODY_TIMEOUT_MS,
}
}

Expand Down
73 changes: 73 additions & 0 deletions crates/openshell-server/src/config_file.rs
Original file line number Diff line number Diff line change
Expand Up @@ -214,12 +214,40 @@ pub struct OtlpConfig {
#[derive(Debug, Clone, Default, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct SupervisorFileSection {
/// Wall-clock limit for accumulating and processing a response through
/// whole-body middleware. Accepts a positive integer followed by `ms`,
/// `s`, or `m`.
#[serde(default)]
pub http_response_whole_body_timeout: Option<String>,

/// Statically registered supervisor middleware services. Registration is
/// operator-owned and changes require a gateway restart.
#[serde(default)]
pub middleware: Vec<MiddlewareServiceFileConfig>,
}

impl SupervisorFileSection {
/// Resolve the configured whole-body timeout to milliseconds.
#[must_use]
pub fn http_response_whole_body_timeout_ms(&self) -> u64 {
self.http_response_whole_body_timeout
.as_deref()
.and_then(parse_positive_duration_ms)
.unwrap_or(openshell_core::DEFAULT_HTTP_RESPONSE_WHOLE_BODY_TIMEOUT_MS)
}
}

fn parse_positive_duration_ms(value: &str) -> Option<u64> {
let value = value.trim();
let (number, multiplier) = value
.strip_suffix("ms")
.map(|number| (number, 1))
.or_else(|| value.strip_suffix('s').map(|number| (number, 1_000)))
.or_else(|| value.strip_suffix('m').map(|number| (number, 60_000)))?;
let number = number.parse::<u64>().ok()?;
(number > 0).then_some(number.checked_mul(multiplier)?)
}

/// One `[[openshell.supervisor.middleware]]` supervisor middleware registration.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
Expand Down Expand Up @@ -415,6 +443,18 @@ pub fn load(path: &Path) -> Result<ConfigFile, ConfigFileError> {
message: "omit the field to use default encrypted gateway credential storage, or specify exactly one external credential driver",
});
}
if file
.openshell
.supervisor
.http_response_whole_body_timeout
.as_deref()
.is_some_and(|value| parse_positive_duration_ms(value).is_none())
{
return Err(ConfigFileError::InvalidValue {
field: "openshell.supervisor.http_response_whole_body_timeout",
message: "expected a positive integer duration ending in ms, s, or m",
});
}

Ok(file)
}
Expand Down Expand Up @@ -604,6 +644,39 @@ service_name = "openshell-gateway-dev"
assert_eq!(otlp.service_name.as_deref(), Some("openshell-gateway-dev"));
}

#[test]
fn parses_http_response_whole_body_timeout() {
let tmp = write_tmp(
r#"
[openshell.supervisor]
http_response_whole_body_timeout = "2m"
"#,
);
let file = load(tmp.path()).expect("valid supervisor timeout parses");
assert_eq!(
file.openshell
.supervisor
.http_response_whole_body_timeout_ms(),
120_000
);
}

#[test]
fn rejects_invalid_http_response_whole_body_timeout() {
for value in ["0s", "120", "later", "18446744073709551615m"] {
let tmp = write_tmp(&format!(
"[openshell.supervisor]\nhttp_response_whole_body_timeout = \"{value}\"\n"
));
let error = load(tmp.path()).expect_err("invalid timeout must be rejected");
assert!(
error
.to_string()
.contains("http_response_whole_body_timeout"),
"{error}"
);
}
}

#[test]
fn otlp_config_requires_only_endpoint() {
let toml = r#"
Expand Down
1 change: 1 addition & 0 deletions crates/openshell-server/src/grpc/policy.rs
Original file line number Diff line number Diff line change
Expand Up @@ -2524,6 +2524,7 @@ pub(super) async fn handle_get_sandbox_config(
.as_str()
.to_string(),
extension_authentication_enabled: state.sandbox_jwt_issuer.is_some(),
http_response_whole_body_timeout_ms: state.http_response_whole_body_timeout_ms,
}))
}

Expand Down
13 changes: 13 additions & 0 deletions crates/openshell-server/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -304,6 +304,9 @@ pub struct ServerState {
/// Validated built-in and operator-registered supervisor middleware.
pub middleware_registry: Arc<MiddlewareRegistry>,

/// Supervisor-wide response whole-body accumulation timeout.
pub http_response_whole_body_timeout_ms: u64,

/// OIDC JWKS cache for JWT validation. `None` when OIDC is not configured.
pub oidc_cache: Option<Arc<auth::oidc::JwksCache>>,

Expand Down Expand Up @@ -419,6 +422,8 @@ impl ServerState {
gateway_shutting_down: AtomicBool::new(false),
extension_mint_limiter: auth::extension_mint_limit::ExtensionMintLimiter::default(),
middleware_registry: Arc::new(MiddlewareRegistry::default()),
http_response_whole_body_timeout_ms:
openshell_core::DEFAULT_HTTP_RESPONSE_WHOLE_BODY_TIMEOUT_MS,
oidc_cache,
sandbox_jwt_issuer: None,
sandbox_jwt_authenticator: None,
Expand Down Expand Up @@ -658,6 +663,14 @@ pub(crate) async fn run_server(
oidc_cache,
credentials,
);
state.http_response_whole_body_timeout_ms = config_file.as_ref().map_or(
openshell_core::DEFAULT_HTTP_RESPONSE_WHOLE_BODY_TIMEOUT_MS,
|file| {
file.openshell
.supervisor
.http_response_whole_body_timeout_ms()
},
);
state.middleware_registry = middleware_registry;
state.gateway_interceptors = gateway_interceptors;
state.provider_profile_sources = provider_profile_sources;
Expand Down
33 changes: 22 additions & 11 deletions crates/openshell-supervisor-middleware/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,15 @@

pub mod headers;
mod remote;
mod response;
mod websocket;

pub use response::{
HttpResponseDiagnostics, HttpResponseFinish, HttpResponseInvocation,
HttpResponseInvocationOutcome, HttpResponseMiddlewareFailure, HttpResponsePreflightInput,
HttpResponsePreflightOutcome, HttpResponseSession, MAX_HTTP_RESPONSE_STREAM_UNIT_BYTES,
};

pub use websocket::{
WebSocketCoverage, WebSocketCoverageState, WebSocketInvocation, WebSocketInvocationOutcome,
WebSocketMessageAdmission, WebSocketMessageOutcome, WebSocketMessageType,
Expand Down Expand Up @@ -626,6 +633,16 @@ impl MiddlewareDispatch {
Self::Grpc(service) => service.open_websocket_session(receiver).await,
}
}

async fn open_http_response_pre_return(
&self,
receiver: tokio::sync::mpsc::Receiver<openshell_core::proto::HttpResponseEvent>,
) -> std::result::Result<HttpResponseResultStream, tonic::Status> {
match self {
Self::InProcess(service) => service.open_http_response_pre_return(receiver).await,
Self::Grpc(service) => service.open_http_response_pre_return(receiver).await,
}
}
}

struct MiddlewareServiceState {
Expand Down Expand Up @@ -831,6 +848,7 @@ fn validate_payload_limit(source: &str, binding: &MiddlewareBinding) -> Result<u
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
enum SupportedBinding {
HttpPreCredentials,
HttpResponsePreReturn,
WebSocketPreCredentials,
}

Expand All @@ -846,9 +864,7 @@ fn supported_binding(source: &str, binding: &MiddlewareBinding) -> Result<Suppor
(
Some(SupervisorMiddlewareOperation::HttpResponse),
Some(SupervisorMiddlewarePhase::PreReturn),
) => Err(miette!(
"{source} advertises HTTP_RESPONSE/PRE_RETURN, which is not yet supported"
)),
) => Ok(SupportedBinding::HttpResponsePreReturn),
(
Some(SupervisorMiddlewareOperation::WebsocketMessage),
Some(SupervisorMiddlewarePhase::PreCredentials),
Expand Down Expand Up @@ -3686,7 +3702,7 @@ mod tests {
}

#[test]
fn manifest_rejects_http_response_pre_return_binding_until_dispatch_is_available() {
fn manifest_accepts_http_response_pre_return_binding_when_dispatch_is_available() {
let registration = external_registration(4096);
let manifest = MiddlewareManifest {
name: "example/response".into(),
Expand All @@ -3700,13 +3716,8 @@ mod tests {
expected_audience: String::new(),
};

let error = validate_external_manifest(&registration, &manifest, 4096, false)
.expect_err("HTTP response pre-return binding must remain unavailable");
assert!(
error
.to_string()
.contains("HTTP_RESPONSE/PRE_RETURN, which is not yet supported")
);
validate_external_manifest(&registration, &manifest, 4096, false)
.expect("HTTP response pre-return binding is supported");
}

#[test]
Expand Down
8 changes: 8 additions & 0 deletions crates/openshell-supervisor-middleware/src/remote.rs
Original file line number Diff line number Diff line change
Expand Up @@ -103,6 +103,14 @@ impl GrpcMiddlewareService {
) -> std::result::Result<WebSocketResponseStream, Status> {
self.service.open_websocket_session(receiver).await
}

/// Open a remote HTTP response pre-return stream through the gRPC adapter.
pub async fn open_http_response_pre_return(
&self,
receiver: tokio::sync::mpsc::Receiver<HttpResponseEvent>,
) -> std::result::Result<HttpResponseResultStream, Status> {
self.service.open_http_response_pre_return(receiver).await
}
}

#[derive(Clone)]
Expand Down
Loading
Loading