Skip to content

bug: distinguish intentional signal stops from runtime restarts #3083

Description

@elezar

User Story

As an OpenShell operator, I want sandbox status to distinguish an intentional shutdown from a runtime interruption, so that stopped sandboxes are not presented as having restarted unexpectedly and real runtime restarts remain recoverable.

Problem Statement

The Docker and Podman drivers currently classify exits 137 (SIGKILL) and 143 (SIGTERM) as ContainerRuntimeRestart. Those codes establish only that a process was terminated by a signal; they do not identify the sender or intent. An explicit gateway stop that forwards SIGTERM therefore produces the same condition as a Podman/Docker machine or daemon restart.

The durable Stopping phase now prevents that ambiguity from promoting an in-flight explicit stop to Error, but a delayed watcher snapshot can still arrive after Stopped is persisted and replace the user-visible status reason with ContainerRuntimeRestart.

Impact / Why This Matters

Operators can see a sandbox in Stopped phase with a contradictory runtime-restart condition after a normal stop. More broadly, treating all 137/143 exits as runtime restarts conflates graceful stop, forced timeout kill, external intervention, and genuine runtime interruption. The current workaround is to infer intent from lifecycle phase, which protects the immediate flow but does not make the driver status semantically precise.

Acceptance Criteria

  • An explicit gateway stop remains Stopped when a late Docker or Podman signal-exit snapshot arrives, and its terminal status continues to report the intentional stop.
  • A signal termination without explicit stop intent remains distinguishable from a confirmed runtime interruption.
  • Gateway restart recovery continues to recover sandboxes interrupted by a real Docker or Podman runtime/machine restart.
  • OOM termination and ordinary application exits keep their existing distinct behavior.
  • Regression coverage covers Docker and Podman for explicit SIGTERM stop, forced SIGKILL timeout, delayed watcher delivery, OOM, and runtime/machine restart.

Reproduction Steps

  1. Start a Docker- or Podman-backed sandbox.
  2. Stop it through the gateway so the supervisor forwards SIGTERM to its workload.
  3. Observe the driver report exit 143 as ContainerRuntimeRestart.
  4. Deliver that watcher snapshot after the gateway has persisted Stopped.
  5. Observe the sandbox phase remain Stopped while its condition reason no longer reflects the intentional stop.

Environment

Agent Investigation

ContainerRuntimeRestart is currently a heuristic for exit 137/143 in both Docker and Podman. The exit status has no provenance, so operation intent and independently observed runtime state must be considered separately.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions