test(install): add Docker lifecycle sandbox - #2252
Conversation
Co-authored-by: Codex <codex@openai.com>
![graphify-labs[bot]](https://avatars.githubusercontent.com/in/4356270?s=60&v=4){kind=small}
There was a problem hiding this comment.
Looks safe to merge — no coupling regressions and no blocking issues, checked against the code graph (not a self-assessment).
Graphify reviewed this change.
Looks safe to merge — no coupling regressions and no blocking issues, checked against the code graph (not a self-assessment).
Graphify review — findings
This PR adds an install-sandbox CI pipeline and its supporting tooling to the tools/install_sandbox package. It introduces a new GitHub Actions workflow (install-sandbox.yml) that runs a full-catalog sandbox on relevant path changes, uploads diagnostic artifacts, writes a job summary, and classifies the result via a new ci_result module; it also adds a Pyright type-check step for tools/install_sandbox to the main CI. The change touches Docker command construction, run artifacts/reporting, models/specs, and a sandbox runner, with accompanying tests for CI result classification, Docker command building/output streaming, run artifacts, effects, lifecycle, and reporting CLI behavior.
No blocking issues surfaced. 3 lower-confidence candidates did not survive cross-model review.
Analysis details — impact, health, verification
Impact & health
Graphify review
Impact — 314 functions depend on the 314 functions this change touches.
Health — this change adds coupling hotspots:
- new:
run_scenario()— 7 callers, 19 callees - new:
validate_installed()— 12 callers, 11 callees - new:
SpecError— 12 callers, 8 callees - new:
run_universal_uninstall_scenario()— 6 callers, 14 callees - new:
main()— 5 callers, 12 callees - new:
validate_removed()— 8 callers, 6 callees - new:
load_target()— 5 callers, 9 callees - new:
file_scenario()— 6 callers, 4 callees - …and 16 more
Verification — 314 functions in the blast radius were not formally verified this run (proofs are advisory here).
Gate & verification
graphify gate
PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.
Advisory (not blocking):
- verification_scope: 314 function(s) in the blast radius were not formally verified this run
· 20 grounded finding(s) anchored inline below; 4 additional anchorable finding(s) not shown (cap).
| assert contains_json(actual, expected) | ||
|
|
||
|
|
||
| def test_json_backup_is_validated_and_allowed_as_a_declared_change(tmp_path): |
There was a problem hiding this comment.
test_json_backup_is_validated_and_allowed_as_a_declared_change()
fans out to 6 callees (efferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| assert (roots.output / "scenarios/demo-project/result.json").is_file() | ||
|
|
||
|
|
||
| def test_lifecycle_reports_not_applicable_user_uninstall(tmp_path): |
There was a problem hiding this comment.
test_lifecycle_reports_not_applicable_user_uninstall()
fans out to 7 callees (efferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| assert (roots.home / "user-owned.txt").is_file() | ||
|
|
||
|
|
||
| def file_scenario(name, scope=Scope.PROJECT): |
There was a problem hiding this comment.
file_scenario()
6 callers depend on it (afferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| assert classify_result(0, complete=False) == ("incomplete", 2) | ||
|
|
||
|
|
||
| def test_reporting_is_concise_and_writes_only_top_level_contract_files(tmp_path): |
There was a problem hiding this comment.
test_reporting_is_concise_and_writes_only_top_level_contract_files()
fans out to 6 callees (efferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| return value | ||
|
|
||
|
|
||
| def main(argv: Sequence[str] | None = None) -> int: |
There was a problem hiding this comment.
main()
high coupling complexity (Ca·Ce = 16).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| HARNESS_SPEC_DIR = Path(__file__).resolve().parent / "specs" | ||
|
|
||
|
|
||
| class RunInterrupted(Exception): |
There was a problem hiding this comment.
RunInterrupted
high coupling complexity (Ca·Ce = 16).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| artifacts.logger.write(stream, text) | ||
|
|
||
|
|
||
| def main( |
There was a problem hiding this comment.
main()
fans out to 12 callees (efferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| WarningSink = Callable[[str], None] | ||
|
|
||
|
|
||
| class ArtifactError(ValueError): |
There was a problem hiding this comment.
ArtifactError
15 callers depend on it (afferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| return selection | ||
|
|
||
|
|
||
| def make_run_id( |
There was a problem hiding this comment.
make_run_id()
high coupling complexity (Ca·Ce = 15).
Grounded coupling-delta finding (deterministic), not an LLM guess.
| return _PruneCandidate(path=path, started_at=started_at, collision=collision) | ||
|
|
||
|
|
||
| def prune_managed_runs( |
There was a problem hiding this comment.
prune_managed_runs()
9 callers depend on it (afferent coupling).
Grounded coupling-delta finding (deterministic), not an LLM guess.
1 participant
Hi @safishamsi,
Apologies for the delay, as per this conversation I created this PR for the docker sandbox validation for graphify installer. Feel free to ask any questions. I hope this could help the product.
Summary
This PR adds a Tier 1 Docker sandbox for Graphify’s installer. It exercises the real public CLI in isolated user and
project roots and verifies the Graphify-owned filesystem effects of installation, reinstallation, repair,
uninstallation, universal uninstallation, and purge.
This is intentionally a diagnostic PR: it does not modify graphify/install.py or include installer fixes. Its purpose
is to establish an independent, reproducible behavioral baseline before refactoring the installer.
What the sandbox adds
The sandbox currently covers 24 install targets and 50 target/scope scenarios. It verifies:
The source checkout is mounted read-only and copied to a separate installation root. HOME, XDG_CONFIG_HOME, the
project directory, user working directory, installed source, and diagnostic output are isolated from one another. The
real user home is never mounted.
Runs produce a readable report.md, machine-readable manifests and per-scenario results, command logs, filesystem
snapshots, and host-owned run.json/runner.log lifecycle diagnostics. Local managed runs use fresh directories and
safe keep-five retention.
The full Docker workflow is advisory while the unchanged installer has known behavioral failures: completed findings
are reported with artifacts and warnings, while incomplete, interrupted, malformed, or infrastructure-failed runs
fail the workflow.
The specs are central
Each YAML filename defines an install-target identity. The YAML owns target-specific facts such as supported scopes,
destinations, packaged payloads, owned markers, expected install surfaces, runtime limitations, and the current
universal-uninstall eligibility.
The strict loader owns schema validation, defaults, and deterministic derivation. The lifecycle code operates
generically over the loaded catalog; it does not maintain a second Python list or grouping of real target names.
The specs are also owned by the Docker image rather than loaded from the repository under test. This prevents a
product change from redefining its own expected result and making a defect pass its test.
The current YAML shape is not intended to be permanent. I would continue improving it by:
separating destination facts, executable recipe facts, ownership/cleanup identities, and independent assertions;
deriving values from filenames, scope, existing facts, or target-independent conventions whenever that is safe;
keeping validation order, reporting, purge behavior, and other cross-target harness policy in generic Python;
replacing transitional command-mode and universal-uninstall declarations when those decisions can be derived
safely;
describing complete generated Markdown and JSON payloads rather than relying only on required fragments;
adding precise current and legacy ownership identities and cleanup-only destinations.
The architectural goal is that adding a supported tool becomes a spec-file change, plus any payload referenced by
that spec—not another collection of target tables, dispatch branches, and cleanup lists scattered through Python.
In this first PR, the specs drive the sandbox and are checked against the existing public target list. Parts 2 and 3
below would make them the production installer’s source of truth as well.
Installer findings
Running the harness against unchanged v8 exposed six product-installer defects. The twelve failing scenario rows in
the clean full-catalog run map to these six root causes; they are not twelve different bugs.
SKILL.mdcan be removed while Graphify-ownedreferences/,references.tmp, or.graphify_versionartifacts remain.~/.claude/CLAUDE.md.graphify uninstallleaves Graphify-owned user integrations at multiple destinations.graphifycan prevent replacement with the current scope-correct registration.None of these fixes is included here, and the oracle has not been weakened to accept them.
If you agree with the classifications, I can open separate GitHub issues containing compact reproductions and
acceptance criteria, and provide independently reviewable fixes. I already have isolated fixes prepared for the first
four findings. I would address stale registration replacement before universal uninstall because it otherwise
interferes with clean aggregate setup and repair observations.
Validation
The harness-only branch is one commit above the verified v8 base and changes only the sandbox tooling, sandbox tests,
target specs, and scoped CI workflows.
Validation completed on the clean branch:
Reproduction:
uv run --frozen python tools/install_sandbox/run.py
--repo .
--all
--scope both
The sandbox proves Graphify-owned filesystem effects. It does not prove that a target application is installed,
authenticates, discovers the integration, loads it, or executes it. Windows scenarios in the Linux container verify
packaged payload consistency rather than native Windows lifecycle behavior.
CI/CD integration and developer impact
The sandbox is integrated at two complementary CI levels.
First, the existing test workflow keeps the fast checks blocking:
normal pytest loads the real YAML catalog and verifies that its filename-derived targets match the public graphify
install --help target list;
tools/install_sandbox is type-checked with Pyright;
these checks run with the existing CI suite, so missing specs, stale specs, invalid schema, and sandbox typing
regressions are caught quickly.
Second, a dedicated Install sandbox workflow runs the full Docker filesystem-effect validation as one Ubuntu/Python
3.12 job:
uv run --frozen python tools/install_sandbox/run.py
--repo .
--all
--scope both
--output "$RUNNER_TEMP/graphify-install-sandbox"
It runs:
on pull requests targeting v8 or main when installer code, packaged skills or commands, sandbox code or tests,
packaging metadata, uv.lock, or the workflow itself changes;
after pushes to v8 or main affecting those same paths;
nightly at 05:27 UTC;
manually through workflow_dispatch.
Unrelated changes do not start the Docker workflow. It is also kept outside the Python-version test matrix, so a
relevant change adds one full-catalog Docker job rather than repeating the matrix for every Python version. When a
pull request is updated, an older in-progress sandbox run for that PR is cancelled.
The workflow always preserves its diagnostic output, including when the sandbox reports failures. It uploads the
complete run directory under a unique run/attempt name, retains it for 14 days, and uses maximum compression. The
GitHub Actions job summary contains:
The result is deliberately classified rather than treating every nonzero sandbox exit as the same kind of failure:
This distinction matters while the unchanged installer still has known behavioral defects. Developers can see and
download the exact failed scenarios without those already-classified product findings blocking unrelated work.
However, failures in the sandbox itself—Docker setup, catalog loading, interrupted execution, missing evidence, or
inconsistent result metadata—remain visible CI failures.
For developers changing the installer, this adds an automatic cross-target regression report with no special local
setup required beyond the existing PR workflow. For developers adding a target, the fast catalog check requires the
public installer target and its spec to remain synchronized. The nightly run then provides continuing full-catalog
coverage even when no qualifying pull request or push has run recently.
Proposed follow-up
Part 2: production-oriented specs and a write-free plan
The next part would complete the YAML contract without changing the filesystem. The loader would normalize a target,
scope, package version, and installer options into a deterministic InstallPlan.
That plan would include complete payloads, destinations, ownership and removal identities, legacy replacements,
cleanup-only locations, runtime placeholders, path-resolution inputs, option variants such as strict mode, and
progressive-reference manifests.
The legacy CLI would remain the system being exercised, while the independent sandbox would compare its observed
effects with the normalized plan. This keeps schema review separate from mutation logic and proves that no
undisclosed target-specific behavior remains hidden in the legacy installer.
Part 3: Installer Core and CLI cutover
Once the plans are complete and independently validated, the final part would add the generic Installer Core that
applies them.
This part would own filesystem and structured-file mutation, transaction and rollback boundaries, symlink/permission/
encoding/invalid-input policy, error reporting, public CLI integration, and removal of superseded target-specific
installer paths.
The existing sandbox would remain independent and become the cutover acceptance contract: replacing the legacy
implementation with the new executor should preserve intended behavior, while deliberate behavior changes remain
explicit.
The result would be one target catalog feeding a generic loader, planner, installer, uninstaller, and behavioral
sandbox—so adding a new tool is primarily a matter of adding its spec.