Summary
deploy_ate_system gates manifests/ate-install/generated/ behind ensure_crds, which returns early when the three CRDs already exist. On any upgrade of an existing cluster that skips the whole directory — and since role.yaml lives there and no other deploy path applies it, both CRD schemas and every ClusterRole stay frozen at first-install state.
A controller image that needs a newly added permission then deadlocks at startup: its informer cache cannot list the resource, so reconciliation never runs, while the rollout itself reports success.
Observed on a real cluster
kind cluster installed 4 days ago, then upgraded via install-ate-kind.sh --deploy-ate-system:
- ate-controller (new image) logged
failed to list *v1.NetworkPolicy: ... User "system:serviceaccount:ate-system:ate-controller" cannot list resource "networkpolicies"
and stopped reconciling entirely — a deleted WorkerPool Deployment was never recreated, with no events anywhere.
- After manually applying
manifests/ate-install/generated/role.yaml and restarting the controller, reconciliation resumed immediately.
- With the fix, the same upgrade run printed
actortemplates.ate.dev configured / workerpools.ate.dev configured — the CRD schemas had also silently drifted.
Reproduction
- Install:
./hack/install-ate-kind.sh --deploy-ate-system
- Advance the tree to any revision whose controller requires a new RBAC rule (or edit the live ClusterRole to drop one).
- Upgrade with the same command: rollout reports success,
deploy_crds never runs, controller deadlocks on informer start.
Suggested fix
Call deploy_crds unconditionally from deploy_ate_system — kubectl apply is idempotent, so re-applying on every install/upgrade is free. ensure_crds remains correct for the demo scripts and per-component deploy flags, where the semantic really is "make sure they exist".
Summary
deploy_ate_systemgatesmanifests/ate-install/generated/behindensure_crds, which returns early when the three CRDs already exist. On any upgrade of an existing cluster that skips the whole directory — and sincerole.yamllives there and no other deploy path applies it, both CRD schemas and every ClusterRole stay frozen at first-install state.A controller image that needs a newly added permission then deadlocks at startup: its informer cache cannot list the resource, so reconciliation never runs, while the rollout itself reports success.
Observed on a real cluster
kind cluster installed 4 days ago, then upgraded via
install-ate-kind.sh --deploy-ate-system:failed to list *v1.NetworkPolicy: ... User "system:serviceaccount:ate-system:ate-controller" cannot list resource "networkpolicies"and stopped reconciling entirely — a deleted WorkerPool Deployment was never recreated, with no events anywhere.
manifests/ate-install/generated/role.yamland restarting the controller, reconciliation resumed immediately.actortemplates.ate.dev configured/workerpools.ate.dev configured— the CRD schemas had also silently drifted.Reproduction
./hack/install-ate-kind.sh --deploy-ate-systemdeploy_crdsnever runs, controller deadlocks on informer start.Suggested fix
Call
deploy_crdsunconditionally fromdeploy_ate_system—kubectl applyis idempotent, so re-applying on every install/upgrade is free.ensure_crdsremains correct for the demo scripts and per-component deploy flags, where the semantic really is "make sure they exist".