🤖 docs: complete the CoderTemplateTest how-to

.cspell.json

@@ -19,6 +19,7 @@
     "rolebinding",
     "subresource",
     "subresources",
+    "finalizers",
     "configmap",
     "healthcheck",
     "ipairs",

README.md

@@ -15,7 +15,7 @@ Use native Kubernetes APIs to run and manage [Coder](https://coder.com).
 
 | Component | Manages | API group |
 | --- | --- | --- |
-| Operator | `CoderControlPlane`, `CoderProvisioner`, `CoderWorkspaceProxy` (custom resources) | `coder.com/v1alpha1` |
+| Operator | `CoderControlPlane`, `CoderProvisioner`, `CoderTemplateTest`, `CoderWorkspaceProxy` (custom resources) | `coder.com/v1alpha1` |
 | Aggregated API server | `CoderWorkspace`, `CoderTemplate`, `CoderTemplateVersion` (served from a live Coder instance) | `aggregation.coder.com/v1alpha1` |
 | MCP server | Tools that inspect and operate these resources over HTTP | — |
 

config/samples/coder_v1alpha1_codertemplatetest.yaml

@@ -0,0 +1,26 @@
+# Tests one Coder template version: the controller creates a workspace from
+# it, waits until every top-level agent is ready, records the result, and
+# deletes the workspace. See docs/how-to/test-templates.md.
+#
+# Prerequisites:
+# - CoderControlPlane codercontrolplane-sample with status.operatorAccessReady
+#   true and spec.templateTests.ownerUserID set to the tester's Coder user ID.
+# - The template below exists in Coder.
+#
+# The spec is immutable. To test again, delete this object or use a new name.
+apiVersion: coder.com/v1alpha1
+kind: CoderTemplateTest
+metadata:
+  name: codertemplatetest-sample
+  namespace: coder
+spec:
+  controlPlaneRef:
+    name: codercontrolplane-sample
+  # <organization>.<template>
+  template: "coder.docker"
+  version:
+    # Exactly one of active, name, or id.
+    active: true
+  timeoutSeconds: 900
+  # Optional: delete the test one hour after it finished.
+  # ttlSecondsAfterFinished: 3600

docs/how-to/deploy-controller.md

@@ -1,6 +1,6 @@
 # Deploy the controller
 
-Run `coder-k8s` as an operator only (`--app=controller`). The operator reconciles `CoderControlPlane`, `CoderProvisioner`, and `CoderWorkspaceProxy` resources.
+Run `coder-k8s` as an operator only (`--app=controller`). The operator reconciles `CoderControlPlane`, `CoderProvisioner`, `CoderWorkspaceProxy`, and `CoderTemplateTest` resources.
 
 Run the commands from a clone of this repository.
 
@@ -24,6 +24,9 @@ kubectl -n coder-system patch deployment/coder-k8s --type=json \
 !!! tip "Pin the image"
     The manifest uses `ghcr.io/coder/coder-k8s:latest`. To pin a version, change the tag before you apply the manifest.
 
+!!! warning "Upgrades: apply the CRDs and RBAC first"
+    Apply `config/crd/bases/` and `config/rbac/` from the new version before or together with the new image. The operator watches every kind it reconciles. If a kind's CRD is missing, for example `CoderTemplateTest` after an upgrade from an earlier version, the manager cannot start.
+
 ## 3. Verify
 
 ```bash

docs/how-to/test-templates.md

@@ -1,6 +1,6 @@
 # Test a template version with CoderTemplateTest
 
-This guide shows how to check that a Coder template version can start a workspace whose agents become ready. A `CoderTemplateTest` creates a throwaway workspace from one template version, waits until every agent is `connected` and `ready`, records `Succeeded` or `Failed`, and deletes the workspace. For every field, see the [`CoderTemplateTest` reference](../reference/api/codertemplatetest.md).
+This guide shows how to check that a Coder template version can start a workspace whose agents become ready. A `CoderTemplateTest` creates a throwaway workspace from one template version, waits until every top-level agent is `connected` and `ready`, records `Succeeded` or `Failed`, and deletes the workspace. For every field, see the [`CoderTemplateTest` reference](../reference/api/codertemplatetest.md).
 
 You need:
 
@@ -61,6 +61,154 @@ The spec is immutable. To test again, create a new object.
 
 The operator's own RBAC comes with the install. People who run tests need `create`, `get`, `list`, `watch`, and `delete` on `codertemplatetests.coder.com` in their namespace. They never need access to Coder or to the operator token.
 
+## 4. Read the result
+
+`status.phase` is `Pending` until the controller may have sent a create request, then `Running`, and finally `Succeeded` or `Failed`. `status.reason` and `status.message` say what the controller waits for or why the test failed. `status.templateVersionName` shows which version ran.
+
+### Waiting reasons
+
+While the test waits, the condition `Reconciling` is `True` with the same reason. Every wait ends at `spec.timeoutSeconds`: the test then fails as `DeadlineExceeded`, and the message names the last wait.
+
+| Reason | Meaning | What to do |
+| --- | --- | --- |
+| `Initializing` | The controller started the test. | Nothing. |
+| `ControlPlaneNotReady` | The `CoderControlPlane` does not exist, is being deleted, or has no usable `status.url`. | Check `spec.controlPlaneRef` and the control plane status. |
+| `OperatorAccessNotReady` | `status.operatorAccessReady` of the control plane is not `true`, or its operator token Secret is missing or has no token. | See [Deploy the controller](deploy-controller.md), and check the Secret in `status.operatorTokenSecretRef` of the control plane. |
+| `OwnerNotConfigured` | The control plane has no `spec.templateTests.ownerUserID`. | Set it, as in step 1. |
+| `OwnerNotEligible` | The tester does not exist, is not `active`, has a person's login type, has a role that is not allowed, or is not a member of the template's organization. | Fix the tester as described in step 1. The message names the problem. |
+| `TemplateNotFound` | The organization or the template in `spec.template` does not exist. | Check `spec.template` (`<organization>.<template>`). |
+| `TemplateVersionNotFound` | The version in `spec.version` does not exist. | Check `spec.version`, or push the version. |
+| `TemplateVersionImporting` | The version is still importing. | Wait. |
+| `CreatingWorkspace`, `CreateRetrying` | The controller sends the create request, or sends it again after a request that had no effect. | Wait. |
+| `ConfirmingCreate` | The answer to the create request was lost. The controller reads Coder until the workspace appears. | Wait. |
+| `WaitingForBuild`, `WaitingForAgents` | The start build runs, or an agent is not yet `connected` and `ready`. | Wait. The message names the agent. |
+| `AgentsReady`, `DeletingWorkspace` | Every top-level agent was ready. The controller does not check devcontainer sub-agents. The controller deletes the workspace. | Wait. |
+| `CoderUnavailable` | A Coder request failed. After HTTP 429, the controller retries with backoff. | Check Coder and its logs. |
+| `CoderAnswerMismatch` | Coder answered about another object than the one asked for. The controller logs it and retries. | Check proxies between the controller and Coder. |
+
+### Failure reasons
+
+A failed test has `Ready=False` and `Stalled=True` with the same reason. It does not run again: create a new test.
+
+| Reason | Meaning | What to do |
+| --- | --- | --- |
+| `DeadlineExceeded` | The test did not finish within `spec.timeoutSeconds`. | Fix the last wait in the message, or raise `spec.timeoutSeconds`. |
+| `TemplateDeprecated` | The template is deprecated and accepts no new workspaces. | Test another template. |
+| `TemplateVersionArchived`, `TemplateVersionImportFailed` | The version is archived, or its import did not succeed. | Push a working version. |
+| `TemplateVersionMismatch` | The version in `spec.version.id` belongs to another template. | Fix `spec.version` or `spec.template`. |
+| `CreateRejected` | Coder rejected the create request. The message holds Coder's answer. No workspace exists. | Fix the cause, for example `spec.parameters`, and create a new test. |
+| `CreateOutcomeUnknown` | No workspace from this test appeared after an uncertain create request. | Check Coder, then create a new test. |
+| `WorkspaceNameConflict` | A workspace with the test's name exists, and this test did not create it, or nothing proves that it did. | Read the `WorkspaceDeleted` condition. `NotCreated` means that this test never created a workspace: the test releases its finalizer by itself, and the workspace belongs to someone else, so leave it alone. For `OwnershipUnknown`, see the escape hatch below. |
+| `BuildFailed`, `BuildCanceled` | The start build failed, or someone else canceled it. | Read the build logs in Coder. |
+| `NoAgents` | The workspace has no top-level agents, so nothing proves that it works. Devcontainer sub-agents do not count. | Add a top-level agent to the template. |
+| `AgentConnectionTimeout`, `AgentStartError`, `AgentStartTimeout`, `AgentStopped` | An agent did not connect in time, its startup script failed or timed out, or it stopped. | Read the agent logs in Coder. The message names the agent. |
+| `WorkspaceChangedExternally`, `WorkspaceDeletedExternally` | Someone else started a build of the workspace, or deleted it. | Leave test workspaces alone, and create a new test. |
+| `DeleteBuildFailed` | Every top-level agent was ready, but the delete build failed. The controller retries the delete. | Read the delete build logs in Coder. |
+| `ControlPlaneGone` | The `CoderControlPlane` was deleted while the workspace could exist. | Look for the workspace in Coder and delete it there. |
+
+### `WorkspaceDeleted` reasons
+
+The condition `WorkspaceDeleted` tracks the workspace after the test ends or while the test is being deleted. The controller keeps its finalizer until the condition is `True`, or the reason is `Retained` or `ControlPlaneGone`.
+
+| Status | Reasons | Meaning |
+| --- | --- | --- |
+| `True` | `Deleted`, `DeletedExternally`, `NotCreated` | No workspace of this test exists. `ttlSecondsAfterFinished` applies. |
+| `False` | `CleanupPending`, `Deleting`, `DeleteRetrying` | The controller deletes the workspace. |
+| `False` | `ControlPlaneUnavailable`, `CoderAnswerMismatch` | Coder is not usable, or answered about another object. The controller retries. |
+| `False` | `Retained` | `retain` is set and allowed, so the workspace stays in Coder. |
+| `Unknown` | `CreateOutcomeUnknown` | The controller reads Coder to learn whether a workspace exists. |
+| `Unknown` | `OwnershipUnknown`, `ControlPlaneGone` | The workspace can exist, and the controller cannot delete it. See the escape hatch below. |
+
+## Limit the cost
+
+Each test is a real workspace: it uses compute and provisioner time until the controller deletes it. To cap the number of tests in a namespace, use a `ResourceQuota`. The quota counts finished tests too, until they expire or someone deletes them:
+
+```yaml
+apiVersion: v1
+kind: ResourceQuota
+metadata:
+  name: template-tests
+  namespace: coder
+spec:
+  hard:
+    count/codertemplatetests.coder.com: "10"
+```
+
+## Run nightly checks
+
+A `CronJob` can test the active version every night. It needs only `create` on `codertemplatetests`. `ttlSecondsAfterFinished` removes old results:
+
+```yaml
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: nightly-template-test
+  namespace: coder
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: Role
+metadata:
+  name: nightly-template-test
+  namespace: coder
+rules:
+  - apiGroups: ["coder.com"]
+    resources: ["codertemplatetests"]
+    verbs: ["create"]
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  name: nightly-template-test
+  namespace: coder
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: Role
+  name: nightly-template-test
+subjects:
+  - kind: ServiceAccount
+    name: nightly-template-test
+    namespace: coder
+---
+apiVersion: batch/v1
+kind: CronJob
+metadata:
+  name: nightly-template-test
+  namespace: coder
+spec:
+  schedule: "0 3 * * *"
+  concurrencyPolicy: Forbid
+  jobTemplate:
+    spec:
+      backoffLimit: 0 # a retry after a lost create response makes a second test
+      template:
+        spec:
+          serviceAccountName: nightly-template-test
+          restartPolicy: Never
+          containers:
+            - name: create-test
+              image: <an image with sh and kubectl>
+              command:
+                - sh
+                - -ec
+                - |
+                  kubectl create -f - <<'TEST'
+                  apiVersion: coder.com/v1alpha1
+                  kind: CoderTemplateTest
+                  metadata:
+                    generateName: docker-nightly-
+                    namespace: coder
+                  spec:
+                    controlPlaneRef:
+                      name: coder
+                    template: default.docker
+                    version:
+                      active: true
+                    ttlSecondsAfterFinished: 604800
+                  TEST
+```
+
+To find failed tests, run `kubectl -n coder get codertemplatetests`, or alert on the condition `Stalled=True`. To gate promotions instead, see [Gate template promotion with GitOps](gitops.md).
+
 ## Cleanup and the escape hatch
 
 The controller keeps a finalizer on each test until it has deleted the workspace or proved that none exists. These cases need an admin:

docs/how-to/troubleshooting.md

@@ -219,3 +219,69 @@ See [Promote a template version](../reference/aggregated-api-behavior.md#promote
 ## Aggregated requests return `400` or `409`
 
 These errors often come from the rules for names or for `resourceVersion`. See [Aggregated API behavior](../reference/aggregated-api-behavior.md).
+
+## A `CoderTemplateTest` stays `Pending`
+
+The test waits for something outside it. `status.reason` names it:
+
+```bash
+kubectl get codertemplatetest <name> -n <namespace> -o jsonpath='{.status.reason}: {.status.message}{"\n"}'
+```
+
+Usual causes:
+
+1. `OwnerNotConfigured` or `OwnerNotEligible`: the control plane has no tester, or the tester is not allowed. The message names the problem. See [Create the tester user](test-templates.md#1-create-the-tester-user).
+2. `OperatorAccessNotReady` or `ControlPlaneNotReady`: the control plane in `spec.controlPlaneRef` is missing or not ready. See [`CoderControlPlane` stays `Pending`](#codercontrolplane-stays-pending).
+3. `TemplateNotFound` or `TemplateVersionNotFound`: `spec.template` or `spec.version` names nothing in Coder. The spec is immutable, so create a new test with the right names.
+4. `CoderUnavailable`: a Coder request failed. Check Coder and its logs.
+
+A new test has no status for a moment: the first reconcile only adds the finalizer `coder.com/template-test-cleanup`, and the next reconcile writes the status. If the status stays empty, check the finalizer:
+
+```bash
+kubectl get codertemplatetest <name> -n <namespace> -o jsonpath='{.metadata.finalizers}{"\n"}'
+```
+
+Without the finalizer, the controller has not reconciled the test. See [The controller runs but nothing reconciles](#the-controller-runs-but-nothing-reconciles). With the finalizer, the controller reconciled the test once: read the controller logs for errors about the test.
+
+Every wait ends at `spec.timeoutSeconds` with `DeadlineExceeded`. For every reason, see [Read the result](test-templates.md#4-read-the-result).
+
+## A `CoderTemplateTest` does not finish deleting
+
+The test keeps the finalizer `coder.com/template-test-cleanup` until the condition `WorkspaceDeleted` is `True`, or its reason is `Retained` or `ControlPlaneGone`. Read the condition:
+
+```bash
+kubectl get codertemplatetest <name> -n <namespace> \
+  -o jsonpath='{range .status.conditions[?(@.type=="WorkspaceDeleted")]}{.status} {.reason}: {.message}{"\n"}{end}'
+```
+
+- `Deleting` or `DeleteRetrying`: the delete build runs or failed. Read its logs in Coder. The controller retries.
+- `ControlPlaneUnavailable` or `CoderAnswerMismatch`: Coder is not usable. Fix Coder, and the controller continues.
+- `OwnershipUnknown`: a workspace with the test's name exists, but nothing proves that the test created it. The controller never touches it.
+
+If the controller cannot finish, use the [escape hatch](test-templates.md#cleanup-and-the-escape-hatch). As a last resort, check Coder for a workspace named `status.workspaceName`, then remove only the controller's finalizer. Other controllers can have their own finalizers on the test, so do not remove the whole list:
+
+1. Find the position of `coder.com/template-test-cleanup` in the list. The first entry has position 0.
+
+    ```bash
+    kubectl get codertemplatetest <name> -n <namespace> -o jsonpath='{.metadata.finalizers}'
+    ```
+
+2. Remove the entry at that position. The `test` operation makes the patch fail if the entry at `<position>` is a different finalizer.
+
+    ```bash
+    kubectl patch codertemplatetest <name> -n <namespace> --type json -p \
+      '[{"op":"test","path":"/metadata/finalizers/<position>","value":"coder.com/template-test-cleanup"},{"op":"remove","path":"/metadata/finalizers/<position>"}]'
+    ```
+
+CAUTION: Delete the workspace in Coder before you remove the finalizer. Otherwise the workspace, and the tester's session key, stay in Coder.
+
+## A namespace stays `Terminating`
+
+1. Tests in the namespace wait for their cleanup. List them with `kubectl get codertemplatetests -n <namespace>`, then see [A `CoderTemplateTest` does not finish deleting](#a-codertemplatetest-does-not-finish-deleting).
+2. With the aggregated API server installed, a namespace without an eligible `CoderControlPlane` never finishes deleting ([#209](https://github.com/coder/coder-k8s/issues/209)). Its condition `NamespaceDeletionContentFailure` is `True` with the message `no eligible CoderControlPlane instances found in namespace "<namespace>"`. The aggregated API answers the namespace controller's LIST with `503`. Read the conditions:
+
+    ```bash
+    kubectl get namespace <namespace> -o jsonpath='{range .status.conditions[*]}{.type}={.status} {.message}{"\n"}{end}'
+    ```
+
+    If `NamespaceContentRemaining` and `NamespaceFinalizersRemaining` are `False`, nothing is left in the namespace, and only #209 holds it.