fix(spartan): keep prover agents alive while proving jobs are in flight

[AztecBot]

What

Adds a third KEDA trigger to the prover-agent ScaledObject that holds one replica per in-progress proving job, so the agent pool is never scaled down (or deactivated to zero) while proofs are actually running.

- type: prometheus
  metadata:
    metricName: "aztec_proving_queue_active_jobs_count"
    query: "(sum(aztec_proving_queue_active_jobs_count{k8s_namespace_name=\"<ns>\"}) or vector(0))"
    threshold: "1"
    activationThreshold: "0"

Why

Testnet epoch 1274 failed to prove today and the chain was pruned back to checkpoint 37924 / block 42512 at 13:24:25 UTC, un-mining 21 txs and taking the in-progress epoch 1275 with it.

The root cause is that the ScaledObject scales purely on pending queue depth:

sum(aztec_proving_queue_size{k8s_namespace_name="testnet"})

ProvingBroker.measureQueueDepth returns this.queues[type].length() — the pending queue only. As soon as the broker hands a job to an agent it moves to this.inProgress and drops out of that metric (yarn-project/prover-client/src/proving_broker/proving_broker.ts:174). So a fully-busy agent pool reports a queue depth of zero, every band evaluates to 0, and with activationThreshold: "0" KEDA deactivates the deployment to zero replicas and SIGTERMs agents that are mid-proof.

Job 1274:PUBLIC_TX_BASE_ROLLUP:50628bb6… was killed this way three times in 2m20s, exhausting its retry budget:

Time (UTC)	Event
12:15:09	Job enqueued, started on agent q27pt
12:15:12	Witness generated (2.7s)
12:15:17	Shutting down... → job aborted
12:15:48	Re-enqueued (retry 1/3), started on agent k5n7k
12:16:00	Pod Killing → job aborted
12:16:30	Re-enqueued (retry 2/3), started on agent 729kp
12:17:00	KEDAScaleTargetDeactivated: from 4 to 0 → job aborted
12:17:30	timed out after 3 attempts. Marking as failed.
12:45:09	Error in EpochSession …: Sub-tree for checkpoint 6 failed: Error: Timed out — epoch 1274 top tree cancelled
13:24:25	Detected epoch prune to 37924; Chain pruned to block 42512

Every attempt produced its witness in under 3 seconds and was then terminated, so the ~30s broker job timeout was never the binding constraint — pod termination was. Across the 2-hour window the deployment hit zero replicas 26 times (4 → 0 and 8 → 0), roughly every 6–7 minutes.

How it works

KEDA takes the largest desired replica count across triggers, and only deactivates a ScaledObject to zero when every trigger is inactive. With threshold: "1" the new trigger asks for exactly active_jobs replicas, so it acts purely as a floor:

• Jobs in flight → the deployment cannot be scaled below the number of running proofs, and cannot be deactivated at all.
• Queue and in-flight both empty → all triggers inactive, scale-to-zero still works exactly as today.

The existing band triggers are unchanged and stay driven by pending queue depth, so the tuning in each environment's PROVER_AGENT_KEDA_SCALING_BANDS keeps its current meaning. The new trigger is appended last so the band triggers keep their current s0-prometheus, s1-prometheus, … indices.

aztec_proving_queue_active_jobs_count is the existing PROVING_QUEUE_ACTIVE_JOBS observable gauge from ProvingBrokerInstrumentation, counting this.inProgress. It is already scraped and plotted in the prover dashboard with the same label selector (spartan/metrics/grafana/dashboards/aztec_provers.json:508), so no new instrumentation or metrics-whitelist change is needed. The or vector(0) guard matches the existing band queries and keeps the expression defined when the series is absent.

The floor assumes one concurrent job per agent pod (PROVER_AGENT_COUNT=1, the default and what testnet runs). With a higher agent count per pod it over-provisions rather than under-provisions, which is the safe direction; a comment in the template records this.

Testing

helm template rendered against a testnet-shaped release and against the no-bands fallback path; output parsed with a YAML loader and triggers inspected. Both branches of the template render valid ScaledObjects and the band trigger indices are unchanged:

s0-prometheus: aztec_proving_queue_size_agents_4_over_0
               ((sum(aztec_proving_queue_size{k8s_namespace_name="testnet"}) or vector(0)) > bool 0) * 4
s1-prometheus: aztec_proving_queue_active_jobs_count
               (sum(aztec_proving_queue_active_jobs_count{k8s_namespace_name="testnet"}) or vector(0))

There is no unit-test harness for these chart templates, so this was not covered by an automated red/green test.

Not covered here

This stops KEDA from choosing to kill a busy agent. It does not make agent shutdown itself graceful — the agent still aborts its job in the same millisecond it receives SIGTERM (Shutting down... and Aborted job controller for jobId=… share a timestamp), so any other eviction (node drain, rollout, preemption) still loses in-flight work. Draining properly needs an app-side change in the proving agent plus a matching terminationGracePeriodSeconds; a longer grace period alone would do nothing while the process aborts on the first signal. Worth doing as a follow-up.

Full log analysis: https://gist.github.com/AztecBot/545c97fe180ff1bcf32bfd2a41453499

---

Created by claudebox · group: slackbot · requested by Alex Gherghisan · Slack thread