feat: Add control plane telemetry - actor crashes counter and operation labels by Angelawork · Pull Request #642 · agent-substrate/substrate

Angela (Angelawork) · 2026-07-30T19:30:05Z

Fixes #564 (Part 2)

Tests pass
Appropriate changes to documentation are included in the PR

Description

This PR implements the second part of #564 by adding telemetry counter instrumentation for ate.actor.crashes in ateapi and establishing bounded label validation for operations and crash failure causes.

Key Changes:

Updated maybeCrashActor and crashActor to reference ateattr.OperationName* constants and normalize opName.
Passed explicit failure reasons (ReasonCorruptedAssignment, ReasonWorkerPodGone, ReasonWorkerReassigned) across workflow_resume.go, workflow_suspend.go, and workflow_pause.go.
Instrumented WorkerPoolSyncer.releaseActorOnDeadWorker in syncer.go to record recordActorCrash when background pod deletion crashes an actor.
Instrumented FinalizePausedStep in workflow_pause.go to record recordActorCrash when nodeName is missing during pause finalization.
Added ate_actor_crashes to PlatformMetricPrefixes in collector_metrics.go.
Added pre-creation resource cleanup in metrics_test.go to prevent AlreadyExists errors on test reruns.
Updated metrics_test.go to trigger an actor crash via UpdateActor and ResumeActor, asserting ate_actor_crashes emission and label presence in OTel collector scrape outputs.

Testing

go test -buildvcs=false ./cmd/ateapi/internal/controlapi/...
go test -buildvcs=false ./cmd/atenet/internal/router/...
make test

E2E Test

./hack/create-kind-cluster.sh
./hack/install-ate-kind.sh --deploy-ate-system --deploy-demo-counter
./hack/run-e2e.sh ./internal/e2e/suites/metrics/...

…n and reason labels

…s suite

…es-telemetry

Jeff Luo (JeffLuoo) · 2026-08-04T14:56:37Z

Overall it LGTM, will let Krisztian F (@krisztianfekete) to also take another round of look.

Jeff Luo (JeffLuoo) · 2026-08-04T17:58:32Z

Please rebase

…-telemetry

Krisztian F (krisztianfekete)

Looking good, added one comment!

…es-telemetry

Jeff Luo (JeffLuoo)

One last change needed on documentation and I can merge it.

…es-telemetry

Angela (Angelawork) added 2 commits July 30, 2026 12:03

define operation constants and NormalizeOperationName

cb63a6f

feat(controlapi): tag ate.actor.crashes metrics with bounded operatio…

5f907e3

…n and reason labels

Angela (Angelawork) changed the title ~~Feature/actor crashes telemetry~~ feat: extend control plane telemetry with actor crashes counter and operation labels Jul 30, 2026

Angela (Angelawork) changed the title ~~feat: extend control plane telemetry with actor crashes counter and operation labels~~ feat: Add control plane telemetry - actor crashes counter and operation labels Jul 30, 2026

Angela (Angelawork) requested a review from Jeff Luo (JeffLuoo) July 30, 2026 19:32

Angela (Angelawork) force-pushed the feature/actor-crashes-telemetry branch 2 times, most recently from 4099ed0 to 0e45f4f Compare July 30, 2026 20:24

test(e2e): verify ate.actor.crashes metric counter in platform metric…

65c4522

…s suite

Angela (Angelawork) force-pushed the feature/actor-crashes-telemetry branch from 0e45f4f to 65c4522 Compare July 30, 2026 20:45

Krisztian F (krisztianfekete) reviewed Jul 31, 2026

View reviewed changes

Merge remote-tracking branch 'upstream/main' into feature/actor-crash…

10879ac

…es-telemetry

Jeff Luo (JeffLuoo) reviewed Jul 31, 2026

View reviewed changes

Comment thread internal/e2e/suites/metrics/metrics_test.go

Comment thread internal/e2e/suites/metrics/metrics_test.go

Comment thread internal/ateattr/ateattr.go Outdated

Angela (Angelawork) force-pushed the feature/actor-crashes-telemetry branch from fa2c10c to f69db52 Compare July 31, 2026 16:56

fix crash metric recording order and centralize failure reasons

5c4183e

Angela (Angelawork) force-pushed the feature/actor-crashes-telemetry branch from f69db52 to 5c4183e Compare July 31, 2026 17:36

Angela (Angelawork) requested a review from Krisztian F (krisztianfekete) July 31, 2026 17:48

Angela (Angelawork) force-pushed the feature/actor-crashes-telemetry branch from 2bb53ba to b19d783 Compare July 31, 2026 17:52

Jeff Luo (JeffLuoo) requested changes Jul 31, 2026

View reviewed changes

Angela (Angelawork) force-pushed the feature/actor-crashes-telemetry branch 2 times, most recently from ab0a9bc to 1d581e2 Compare July 31, 2026 20:59

Angela (Angelawork) requested a review from Jeff Luo (JeffLuoo) July 31, 2026 21:29

Krisztian F (krisztianfekete) reviewed Aug 3, 2026

View reviewed changes

Comment thread internal/ateattr/ateattr.go Outdated

Comment thread internal/e2e/suites/metrics/metrics_test.go

Angela (Angelawork) force-pushed the feature/actor-crashes-telemetry branch 4 times, most recently from ca5b1d8 to ce632e6 Compare August 4, 2026 14:00

Jeff Luo (JeffLuoo) requested changes Aug 4, 2026

View reviewed changes

Comment thread cmd/ateapi/internal/controlapi/metrics.go Outdated

Comment thread cmd/ateapi/internal/controlapi/crash.go Outdated

Comment thread cmd/ateapi/internal/controlapi/crash.go

Angela (Angelawork) force-pushed the feature/actor-crashes-telemetry branch from ce632e6 to 7b01c9a Compare August 4, 2026 14:41

Merge branch 'main' into feature/actor-crashes-telemetry

0040b12

Angela (Angelawork) force-pushed the feature/actor-crashes-telemetry branch from 7b01c9a to 0040b12 Compare August 4, 2026 14:48

Angela (Angelawork) added 2 commits August 4, 2026 15:50

Merge branch 'main' into feature/actor-crashes-telemetry

872ed8c

Merge remote-tracking branch 'origin/main' into feature/actor-crashes…

5ba57be

…-telemetry

Angela (Angelawork) force-pushed the feature/actor-crashes-telemetry branch 2 times, most recently from 70156c4 to 5ba57be Compare August 4, 2026 23:12

Krisztian F (krisztianfekete) reviewed Aug 5, 2026

View reviewed changes

Comment thread internal/ateattr/ateattr.go Outdated

Angela (Angelawork) force-pushed the feature/actor-crashes-telemetry branch 3 times, most recently from 9386e4d to 4a7bcb3 Compare August 5, 2026 14:39

Merge remote-tracking branch 'upstream/main' into feature/actor-crash…

8e3f258

…es-telemetry

Angela (Angelawork) force-pushed the feature/actor-crashes-telemetry branch 3 times, most recently from cdedced to 979a94a Compare August 5, 2026 15:27

Angela (Angelawork) requested a review from Jeff Luo (JeffLuoo) August 5, 2026 15:41

Jeff Luo (JeffLuoo) requested changes Aug 5, 2026

View reviewed changes

Comment thread docs/observability.md Outdated

Merge remote-tracking branch 'upstream/main' into feature/actor-crash…

315c552

…es-telemetry

Angela (Angelawork) force-pushed the feature/actor-crashes-telemetry branch from 979a94a to 315c552 Compare August 5, 2026 19:59

Angela (Angelawork) requested a review from Jeff Luo (JeffLuoo) August 5, 2026 20:31

Jeff Luo (JeffLuoo) approved these changes Aug 5, 2026

View reviewed changes

Jeff Luo (JeffLuoo) merged commit 1f8f842 into agent-substrate:main Aug 5, 2026
11 checks passed

Uh oh!

Conversation

Angela (Angelawork) commented Jul 30, 2026

Description

Key Changes:

Testing

E2E Test

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Jeff Luo (JeffLuoo) commented Aug 4, 2026

Uh oh!

Jeff Luo (JeffLuoo) commented Aug 4, 2026

Uh oh!

Krisztian F (krisztianfekete) left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Jeff Luo (JeffLuoo) left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants