You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Actor records are persisted as protoJSON. protojson.Unmarshal in the store layer runs in strict mode, so any record written before a proto field rename becomes a poison pill after the rename:
ListActors fails entirely — one old record breaks listing for the whole atespace (and the e2e preflight, which lists across all atespaces).
DeleteActor cannot remove it either: the workflow's first step (LoadActorForDelete) unmarshals the record and dies on the same error. The record is unreachable through the API in both directions.
The only remedy is deleting the key directly in Valkey.
Concrete case
Commit ef7b29da renamed latest_snapshot_info → latest_snapshot and did it correctly at the proto level (reserved "latest_snapshot_info", new field number). That protects the wire format — but protoJSON stores field names, so reserved does not help persisted records. On a cluster installed before the rename and upgraded after it:
rpc error: code = Internal desc = internal server error: while listing actors in db:
in protojson.Unmarshal: proto: (line 1:391): unknown field "latestSnapshotInfo"
Error: ... workflow failed at step LoadActorForDelete: while fetching actor:
while unmarshaling actor: proto: unknown field "latestSnapshotInfo"
12 stale records (4 suspended actors + 8 orphaned golden actors) blocked every e2e suite's preflight until they were deleted with valkey-cli DEL against the raw keys.
Reproduction
Install a cluster, create + suspend an actor (its record now embeds the current field names).
Upgrade the control plane across any commit that renames a persisted proto field (e.g. ef7b29da).
kubectl ate get actors -a <atespace> → Internal error; kubectl ate delete actor ... → same error from LoadActorForDelete.
Suggested fix
Store-layer reads should use protojson.UnmarshalOptions{DiscardUnknown: true} — persisted data must tolerate fields the current schema no longer knows. (Related: Adopt binary protobuf encoding for control-plane records (Actor/Worker) #307 proposes binary protobuf encoding for control-plane records, which makes renames safe via field numbers.)
Alternatively (or additionally): treat persisted-JSON field names as frozen, and gate renames on a migration.
A doc note in the API guide that reserved alone does not protect protoJSON-persisted records would prevent the next instance.
Summary
Actor records are persisted as protoJSON.
protojson.Unmarshalin the store layer runs in strict mode, so any record written before a proto field rename becomes a poison pill after the rename:ListActorsfails entirely — one old record breaks listing for the whole atespace (and the e2e preflight, which lists across all atespaces).DeleteActorcannot remove it either: the workflow's first step (LoadActorForDelete) unmarshals the record and dies on the same error. The record is unreachable through the API in both directions.Concrete case
Commit
ef7b29darenamedlatest_snapshot_info→latest_snapshotand did it correctly at the proto level (reserved "latest_snapshot_info", new field number). That protects the wire format — but protoJSON stores field names, soreserveddoes not help persisted records. On a cluster installed before the rename and upgraded after it:12 stale records (4 suspended actors + 8 orphaned golden actors) blocked every e2e suite's preflight until they were deleted with
valkey-cli DELagainst the raw keys.Reproduction
ef7b29da).kubectl ate get actors -a <atespace>→ Internal error;kubectl ate delete actor ...→ same error fromLoadActorForDelete.Suggested fix
protojson.UnmarshalOptions{DiscardUnknown: true}— persisted data must tolerate fields the current schema no longer knows. (Related: Adopt binary protobuf encoding for control-plane records (Actor/Worker) #307 proposes binary protobuf encoding for control-plane records, which makes renames safe via field numbers.)reservedalone does not protect protoJSON-persisted records would prevent the next instance.