Skip to content

Stored actor records with renamed proto fields become unlistable and undeletable #701

Description

Summary

Actor records are persisted as protoJSON. protojson.Unmarshal in the store layer runs in strict mode, so any record written before a proto field rename becomes a poison pill after the rename:

  • ListActors fails entirely — one old record breaks listing for the whole atespace (and the e2e preflight, which lists across all atespaces).
  • DeleteActor cannot remove it either: the workflow's first step (LoadActorForDelete) unmarshals the record and dies on the same error. The record is unreachable through the API in both directions.
  • The only remedy is deleting the key directly in Valkey.

Concrete case

Commit ef7b29da renamed latest_snapshot_infolatest_snapshot and did it correctly at the proto level (reserved "latest_snapshot_info", new field number). That protects the wire format — but protoJSON stores field names, so reserved does not help persisted records. On a cluster installed before the rename and upgraded after it:

rpc error: code = Internal desc = internal server error: while listing actors in db:
  in protojson.Unmarshal: proto: (line 1:391): unknown field "latestSnapshotInfo"

Error: ... workflow failed at step LoadActorForDelete: while fetching actor:
  while unmarshaling actor: proto: unknown field "latestSnapshotInfo"

12 stale records (4 suspended actors + 8 orphaned golden actors) blocked every e2e suite's preflight until they were deleted with valkey-cli DEL against the raw keys.

Reproduction

  1. Install a cluster, create + suspend an actor (its record now embeds the current field names).
  2. Upgrade the control plane across any commit that renames a persisted proto field (e.g. ef7b29da).
  3. kubectl ate get actors -a <atespace> → Internal error; kubectl ate delete actor ... → same error from LoadActorForDelete.

Suggested fix

  • Store-layer reads should use protojson.UnmarshalOptions{DiscardUnknown: true} — persisted data must tolerate fields the current schema no longer knows. (Related: Adopt binary protobuf encoding for control-plane records (Actor/Worker) #307 proposes binary protobuf encoding for control-plane records, which makes renames safe via field numbers.)
  • Alternatively (or additionally): treat persisted-JSON field names as frozen, and gate renames on a migration.
  • A doc note in the API guide that reserved alone does not protect protoJSON-persisted records would prevent the next instance.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions