Merge pull request #90 from cardmagic/test/caller-process-lifecycle

Back off lease-loss recovery and verify caller lifecycle

Author: cardmagic

CHANGELOG.md

@@ -1,5 +1,28 @@
 # Changelog
 
+## 0.17.5 - 2026-10-12
+
+- Back off repeated interrupted-message recovery using `retry_delay`.
+  The first claimed attempt recovers immediately; after attempts 2, 3, 4,
+  and onward, the delay callback receives 1, 2, 3, and onward (defaults:
+  1, 2, 4 seconds, capped at 60 seconds). Total persisted claim attempts,
+  including earlier handler failures, determine the delay. The ready timestamp
+  survives worker replacement and preserves mailbox order while other actors
+  can proceed. Lease loss alone still does not dead-letter a message; handler
+  failure attempt limits, setting names, defaults, and schema stay unchanged.
+- Emit `activation.lost` with the message ID and total attempt count when a
+  turn loses its fence, including while saving a handler failure or rejection,
+  matching JavaScript instrumentation. Shutdown requested
+  by that callback stops the worker before another pass. Skip a cached actor
+  whose first message is delayed so later ready messages cannot starve peers.
+
+- Preserve caller lifecycle guarantees with regression tests for concurrent
+  synchronous calls retaining their staged effects and for caller process
+  re-registration after stale-process cleanup and retention. Both cases
+  already work in Ruby; no runtime behavior or configuration changes.
+  The concurrency test waits for the second caller's busy-actor wait before
+  allowing the first caller to commit its staged effect.
+
 ## 0.17.4 - 2026-10-12
 
 - Renew activations between turns in a drain pass when renewal is due,

Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    solid_objects (0.17.4)
+    solid_objects (0.17.5)
       actioncable (>= 7.1)
       actionpack (>= 7.1)
       actionview (>= 7.1)
@@ -392,7 +392,7 @@ CHECKSUMS
   rubocop-rails-omakase (1.1.0) sha256=2af73ac8ee5852de2919abbd2618af9c15c19b512c4cfc1f9a5d3b6ef009109d
   ruby-progressbar (1.13.0) sha256=80fc9c47a9b640d6834e0dc7b3c94c9df37f08cb072b7761e4a71e22cff29b33
   securerandom (0.4.1) sha256=cc5193d414a4341b6e225f0cb4446aceca8e50d5e1888743fac16987638ea0b1
-  solid_objects (0.17.4)
+  solid_objects (0.17.5)
   sqlite3 (2.9.5-aarch64-linux-gnu) sha256=78075b6337d3d182c6d2b4691049ed45cd220826160c9ea18946bf6a1de200dc
   sqlite3 (2.9.5-aarch64-linux-musl) sha256=18c801185deb4adc01ddb281e8f672a39e3d1729979ca91e39439cd3eac0402d
   sqlite3 (2.9.5-arm-linux-gnu) sha256=1bdfca0c7d63998c60b0f4a8e3c8df2d33800ccc4abd2d612eddbbbc92a4c48b

compatibility/telemetry-events.json

@@ -84,6 +84,21 @@
         "sequence"
       ]
     },
+    {
+      "name": "activation.lost",
+      "attributes": [
+        "actorId",
+        "actorType",
+        "attempt",
+        "deliveryMode",
+        "durationMilliseconds",
+        "instanceId",
+        "messageId",
+        "operation",
+        "requestId",
+        "sequence"
+      ]
+    },
     {
       "name": "activation.started",
       "attributes": ["actorId", "actorType", "generation", "instanceId", "ownerId"],

docs/agents.md

@@ -173,6 +173,9 @@ Obey these rules in actor code:
   write in the same database.
 - Do not call an external API in a handler. Use `emit` and an effect handler.
 - Write each handler so that it can run again. Delivery is at least once.
+- Repeated lease losses back off but do not become dead letters. Observe
+  `activation.lost` and set an application stopping policy if a hard bound is
+  required; increasing the handler-failure attempt limit does not bound lease loss.
 - A reminder changes state only when it runs, and it runs only while
   `solid_objects start` runs. Do not compute expiry from the clock in a query;
   read the state that the reminder committed.

docs/observability.md

@@ -45,6 +45,7 @@ JavaScript does not await exporters. Telemetry is not a durable audit trail.
 | ------------------------------------------- | ---------------------------------------------------------------------------- |
 | `activation.started/completed/failed`       | Local actor activation hook lifecycle                                        |
 | `message.started/completed/failed/rejected` | One attempt's execution outcome                                              |
+| `activation.lost`                         | A message turn lost its activation fence; includes message ID and total attempt count |
 | `message.retry`                             | Failed attempt durably queued for another attempt                            |
 | `dead_letter.created`                       | Message exhausted retries or failed permanently                              |
 | `commit_action.started/completed/failed`    | One registered commit action inside the commit transaction                   |

docs/operations.md

@@ -606,3 +606,24 @@ heartbeat cleanup.
 Back up actor tables with the same consistency guarantees as application data.
 Restoring only instances without their mailboxes/outboxes, or vice versa, can
 violate application expectations.
+
+## Interrupted-message recovery
+
+An interrupted first claimed attempt recovers immediately. Recovery after
+persisted attempts 2, 3, 4, ... calls `retry_delay` with 1, 2, 3, ... . The
+callback returns seconds; defaults grow from 1 to 2 to 4 seconds up to 60 seconds.
+This uses total claim attempts, including earlier handler failures, not a
+separate lease-loss counter. The delay is stored in ready-message availability
+when a stale claim is recovered, survives worker replacement, and keeps later
+messages behind that message. Other actors can still run. Custom delays,
+including zero, remain authoritative.
+
+Lease loss alone does not create a dead letter, even with `max_attempts: 1`.
+Handler failures still use the full attempt count for `max_attempts` and
+`retry_delay`. An always-losing message remains pending indefinitely. Observe
+`activation.lost` (message ID and total attempt count) and impose an application
+stopping policy if a hard bound is required. Calling `request_shutdown` from
+instrumentation stops the worker before its next pass. Ruby uses a separate
+renewal thread and waits after lost turns; it does not have the JavaScript
+microtask starvation pattern. A handler must still return before its worker
+can finish shutting down.

docs/roadmap.md

@@ -25,7 +25,12 @@
   repeatable read. The tests count statements, and do not measure latency.
 - Activation leases, renewal between turns as well as in the renewal thread,
   unique activation tokens, generations, and
-  fenced commits
+  fenced commits. Interrupted first attempts recover immediately; repeated
+  interruptions use durable retry delays from total claim attempts minus one,
+  preserving mailbox order and progress for other actors. Lease loss emits
+  `activation.lost` with the message ID and attempt, including losses while
+  saving a handler failure or rejection. There is no hard retry cap;
+  applications needing one must impose a stopping policy.
 - Bounded activation passes, idle cache, hot-actor yield, and process records
 - At-least-once retries, terminal domain rejection, strict poison ordering,
   dead letters, and tail retry

docs/virtual-actors.md

@@ -170,6 +170,10 @@ again without harm.
   a crash or a lost lease.
 - Ordered turns do not cancel stale Ruby code. Fencing stops a stale activation
   from a commit, but that code can continue to run.
+- An interrupted first attempt recovers immediately. Repeated interruptions
+  use growing, durable retry delays while keeping mailbox order. Lease loss
+  alone has no attempt limit; use instrumentation and an application stopping
+  policy when a hard bound is required.
 - External effects can run more than once. Use the stable effect ID, or
   another durable key, as the idempotency key at the provider.
 - There are no transactions across actor identities, and there is no replay of

lib/solid_objects/activation.rb

@@ -52,7 +52,8 @@ def drain_until(message_id:, deadline:)
     # @rbs () -> bool
     def ready?
       now = SolidObjects.database_adapter.database_now
-      ReadyMessage.where(instance_id: lease.instance_id, available_at: ..now).exists? ||
+      ready_message = ReadyMessage.where(instance_id: lease.instance_id).order(:sequence).first
+      (ready_message && ready_message.available_at <= now) ||
         ClaimedMessage.where(
           instance_id: lease.instance_id,
           process_id: lease.owner_id,
@@ -227,11 +228,12 @@ def recover_stale_claim(instance)
       claimed_message.destroy!
       return if message.completed? || message.dead?
 
+      delay = (message.attempt_count <= 1) ? 0 : SolidObjects.configuration.retry_delay.call(message.attempt_count - 1)
       ReadyMessage.create!(
         message:,
         instance:,
         sequence: message.sequence,
-        available_at: SolidObjects.database_adapter.database_now
+        available_at: SolidObjects.database_adapter.database_now + delay
       )
     end
 

lib/solid_objects/executor.rb

@@ -16,6 +16,18 @@ def initialize(activation:, message:)
 
     # @rbs () -> bool
     def call
+      execute
+    rescue LostActivation
+      SolidObjects.instrument(:"activation.lost", **instrumentation_payload, duration_milliseconds: elapsed_milliseconds)
+      raise
+    end
+
+    private
+
+    attr_reader :activation, :message
+
+    # @rbs () -> bool
+    def execute
       state_before = actor.state.to_h
       observables_before = actor.observable_values
       message_context = MessageContext.new(
@@ -53,10 +65,6 @@ def call
       false
     end
 
-    private
-
-    attr_reader :activation, :message
-
     # @rbs () -> Actor
     def actor
       activation.actor