Repository navigation
ateworker-microvm: give the guest CRNG reseed 60s on restore - #2743
Merged
Benjamin Elder (BenTheElder) merged 1 commit intoOct 20, 2026
Merged
Benjamin Elder (BenTheElder) merged 1 commit into
Benjamin Elder (BenTheElder) merged 1 commit into
Conversation
Haowei Cai (Roy) (roycaihw)
force-pushed
the
fix/microvm-reseed-retry
branch
2 times, most recently
from
October 16, 2026 05:22
d240255 to
1e8cedf
Compare
Contributor
Author
Haowei Cai (Roy) (roycaihw)
force-pushed
the
fix/microvm-reseed-retry
branch
from
October 16, 2026 05:37
1e8cedf to
56bbbf0
Compare
Contributor
Author
| return ch.MemRestoreEager | ||
| } | ||
|
|
||
| const reseedAttempts = 3 |
Collaborator
There was a problem hiding this comment.
The description says the campaign went to zero crashes with a 15s attempt plus one retry (up to 30s); this lands 3×5s (15s). Was a single 15s attempt tried? The only thing the retry adds over one longer call is a fresh ttrpc stream, and the agent still processes the abandoned 5s call, so attempt 2 queues behind it. Worth stating in the comment why per-attempt cancellation is preferred over one 15s deadline.
Collaborator
There was a problem hiding this comment.
+1 to this question :-)
Bowei Du (bowei)
added
area/microVM
area/reliability
kind/bug
Haowei Cai (Roy) (roycaihw)
force-pushed
the
fix/microvm-reseed-retry
branch
from
October 19, 2026 21:06
56bbbf0 to
dbc33ae
Compare
Haowei Cai (Roy) (roycaihw)
changed the title
Haowei Cai (Roy) (roycaihw)
force-pushed
the
fix/microvm-reseed-retry
branch
2 times, most recently
from
October 19, 2026 21:29
fb045b5 to
769f8d1
Compare
Collaborator
Collaborator
| // reseedTimeout bounds the guest CRNG reseed. It matches the 15s dialAgentRetry | ||
| // gives the kata-agent after resume: under concurrent restores on one node, a | ||
| // freshly unpaused guest can take more than 5s to schedule its kata-agent and | ||
| // answer, even after the vsock handshake succeeded. |
Collaborator
There was a problem hiding this comment.
Suggested change
| // answer, even after the vsock handshake succeeded. | |
| // reseedTimeout bounds the guest CRNG reseed. It needs to be long enough to handle concurrent reseeds. | |
| // a freshly unpaused guest can take more than 5s to schedule its guest agent and answer, so we choose 3x |
Under concurrent restores on one node, a freshly unpaused micro-VM guest can take more than 5s to schedule its kata-agent and answer ReseedRandomDev, even after the hybrid-vsock handshake succeeded, and the 5s timeout then failed the restore and crashed the actor. Give the reseed a single 60s deadline, which covers a resume wave on a heavily loaded host; a healthy guest still answers in milliseconds. Log the reseed duration in the restore phases. Co-authored-by: Davanum Srinivas <davanum@gmail.com>
Haowei Cai (Roy) (roycaihw)
force-pushed
the
fix/microvm-reseed-retry
branch
from
October 19, 2026 22:52
769f8d1 to
c2bcb8a
Compare
Haowei Cai (Roy) (roycaihw)
changed the title
Davanum Srinivas (dims)
approved these changes
Oct 19, 2026
|
|
||
| // reseedTimeout bounds ReseedRandomDev. It needs to cover a whole resume wave, not a | ||
| // single quiet guest. | ||
| const reseedTimeout = 60 * time.Second |
Collaborator
There was a problem hiding this comment.
this is REALLY high in substrate land, but I suggest we merge and revisit
Benjamin Elder (BenTheElder)
approved these changes
Oct 19, 2026
Contributor
|
/lgtm |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Under bursts of concurrent micro-VM restores on one node, a freshly unpaused guest can take more than 5s to schedule its
kata-agentand answer theReseedRandomDevttrpc call, even afterdialAgentRetrycompleted the hybrid-vsock handshake. The 5s timeout inreseedGuestCRNGthen failsRestoreWorkloadand crashes the actor.This PR gives
reseedGuestCRNGa single 60s deadline (reseedTimeout) and adds unit tests incmd/ateworker-microvm/restore_test.gofor the deadline and error propagation. It also adds areseedduration (the kata-agent dial plus the reseed, the head ofwakeup_probe) to theActor restore phaseslog, from #2878.Why 60s rather than 15s (from Davanum Srinivas (@dims) in #2878): on a heavily loaded host, the agent dial before the reseed already redials for up to 15s, and the slowest actors take more than 15s from resume to a reseeded guest. The deadline is a cap, not a wait: a healthy guest answers in milliseconds, so 60s costs nothing on the normal path and still fails a dead guest within a minute.
How I tested it
go test -race ./cmd/ateworker-microvm/..../hack/verify/gofmt.shand./hack/verify/boilerplate.shIn the 5,000-actor
nano-personal-agentmicro-VM benchmark campaign, a 15s reseed timeout plus one retry took these crashes from ~1 in 500–1,000 restores to 0 in later runs.From ateworker-microvm: give the post-restore CRNG reseed 60 s #2878: on a nested-virtualization host, a burst of 64 restores crashed 3 at 5s and ran clean at 60s. On bare metal, 5,976 restores in waves of up to 384 reseeded in 0.00s median and 0.28s max.
Tests pass
Appropriate changes to documentation are included in the PR