fix(lambda): add post-deregister busy check to prevent terminating active runners#5086
Open
JVenberg wants to merge 1 commit intogithub-aws-runners:mainfrom
Open
Conversation
…tive runners The scale-down lambda had a TOCTOU race condition where a job could be assigned to a runner between checking its busy state and terminating the EC2 instance. This caused in-flight jobs to be killed mid-execution. The fix adds a post-deregistration busy re-check: 1. Check busy (fast-path to skip busy runners) 2. Deregister from GitHub (prevents new job assignment) 3. Re-check busy (now stable since no new jobs can be assigned) If the runner became busy between step 1 and 2, the in-flight job completes using its job-scoped OAuth token and the instance is left for orphan cleanup. Fixes github-aws-runners#5085
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #5085
The scale-down lambda can terminate an EC2 instance while it's actively running a job. This happens because a job can be assigned to a runner between checking its busy state and calling
TerminateInstances.We hit this in production: a Helm deploy was killed mid-execution when the scale-down lambda terminated the instance 13 seconds after the job started. CloudTrail confirmed the
TerminateInstancescall came from the scale-down lambda at the exact moment the runner received a shutdown signal.The race condition
Current flow in
removeRunner:falseThe fix
Add a post-deregistration busy re-check:
If the re-check finds the runner busy, we skip termination and let the instance be cleaned up as an orphan once the job finishes.
Why this is safe
Deregistering a runner does not affect in-flight jobs. The runner worker uses job-scoped OAuth credentials from the job message, not the runner registration:
JobRunner.cslines 80-95: the worker creates its ownVssConnectionusingsystemConnectioncredentials from the job messageTest plan