We observed a gradual buildup of runners and instances that ultimately led to our CI grind to a halt.
The problem seems to be that the start job is not marked as always(). This open a race condition where an EC2 is started, but the start job gets cancelled before it reports back. In that case, the stop job can't terminate the instance because it has not yet received its name. Similarly, a runner can be left orphaned.
It seems that the start job in a ec2-github-runner based workflow must be marked always() so it cannot be cancelled, and the above race does not happen.
Note that if the cancellation of start jobs is common if the workflow is part of a concurrency group. For example, if it is triggered upon updates to a fixed PR, occasional fast back-to-back updates to the same PR would lead to the race, and the buildup of orphaned runners and instances.
Suggestion: Change the REAME.md to mark start as always(), and document that this is important.
We observed a gradual buildup of runners and instances that ultimately led to our CI grind to a halt.
The problem seems to be that the
startjob is not marked asalways(). This open a race condition where an EC2 is started, but thestartjob gets cancelled before it reports back. In that case, thestopjob can't terminate the instance because it has not yet received its name. Similarly, a runner can be left orphaned.It seems that the
startjob in aec2-github-runnerbased workflow must be markedalways()so it cannot be cancelled, and the above race does not happen.Note that if the cancellation of
startjobs is common if the workflow is part of a concurrency group. For example, if it is triggered upon updates to a fixed PR, occasional fast back-to-back updates to the same PR would lead to the race, and the buildup of orphaned runners and instances.Suggestion: Change the
REAME.mdto markstartasalways(), and document that this is important.