Skip to content

Make sure workers don't outlive the manager - #214

Draft
giordano wants to merge 3 commits into
mainfrom
kill-workers-with-manager
Draft

giordano wants to merge 3 commits into
mainfrom
kill-workers-with-manager

Conversation

@giordano

Copy link
Copy Markdown
Collaborator

Haven't looked too closely at the implementation (so I'm opening as draft for the time being, but I want to save the progress) and it seems a bit too overcomplicated for my taste (but perhaps there isn't much to do about it), but the problem I'm trying to solve is to make sure workers are terminated when the driver process receives a SIGTERM or SIGKILL signal (for example when a CI job is cancelled): currently the workers may remain alive, because they don't receive the signal, if they're busy running the tests they keep going.

Malt spawns workers detached, so they don't receive the signals sent to the process group of a cancelled CI job, and a worker busy running a test doesn't notice the manager is gone. Ask the OS to kill the workers when the manager dies (parent-death signal on Linux and FreeBSD, a kill-on-close job object on Windows), and on exit, e.g. on SIGTERM, stop all remaining workers at once instead of one at a time with long timeouts.

Malt spawns workers detached, so they don't receive the signals sent to the
process group of a cancelled CI job, and a worker busy running a test doesn't
notice the manager is gone. Ask the OS to kill the workers when the manager
dies (parent-death signal on Linux and FreeBSD, a kill-on-close job object on
Windows), and on exit, e.g. on SIGTERM, stop all remaining workers at once
instead of one at a time with long timeouts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NkNLbZCkdtRSrsVx1XJQNr
Comment thread src/lifeline.jl
exits. Returns whether this was successful.
"""
function arm_manager_death!(manager_pid::Integer)
@static if Sys.islinux() || Sys.isfreebsd()

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reminder to self: test this PR on FreeBSD

giordano and others added 2 commits October 11, 2026 06:12
Calling the Windows API with `@ccall`, which always uses the C calling
convention, corrupts the stack on 32-bit Windows.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NkNLbZCkdtRSrsVx1XJQNr
The Windows API uses the stdcall calling convention on 32-bit Windows, so call
it with `ccall` rather than `@ccall`, and drop the restriction to 64-bit.
On Windows `kill` always terminates the process abruptly, also with SIGTERM,
so without the job object the workers were left running.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NkNLbZCkdtRSrsVx1XJQNr

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant