runtests: Add parallel mode

[hdiethelm]

This adds a parallel execution mode for runtests and uses it in CI.

The linuxcnc instances are isolated using bwrap, so this works without #2722

Solves: #4588

Additionally, I fixed the issue that ctrl-c did not work with runtests.

ToDo:

• Properly check if all tests pass and also fail if they should
• Check the gui test CI artifacts

TBD if an issue:

• There is one issue: with -v, the stdout and stderr are not in order any more
• The output is not in order of the tests. --keep-order would allow to keep the order. However: parallel: Warning: No more file handles. with many threads

[hdiethelm]

Bummer: https://github.com/containers/bubblewrap/releases
--overlay-src is only available in 0.11.0 so it needs #4477 and won't work on debian bookworm.

[BsAtHome]

It is still a useful tool for running the tests on the local dev machine. Once CI is upgraded we can at least run trixie and sid on parallel tests.

[BsAtHome]

When you propagate -n in the WORKER_OPT, then you need to add bwrap options to map file stdout and stderr in the test's directory. Otherwise they will get lost when the overlay is removed.

To easily add extra options, you may want to make a list of bwrap options like:

BWRAPOPTS=(
    "--ro-bind" "/" "/"
    "--dev" "/dev"
    "--tmpfs" "/tmp"
    "--tmpfs" "/var/tmp"
    "--overlay-src" "$HOME" "--tmp-overlay" "$HOME"
    "--overlay-src" "$TOPDIR" "--tmp-overlay" "$TOPDIR"
    "--bind" "$TOPDIR/tests" "$TOPDIR/tests"
    "--unshare-ipc"
    "--unshare-pid"
    "--unshare-net"
    "--proc" "/proc"
    "--die-with-parent"
)
...
        CMD="bwrap ${BWRAPOPTS[*]} -- scripts/runtests ${WORKER_OPT[*]} -w {}"

[hdiethelm]

It is still a useful tool for running the tests on the local dev machine. Once CI is upgraded we can at least run trixie and sid on parallel tests.

Might be there are options... Backports, i could manually create overlays. Or just build the new bwrap in ci... ;-)

As long as one job takes longer, the whole CI run stays constant.

[BsAtHome]

Propagating -c does not make sense. It is resolved immediately below.
Propagating -s does not make sense because each test runs as a singular instance that always stops. However, parallel can be instructed to stop processing files when one process fails. That is where this option should be redirected to.
Propagating -d probably requires extra bwrap options to ensure the crash dump is accessible.

[hdiethelm]

When you propagate -n in the WORKER_OPT, then you need to add bwrap options to map file stdout and stderr in the test's directory. Otherwise they will get lost when the overlay is removed.

This option binds the whole test folder, so the results are directly written to the host's filesystem:
--bind \"$TOPDIR/tests\" \"$TOPDIR/tests\"

To easily add extra options, you may want to make a list of bwrap options like:

BWRAPOPTS=(
    "--ro-bind" "/" "/"
    "--dev" "/dev"
    "--tmpfs" "/tmp"
    "--tmpfs" "/var/tmp"
    "--overlay-src" "$HOME" "--tmp-overlay" "$HOME"
    "--overlay-src" "$TOPDIR" "--tmp-overlay" "$TOPDIR"
    "--bind" "$TOPDIR/tests" "$TOPDIR/tests"
    "--unshare-ipc"
    "--unshare-pid"
    "--unshare-net"
    "--proc" "/proc"
    "--die-with-parent"
)
...
        CMD="bwrap ${BWRAPOPTS[*]} -- scripts/runtests ${WORKER_OPT[*]} -w {}"

Thanks, looks better.

[BsAtHome]

When you propagate -n in the WORKER_OPT, then you need to add bwrap options to map file stdout and stderr in the test's directory. Otherwise they will get lost when the overlay is removed.

This option binds the whole test folder, so the results are directly written to the host's filesystem: --bind \"$TOPDIR/tests\" \"$TOPDIR/tests\"

That is a problem. Running parallel tests may have side effects. Some tests share files that are not meant to be accessed/changed in parallel.

[hdiethelm]

When you propagate -n in the WORKER_OPT, then you need to add bwrap options to map file stdout and stderr in the test's directory. Otherwise they will get lost when the overlay is removed.

This option binds the whole test folder, so the results are directly written to the host's filesystem: --bind \"$TOPDIR/tests\" \"$TOPDIR/tests\"

That is a problem. Running parallel tests may have side effects. Some tests share files that are not meant to be accessed/changed in parallel.

Hmm, are you sure? Each test is in a separate folder and I guess it should not write files in an other folder.

I could bind a temporary folder for each test, copy all content in and at the end, copy the results together. But this would be cumbersome. Might be there is an overlay option to do something similar, just more efficient.

[BsAtHome]

This option binds the whole test folder, so the results are directly written to the host's filesystem: --bind \"$TOPDIR/tests\" \"$TOPDIR/tests\"

That is a problem. Running parallel tests may have side effects. Some tests share files that are not meant to be accessed/changed in parallel.

Hmm, are you sure? Each test is in a separate folder and I guess it should not write files in an other folder.
I could bind a temporary folder for each test.

Yes, some tests share stuff. Most often they are in a sub-subdirectory of tests. For example, there are written variable files or intermediaries. The only two files we know of that should move out of the overlay are stdout and stderr. The rest should remain private.

[hdiethelm]

This option binds the whole test folder, so the results are directly written to the host's filesystem: --bind \"$TOPDIR/tests\" \"$TOPDIR/tests\"

That is a problem. Running parallel tests may have side effects. Some tests share files that are not meant to be accessed/changed in parallel.

Hmm, are you sure? Each test is in a separate folder and I guess it should not write files in an other folder.
I could bind a temporary folder for each test.

Yes, some tests share stuff. Most often they are in a sub-subdirectory of tests. For example, there are written variable files or intermediaries. The only two files we know of that should move out of the overlay are stdout and stderr. The rest should remain private.

Do you have an example of such a test? For the gui tests, there are also some images needed.

I need to see how to move files out of overlays with bwrap.

Just rebased on top of #4477 to see how well it works but in CI, some tests fail. And in CI+Docker, there are still permission issues.

Any clue where:
+SET_TERM_COND termCond=2, tolerance=0.001
arrives from? I had that also locally but then it just disappeared.

But 14min down to 1m45 would be quite an improvement. In CI, doc's are anyway the longest running process, so it would not decrease the overall runtime, just the worker usage.

Locally, it would be nice anyway.