Warn when the pre-test init time becomes much slower than a worker restart

[christiangnrd]

Close #182

Screenshots:

[giordano]

Haven't looked at the code, I appreciate the idea, but that message so long feels a bit too much, probably a very brief explanation and a pointer to the docs would be better

[christiangnrd]

Haven't looked at the code, I appreciate the idea, but that message so long feels a bit too much, probably a very brief explanation and a pointer to the docs would be better

I agree I'll fix it up in the near future

[christiangnrd]

@giordano This should be ready for final review. I improved the job selection docs so that the warning message has something to refer to, and I got rid of the pointless ptr_default face.

[giordano]

Looks good, apart from making default_njobs private

[giordano]

Uhm, default_njobs used to be non-internal, and I'm already using it in the wild 🥲 it's useful to query what would be the default number of jobs

[christiangnrd]

I’ll revert that

[christiangnrd]

I fixed this up. I somehow missed that this had been purposely made public so I also reverted the docstring changes I made in #142

[christiangnrd]

I'm going to clean up the commits and merge with rebase since I snuck in a few unrelated changes (like getting rid of the :ptr_default face)

[giordano]

Uhm, I'm seeing the warning at https://buildkite.com/clima/oceananigans/builds/33771#01a0c0e5-d574-45c1-b8fd-2abf0eb504b8/L719, but I don't think the machine is memory-constrained, GC time is good, following jobs have ~stable init time.

[christiangnrd]

Maybe I should increase the multiplier threshold to 3x?

[christiangnrd]

Or maybe same multipler but also a minimum threshold of 5 or 10 seconds or so

[giordano]

Yeah, another warning at https://buildkite.com/clima/oceananigans/builds/33771#01a0c0e5-d573-476c-bf5f-64e2bd93153e/L713, but there are only 11 concurrent workers out of 32 available cores at the moment, and memory usage is expected and well below critical values.

Maybe I should increase the multiplier threshold to 3x?

For the numbers I've seen so far 3x should suffice in these jobs, I'm now getting worried that any factor could have counterexamples 😕

[giordano]

Or maybe same multipler but also a minimum threshold of 5 or 10 seconds or so

A threshold may help

[christiangnrd]

Once I’m at my computer I’ll try threshold I have a feeling this is only a problem for fast init time testsuites. But that’s probably most.