pyln-testing: replace ephemeral-port-reserve with a filesystem lock approach

[daywalker90]

using pytest-xdist the ephemeral-port-reserve approach would only ensure no port conflicts per pytest worker. There could still be race conditions where we get "address already in use".

This new filesystem approach should work across pytest workers

Changelog-None

[cdecker]

I don't quite get why it is necessary to keep a list of reserved ports at all. The idea is to have the OS itself track the reservations, and have them time out after the usual 30s - 60s cleanup timeout, after which the TIME_WAIT state on the socket expires, and the port goes back into the pool of available ports (i.e., bind with that specific port no longer needs SO_REUSEADDR set, and the random port selection may return it).

So unless we have a test that somehow assumes we have control over a port, while not actively bound to it, for prolonged periods (30s - 60s+) we should never need to materialize the list of reservations, even worse, the list of reservations may think it has control over a port, when the OS has already given that port to someone else. So the materialized port list may actually be causing collisions that are harder to debug. To me, a test that runs for longer than 30s and still expects to have exclusive control over a port is the bigger problem. Due to the port returning to the free-pool, and the OS being allowed to reassign it, the list reservation approach will never work I think.

[daywalker90]

I don't quite get why it is necessary to keep a list of reserved ports at all. The idea is to have the OS itself track the reservations, and have them time out after the usual 30s - 60s cleanup timeout, after which the TIME_WAIT state on the socket expires, and the port goes back into the pool of available ports (i.e., bind with that specific port no longer needs SO_REUSEADDR set, and the random port selection may return it).

So unless we have a test that somehow assumes we have control over a port, while not actively bound to it, for prolonged periods (30s - 60s+) we should never need to materialize the list of reservations, even worse, the list of reservations may think it has control over a port, when the OS has already given that port to someone else. So the materialized port list may actually be causing collisions that are harder to debug. To me, a test that runs for longer than 30s and still expects to have exclusive control over a port is the bigger problem. Due to the port returning to the free-pool, and the OS being allowed to reassign it, the list reservation approach will never work I think.

For some reason ports go back into the available pool sooner than what we would expect. I'm not sure why, but it clearly happens.
ephemeral_port_reserve puts them in TIME_WAIT, we can bind to them immediately because of SO_REUSEADDR set, but then, probably because of node restarts, during a test the port is somehow returned to a state where ephemeral_port_reserve will return it again immediately for another test.
So we need an additional mechanism for the ports to stay consistent during a test with node restarts. Our current mechanism was flawed in combination with pytest-xdist since it was tracking per worker, not per machine. I did five runs with this on my fork repo and one run here with no "address already in use" errors, where before it would happen in atleast 50% of runs.

[daywalker90]

Did three more full runs and sadly got one with one case of "address already in use" (😭 ), still feels much more reliable though..

Also added two fixes for slow teardown because of improper exiting after shutdown notification and a fix for a grpc test flake, where there was a race between the grpc channel being ready and the custommsg already being sent.