Stabilize tls_stress_test metrics queries under exhausted session caps

[eddyashton]

Summary

tls_stress_test is intermittently failing in AL4 VMSS Virtual C while the
connection-cap scenario deliberately exhausts sessions and file descriptors.
After expected connection failures, the test performs an unguarded
GET /node/metrics through clients[0]. That request may be reset or receive
503 SessionCapExhausted, failing the whole CI job.

This has occurred on unrelated PRs and predates the PR where it was most
recently observed.

Most recent occurrence

• PR: Reduce perf submitter memory use #8165
• Failed job:
  https://github.com/microsoft/CCF/actions/runs/32135397827/job/95705417848
• Workflow/job: Continuous Integration AL4 / AL4 VMSS Virtual C
• Test: tls_stress_test
• Failure path:

tests/connections.py:231 run_connection_caps_tests
tests/connections.py:176 create_connections_until_exhaustion
  r = clients[0].get("/node/metrics")
tests/infra/clients.py:696
  raise CCFIOException from exc
infra.clients.CCFIOException

The underlying exception was:

ConnectionResetError: [Errno 104] Connection reset by peer
httpx.ReadError: [Errno 104] Connection reset by peer

The normal VMSS Virtual C job passed on the same PR commit; only the AL4 job
failed:

• Passed:
  https://github.com/microsoft/CCF/actions/runs/32135397799/job/95705418136
• Failed:
  https://github.com/microsoft/CCF/actions/runs/32135397827/job/95705417848

Evidence that this is pre-existing and intermittent

The same request at tests/connections.py:176 failed with
infra.clients.CCFIOException on July 26:

• Historical job:
  https://github.com/microsoft/CCF/actions/runs/30214888955/job/89827281197
• Unrelated recovery PR:
  Validate recovery snapshot endorsement chains in memory #8092

That recovery PR did not modify tests/connections.py or the TLS/session-cap
implementation. Its preceding AL4 run passed, this run failed, and its next 11
AL4 runs passed without a change to tests/connections.py.

Related failures in the same connection-cap scenario have also returned
503 SessionCapExhausted from metrics queries:

• https://github.com/microsoft/CCF/actions/runs/30554767740/job/90912282664
• https://github.com/microsoft/CCF/actions/runs/30549183462/job/90893078340
• https://github.com/microsoft/CCF/actions/runs/30535447937/job/90847555194

The metrics request at tests/connections.py:176 dates from 2021. The loop
immediately before it already treats CCFConnectionException,
CCFIOException, and RuntimeError as expected while searching for the
session/file-descriptor limit, but the subsequent metrics request has no
equivalent handling.

Relevant code

• tests/connections.py
  • get_session_metrics()
  • run_connection_caps_tests()
  • nested create_connections_until_exhaustion()
  • particularly the unguarded clients[0].get("/node/metrics")
• tests/infra/clients.py
  • conversion of httpx.ReadError to CCFIOException

Investigation notes

Determine which invariant is intended:

1. An already admitted session must remain usable while new sessions are
   rejected. If so, the reset/503 may expose a CCF session-cap bug and the test
   should retain a strict assertion.
2. A reset or temporary 503 is valid while the test is deliberately exhausting
   file descriptors and session caps. If so, the test should explicitly wait
   for cap state to settle and retry the metrics query using an appropriate
   existing session, rather than failing on a timing-dependent request.

Avoid simply swallowing the exception. Preserve the assertions on active/peak
session metrics and ensure a persistent inability to query metrics still fails
with useful diagnostics.

Also check whether the test should use the existing get_session_metrics()
helper consistently rather than making a separate direct request.

Suggested validation

1. Build Debug on Azure Linux 4, matching the failing workflow.
2. Run tls_stress_test repeatedly through the repository tests.sh wrapper.
3. Confirm the connection-cap assertions still detect real cap regressions.
4. Confirm transient reset/503 behavior no longer fails the test if it is
   considered valid.
5. Run the complete AL4 bucket C job.

Acceptance criteria

• The intended behavior of existing sessions at the soft/hard cap is explicit
  in the test.
• tls_stress_test is stable under repeated AL4 runs.
• Persistent metrics-query failure remains actionable and fails the test.
• Session active/peak/soft-cap/hard-cap assertions are preserved.