[nexus] tune libpq keepalive to catch dead CockroachDB connections faster#10756
Open
jam-mad wants to merge 2 commits into
Open
[nexus] tune libpq keepalive to catch dead CockroachDB connections faster#10756jam-mad wants to merge 2 commits into
jam-mad wants to merge 2 commits into
Conversation
…ster
Nexus used libpq's default TCP keepalive settings. On illumos, a dead
connection could go undetected for about 2 hours before the OS even
sent a probe, then another 8 minutes before giving up.
davepacheco tracked this down in a real incident: a Nexus instance
hung a rack update for 45+ minutes holding two dead connections to a
CockroachDB node that had reset. The cause: keepalives_idle,
keepalives_interval, and keepalives_count were never set, so Nexus
just used the OS defaults.
This sets those four parameters in make_postgres_connector()
(nexus/db-queries/src/db/pool.rs), cutting worst-case detection time
to about 2 minutes:
keepalives=1
keepalives_idle=10
keepalives_interval=10
keepalives_count=12
The connection args list already existed
(previously just sslmode=disable) and already flows through
DieselPgConnector::to_url() into libpq.
Added two tests in nexus/db-queries/src/db/pool_connection.rs (had no
coverage before):
- to_url_includes_keepalive_args: checks the connection string has the
new parameters.
- connection_actually_gets_tcp_keepalive_settings: opens a real
connection and reads back the actual kernel TCP settings, the
automated version of the mdb -k check davepacheco did by hand. Works
on Linux, macOS, and illumos via socket2's keepalive accessors, which
handle each OS's different option names. Only run on Linux so far.
Also re-ran the existing db::pool::test tests against a real
CockroachDB. Both still pass.
These timing values are a starting point, not final. Worth a second
look before merging.
Closes oxidecomputer#10668
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Tune libpq keepalive so dead CockroachDB connections are caught in minutes, not hours
Fixes #10668.
Summary
Nexus used libpq's default TCP keepalive settings. On illumos, that
means a dead CockroachDB connection can sit undetected for about 2
hours. This sets four keepalive parameters so detection takes about 2
minutes instead.
All credit for finding this goes to @davepacheco, from a real incident
where it hung a rack update for 45+ minutes. This PR implements his fix
and adds tests, including one that opens a real connection and reads
back the actual kernel TCP settings, the same check @davepacheco did by
hand with
mdb -k.The problem
A Nexus instance hung during a rack update, holding two dead
CockroachDB connections for 47+ minutes. @davepacheco traced it (full
details in #10668): the CockroachDB sled had reset, but Nexus never
noticed. Why:
keepalives_idle,keepalives_interval, andkeepalives_countwere never set, so Nexus used the OS defaults, 2hours to the first probe and 8 more minutes to give up.
The change
make_postgres_connector()inpool.rsalready builds a list of libpqconnection parameters (just
sslmode=disablebefore). This adds fourmore:
10 seconds to the first probe, 120 more seconds to give up. Worst case:
about 2 minutes instead of 2 hours.
These numbers are a starting point, close to what @davepacheco
suggested, not final. Worth a second look before merging: too
aggressive and we might drop connections during a normal CockroachDB
pause; too lax and this doesn't fix anything.
Testing
pool_connection.rshad zero test coverage before this. Added twotests:
to_url_includes_keepalive_args: checks the connection string hasthe right parameters. Fast, no database needed.
connection_actually_gets_tcp_keepalive_settings: opens a realconnection to a real (test) CockroachDB and reads back the actual
kernel TCP settings. The automated version of the
mdb -kcheck@davepacheco did by hand.
Diesel hides the raw socket, so the test opens a second, plain libpq
connection with the same connection string and reads that one
instead. Works on Linux, macOS, and illumos with no extra code, since
socket2's keepalive getters already handle each OS's differentoption names. I've only run this on Linux; illumos should work the
same way but I haven't confirmed it there.
Also re-ran the existing
db::pooltests against a real CockroachDB.Both still pass.
New dev-dependencies (
socket2,libc) were already workspace-managed,just not used by this crate before. Nothing new added to the dependency
graph.
How to re-run
Risk / non-goals
pool.rs,pool_connection.rs, and aCargo.tomlupdate. No schema, no API change.
too; not yet confirmed there.
raised in CockroachDB lost quorum on critical ranges during racklette update #10658.