Skip to content

[nexus] tune libpq keepalive to catch dead CockroachDB connections faster#10756

Open
jam-mad wants to merge 2 commits into
oxidecomputer:mainfrom
jam-mad:jam-mad/tune-cockroachdb-keepalive
Open

[nexus] tune libpq keepalive to catch dead CockroachDB connections faster#10756
jam-mad wants to merge 2 commits into
oxidecomputer:mainfrom
jam-mad:jam-mad/tune-cockroachdb-keepalive

Conversation

@jam-mad

@jam-mad jam-mad commented Jul 6, 2026

Copy link
Copy Markdown

Tune libpq keepalive so dead CockroachDB connections are caught in minutes, not hours

Fixes #10668.

Summary

Nexus used libpq's default TCP keepalive settings. On illumos, that
means a dead CockroachDB connection can sit undetected for about 2
hours. This sets four keepalive parameters so detection takes about 2
minutes instead.

All credit for finding this goes to @davepacheco, from a real incident
where it hung a rack update for 45+ minutes. This PR implements his fix
and adds tests, including one that opens a real connection and reads
back the actual kernel TCP settings, the same check @davepacheco did by
hand with mdb -k.

The problem

A Nexus instance hung during a rack update, holding two dead
CockroachDB connections for 47+ minutes. @davepacheco traced it (full
details in #10668): the CockroachDB sled had reset, but Nexus never
noticed. Why: keepalives_idle, keepalives_interval, and
keepalives_count were never set, so Nexus used the OS defaults, 2
hours to the first probe and 8 more minutes to give up.

The change

make_postgres_connector() in pool.rs already builds a list of libpq
connection parameters (just sslmode=disable before). This adds four
more:

let args = vec![
    ("sslmode", "disable"),
    ("keepalives", "1"),
    ("keepalives_idle", "10"),
    ("keepalives_interval", "10"),
    ("keepalives_count", "12"),
];

10 seconds to the first probe, 120 more seconds to give up. Worst case:
about 2 minutes instead of 2 hours.

These numbers are a starting point, close to what @davepacheco
suggested, not final. Worth a second look before merging: too
aggressive and we might drop connections during a normal CockroachDB
pause; too lax and this doesn't fix anything.

Testing

pool_connection.rs had zero test coverage before this. Added two
tests:

  • to_url_includes_keepalive_args: checks the connection string has
    the right parameters. Fast, no database needed.

  • connection_actually_gets_tcp_keepalive_settings: opens a real
    connection to a real (test) CockroachDB and reads back the actual
    kernel TCP settings. The automated version of the mdb -k check
    @davepacheco did by hand.

    Diesel hides the raw socket, so the test opens a second, plain libpq
    connection with the same connection string and reads that one
    instead. Works on Linux, macOS, and illumos with no extra code, since
    socket2's keepalive getters already handle each OS's different
    option names. I've only run this on Linux; illumos should work the
    same way but I haven't confirmed it there.

Also re-ran the existing db::pool tests against a real CockroachDB.
Both still pass.

New dev-dependencies (socket2, libc) were already workspace-managed,
just not used by this crate before. Nothing new added to the dependency
graph.

How to re-run

cargo test -p nexus-db-queries --lib pool_connection
cargo nextest run -p nexus-db-queries -E 'test(pool_connection)'
cargo nextest run -p nexus-db-queries -E 'test(db::pool::test)'

Risk / non-goals

  • Small change: pool.rs, pool_connection.rs, and a Cargo.toml
    update. No schema, no API change.
  • Kernel-state test only ran on Linux so far. Should work on illumos
    too; not yet confirmed there.
  • Doesn't touch the separate duplicate-connection issue sunshowers
    raised in CockroachDB lost quorum on critical ranges during racklette update #10658.
  • Timing values are open to change; see above.

jam-mad and others added 2 commits July 4, 2026 03:51
…ster

Nexus used libpq's default TCP keepalive settings. On illumos, a dead
connection could go undetected for about 2 hours before the OS even
sent a probe, then another 8 minutes before giving up.

davepacheco tracked this down in a real incident: a Nexus instance
hung a rack update for 45+ minutes holding two dead connections to a
CockroachDB node that had reset. The cause: keepalives_idle,
keepalives_interval, and keepalives_count were never set, so Nexus
just used the OS defaults.

This sets those four parameters in make_postgres_connector()
(nexus/db-queries/src/db/pool.rs), cutting worst-case detection time
to about 2 minutes:

    keepalives=1
    keepalives_idle=10
    keepalives_interval=10
    keepalives_count=12

The connection args list already existed
(previously just sslmode=disable) and already flows through
DieselPgConnector::to_url() into libpq.

Added two tests in nexus/db-queries/src/db/pool_connection.rs (had no
coverage before):

- to_url_includes_keepalive_args: checks the connection string has the
  new parameters.
- connection_actually_gets_tcp_keepalive_settings: opens a real
  connection and reads back the actual kernel TCP settings, the
  automated version of the mdb -k check davepacheco did by hand. Works
  on Linux, macOS, and illumos via socket2's keepalive accessors, which
  handle each OS's different option names. Only run on Linux so far.

Also re-ran the existing db::pool::test tests against a real
CockroachDB. Both still pass.

These timing values are a starting point, not final. Worth a second
look before merging.

Closes oxidecomputer#10668
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Nexus handoff hung on stuck database connections

1 participant