acc: run ssh/connect-serverless-gpu locally#5878
Merged
Merged
Conversation
Flip acceptance/ssh/connect-serverless-gpu to Local = true so it runs against the in-process testserver (it was disabled on both cloud and local in #4838). A full SSH handshake can't complete locally (no real compute/sshd + a wss driver-proxy), so the local run drives the connect flow up to the connection attempt and asserts the serverless GPU bootstrap job the CLI submits (accelerator, environment, secret scope). Teach libs/testserver enough of the SSH tunnel backend for the flow to get there deterministically and fast: - secrets/list (ListSecretsByScope), returning RESOURCE_DOES_NOT_EXIST for a missing scope so CreateKeysSecretScope creates it - synthesize metadata.json on the ssh-server-bootstrap run submit, as a real tunnel server would publish it, so getServerMetadata succeeds - driver-proxy metadata/logs endpoints - workspace/export now 404s a missing object instead of returning a nil body (which the response normalizer rejects) The connect attempt itself is expected to fail locally; its noisy output goes to LOG.stderr (excluded from the golden files).
Collaborator
Integration test reportCommit: 102a096
10 interesting tests: 4 SKIP, 3 flaky, 3 RECOVERED
Top 9 slowest tests (at least 2 minutes):
|
Add a proxy-level test that runs a real OpenSSH server (sshd -i) behind the websocket proxy and a real SSH client through RunClientProxy, exercising the actual handshake, pubkey auth, and remote command exec end-to-end without any Databricks compute. This complements the existing `cat`-based transport tests by validating that the tunnel carries the SSH protocol faithfully, giving us local coverage of the connectivity path that otherwise needs a cluster. The test is skipped when sshd (openssh-server) is not installed and is gated to non-Windows, matching the existing proxy tests.
The runner images ship only openssh-client, so the real SSH handshake test would silently skip on Linux (no sshd) and is unreliable to run as a non-root user on macOS. Restrict it to Linux and install openssh-server in the test-exp-ssh job so it runs deterministically there; it still skips gracefully when sshd is absent (e.g. the general test job, which doesn't install it).
anton-107
approved these changes
Jul 13, 2026
anton-107
left a comment
anton-107
left a comment
Contributor
There was a problem hiding this comment.
LGTM, but let's try and converge these tests in two different styles into one (acceptance) in one of the follow-up PRs
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
Local test coverage for
databricks ssh connect, which was disabled on cloud and local in #4838, across the two layers we can exercise without compute:acceptance/ssh/connect-serverless-gpuruns against the in-process testserver and asserts the serverless-GPU job the CLI submits (GPU_1xA10, serverless env, secret scope, notebook task). This needs newlibs/testserverstubs:secrets/list→RESOURCE_DOES_NOT_EXISTfor a missing scope, a synthesizedmetadata.jsononssh-server-bootstrap-*submit, driver-proxymetadata/logs, andworkspace/export→404for a missing object.client_server_sshd_test.gostands up a realsshdbehindproxy.NewProxyServerand drives a real SSH client (x/crypto/ssh) throughRunClientProxy. Thetest-exp-sshjob installsopenssh-serveron Linux to support it.Why
ssh connectcurrently has no automated coverage. These tests exercise the two layers reachable locally: the control-plane setup (the bootstrap job the CLI submits) and the tunnel transport carrying the real SSH protocol.A full local handshake via the connect command can't complete (no compute, a
wss://driver-proxy dial, realsshspawn), so the acceptance test stops at the connection attempt — failure output goes toLOG.stderr(excluded from goldens), and the cloudConnection successfulpath is preserved for when cloud is re-enabled. The handshake test is Linux-only (//go:build linux): runningsshd -ias non-root is reliable there but not on macOS; it skips whensshdis absent. The end-to-end path through the real driver proxy (bootstrap notebook →databricks ssh serveron compute) still needs an integration test and is out of scope.Tests
acceptance/ssh/connect-serverless-gpu(asserts the submitted bootstrap job payload)TestClientServerRealSSHDinexperimental/ssh/internal/proxy(real SSH handshake, pubkey auth, and remote command over the proxy tunnel), running on Linux in thetest-exp-sshjob