fix(stream): support upstream client certificate (mTLS) in L4 proxy#13596
Draft
AlinsRan wants to merge 4 commits into
Draft
fix(stream): support upstream client certificate (mTLS) in L4 proxy#13596AlinsRan wants to merge 4 commits into
AlinsRan wants to merge 4 commits into
Conversation
The stream (L4) subsystem could not present a client certificate when proxying to a TLS upstream, unlike the http subsystem. The http path injects the client cert via the apisix-nginx-module C API (set_cert_and_key), which has no stream counterpart. Instead, wire the native nginx stream proxy_ssl_certificate / proxy_ssl_certificate_key directives with variables, filled in the preread phase with the upstream tls.client_cert/client_key (or the ssl object referenced by tls.client_cert_id) using the inline data: PEM scheme. An empty value means no client certificate is presented. Fixes apache#12472
The stream upstream-mtls test referenced the upstream server certs via `../t/certs/...`, which resolves to `t/servroot/conf/../t/certs/...` (a non-existent path) and made nginx fail to start. Use the same `../../certs/...` prefix as other .t tests (e.g. healthcheck-https.t), which resolves to the repo `t/certs/` directory. The new `proxy_ssl_certificate` directive in every stream server block made the per-port PROXY protocol cli test mis-match: its plain-vs-TLS `grep -E "ssl_certificate "` matched the substring inside `proxy_ssl_certificate `. Anchor the grep on a leading whitespace boundary so it only matches the downstream `ssl_certificate` directive.
The stream mTLS path fed `tls.client_key` (or the ssl object key) directly into the `data:` proxy_ssl_certificate_key variable. That key is stored AES-encrypted at rest, so nginx received base64 ciphertext instead of PEM and failed with "cannot load certificate key". Decrypt it with aes_decrypt_pkey (a no-op for plaintext PEM) before building the data: value, matching the http path which decrypts via fetch_pkey. Also correct the no-client-cert test assertion: an mTLS upstream that rejects a certless handshake logs "client sent no required SSL certificate", not "upstream SSL certificate verify error" (which only applies to proxy_ssl_verify server-cert checks).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Fixes #12472
The stream (L4 TCP/TLS) subsystem could not present a client certificate (mTLS) when APISIX proxies to a TLS upstream, unlike the http subsystem which honors upstream
tls.client_cert/client_key/client_cert_id. This was confirmed by a maintainer in the issue:apisix/upstream.luaset the stream upstream TLS (SNI/enable) but never applied the client certificate.Root cause
The http subsystem injects the client cert per-request through the apisix-nginx-module C API
ngx_http_apisix_upstream_set_cert_and_key(applied viaSSL_use_certificateduring the upstream handshake). The stream subsystem only exposesngx_stream_apisix_upstream_enable_tls(a boolean flag) — there is no stream counterpart that sets the client cert, so the http mechanism cannot be reused as-is.Approach (self-contained, no nginx/openresty-C changes)
Use the native nginx stream
proxy_ssl_certificate/proxy_ssl_certificate_keydirectives, which support variables and inline PEM via thedata:scheme (nginx >= 1.21.4, satisfied by APISIX-Runtime):apisix/cli/ngx_tpl.lua: declare$upstream_mtls_cert/$upstream_mtls_key(empty by default) and addproxy_ssl_certificate/proxy_ssl_certificate_keyin the stream server block (guarded byuse_apisix_base). An empty value means no client certificate is presented.apisix/upstream.lua: in the streamscheme == "tls"branch, fill those vars fromup_conf.tls.client_cert/client_key(inline) or from the ssl object referenced bytls.client_cert_id, using thedata:inline PEM form (no temp files).apisix/init.lua: extract theclient_cert_id->api_ctx.upstream_sslresolution into a sharedresolve_upstream_client_certhelper and call it fromstream_preread_phasetoo (previously it ran only on the http path).t/stream-node/upstream-mtls.t(inline cert success/failure +client_cert_id).docs/en/latest/mtls.md.No schema change needed —
tls.client_cert/client_key/client_cert_idalready exist on the upstream schema.