Coerce explicit None _request_timeout in async/sync k8s API clients#69611
Open
ZhaoMJ wants to merge 2 commits into
Open
Coerce explicit None _request_timeout in async/sync k8s API clients#69611ZhaoMJ wants to merge 2 commits into
ZhaoMJ wants to merge 2 commits into
Conversation
The _TimeoutAsyncK8sApiClient and _TimeoutK8sApiClient wrappers attempt
to enforce a client-side read timeout via kwargs.setdefault(). However,
every generated kubernetes / kubernetes_asyncio API method (e.g.
read_namespaced_pod) always forwards _request_timeout=None explicitly:
_request_timeout=local_var_params.get('_request_timeout') # → None
setdefault() is a no-op when the key already exists, even as None. So
None reaches aiohttp, which creates ClientTimeout() with no total or
sock_read limit — an infinite socket timeout.
On a half-open TCP connection (what a 429 Too Many Requests or an
LB-side "500 context canceled" leaves behind when the LB drops the
socket without FIN/RST), the awaited API call blocks forever. Because
no exception is raised, generic_api_retry never fires and a deferrable
KubernetesPodOperator trigger hangs indefinitely with no error visible
in the task log. Only a triggerer restart clears it.
Fix: replace setdefault with an explicit None-check so that a None
value (from the generated client) is correctly treated as "not set",
while any intentionally-set value — including the (1800, 300) tuple
passed by the async log-streaming path — is preserved.
Confirmed by live introspection in production (kubernetes_asyncio 35.0.1,
aiohttp 3.13.2, apache-airflow-providers-cncf-kubernetes 10.17.1):
before the fix, _request_timeout=None was forwarded to aiohttp regardless
of the wrapper; after the fix, it correctly receives 60.
Ref: https://github.com/kubernetes-client/python/blob/master/examples/watch/timeout-settings.md
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The
_TimeoutAsyncK8sApiClientand_TimeoutK8sApiClientwrappers were introduced to enforce a 60s client-side read timeout on all Kubernetes API calls. However, the timeout is silently dropped, leaving an infinite socket read timeout that causes deferrableKubernetesPodOperatortriggers to hang indefinitely.Root cause
Every generated
kubernetes/kubernetes_asyncioAPI method (e.g.read_namespaced_pod) always forwards_request_timeoutexplicitly asNonewhen the caller doesn't set it:The wrapper tried to inject the default using
kwargs.setdefault("_request_timeout", ...). Butsetdefaultis a no-op when the key already exists, even asNone. SoNonereaches aiohttp, which createsClientTimeout()with nototalorsock_readlimit — an infinite socket timeout.On a half-open TCP connection (what a 429 Too Many Requests or an LB-side "500 context canceled" leaves behind when the load balancer drops the socket without FIN/RST), the awaited API call blocks forever. Because no exception is raised,
generic_api_retrynever fires and a deferrable KPO trigger hangs indefinitely with no error visible in the task log. Only a triggerer restart clears it.This is confirmed by the official kubernetes Python client documentation:
Ref: https://github.com/kubernetes-client/python/blob/master/examples/watch/timeout-settings.md
Fix
Replace
setdefaultwith an explicitNone-check, so aNonevalue (from the generated client) is treated as "not set" and receives the 60s default, while any intentionally-set value is preserved — including the(1800, 300)tuple passed by the async log-streaming path, which must not be clobbered.Verification
Confirmed by live introspection in a production triggerer pod (kubernetes_asyncio==35.0.1, aiohttp==3.13.2, apache-airflow-providers-cncf-kubernetes==10.17.1):
All cases validated:
None(generated client path) → 60s ✓(1800, 300)tuple (log streaming) → preserved ✓timeout_secondsscaling → intact ✓Tests
Added two new parametrize cases to both
TestTimeoutK8sApiClientandTestTimeoutAsyncK8sApiClient:explicit-none-timeout: the actual failure mode (generated client always passesNone)explicit-tuple-preserved: the log-streaming invariant ((1800, 300) must not be overwritten)The existing
default-timeoutcase ({}) only covers the key-absent path, not the key-present-as-None path that the generated client always exercises.Checklist
🤖 Generated with Claude Code