HDFS-17976. HDFS DataNode add configurable inactivity-based timeout for DataNode block transfers - #8718
Open
rdhabalia wants to merge 1 commit into
Open
Conversation
|
💔 -1 overall
This message was automatically generated. |
rdhabalia
force-pushed
the
datanode-block-transfer-timeout
branch
from
September 5, 2026 03:15
bfdcbbf to
37aafe7
Compare
…ansfers
Motivation
----------
A block written to a DataNode stays active until the client sends the final
packet and the block is finalized. When a client crashes, loses network
connectivity, or hangs before sending the last packet, the DataNode transfer
thread can remain blocked indefinitely waiting for input. Over time these
stalled transfers accumulate and cause transfer-thread starvation, resource
leakage, and degraded DataNode responsiveness. Today there is no built-in
inactivity detection to reclaim resources from such stalled transfers; the
system relies on client behavior or eventual lease recovery, which may occur
long after the actual failure.
Solution
--------
Introduce a configurable, inactivity-based timeout for block transfers on the
DataNode. When enabled, the DataNode tracks packet-arrival activity for each
ongoing block write using a monotonic clock. A single scheduled check per
transfer polls every timeout/2 and compares the time since the last received
packet against timeout/2:
* if a packet arrived within timeout/2, the stream is active and the check
is rescheduled (automatic reset for healthy streams);
* otherwise the transfer is treated as stalled and is aborted.
Polling every timeout/2 bounds the detection latency to between timeout/2 and
timeout after the client stops sending.
The abort runs on the shared scheduler thread, so it deliberately performs no
disk I/O: the on-disk streams (out/checksumOut) are written by the receive
thread WITHOUT holding the BlockReceiver monitor, so flushing them from the
scheduler thread would race those writes and could corrupt the replica.
Instead the abort only closes the client input stream, which unblocks the
receive thread's blocked socket read so it unwinds and runs its own
single-threaded cleanup. Data already acknowledged to the client is already
durable on disk, and the replica is left in the RBW (Replica Being Written)
state, so the NameNode's existing lease-recovery / block-synchronization path
finalizes the block without data loss.
The checks run on a shared ScheduledThreadPoolExecutor (2 daemon threads) that
is created lazily at DataNode startup only when the timeout is configured, so
the feature adds zero threads and zero overhead when disabled. It is configured
with remove-on-cancel so the per-block checks cancelled when a transfer
completes do not accumulate in the delay queue, and to not run delayed tasks
after shutdown. The last-packet timestamp is only updated when the timeout is
enabled to avoid unnecessary atomic writes on the hot path.
Because a stall is declared after no packet for timeout/2, the timeout must be
configured comfortably larger than the client socket read timeout (a healthy
idle hflush/hsync stream still sends heartbeat packets about every half the
socket timeout); the DataNode logs a warning if it is not.
Configuration
-------------
New key dfs.datanode.last.packet.receive.timeout.ms (default 0 = disabled).
A typical enabled value is 600000 (10 minutes). Fully backward compatible and
disabled by default.
Testing
-------
TestBlockReceiverLastPacketTimeout verifies the DataNode-side wiring: the shared
scheduler is created only when the timeout is enabled and is configured with the
correct lifecycle policies (remove-on-cancel and no delayed execution after
shutdown). TestBlockReceiverTransferTimeout drives MiniDFSCluster end-to-end and
asserts, deterministically, that a genuinely stuck client is aborted (single
DataNode with datanode replacement disabled so the broken pipeline cannot be
silently recovered), while healthy streams are never falsely aborted: full
transfers, a slow-but-steady writer, a writer whose gaps stay just under the
threshold (total time exceeding the timeout), the disabled configuration, and
multiple concurrent transfers all complete successfully.
|
💔 -1 overall
This message was automatically generated. |
rdhabalia
force-pushed
the
datanode-block-transfer-timeout
branch
from
September 5, 2026 07:44
37aafe7 to
d6d7451
Compare
|
🎊 +1 overall
This message was automatically generated. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of PR
A block written to a DataNode remains active until the client sends the final
packet and the block is finalized.
When a client crashes, loses network connectivity, or hangs before sending the
last packet, the DataNode transfer thread can remain blocked indefinitely while
waiting for input. Over time, these stalled transfers can accumulate and cause:
Today, there is no built-in inactivity detection to reclaim resources from
stalled transfers. The system relies on client behavior or eventual lease
recovery, which may occur long after the actual failure.
Solution
Introduce a configurable, inactivity-based timeout for block transfers on the
DataNode.
When enabled, the DataNode tracks packet-arrival activity for each ongoing
block write using a monotonic clock. A single scheduled check per transfer
runs every
timeout / 2and compares the time since the last received packetagainst
timeout / 2.Transfer Activity Detection
If a packet arrived within
timeout / 2:Otherwise:
Polling every
timeout / 2bounds the detection latency to betweentimeout / 2andtimeoutafter the client stops sending.Safe Transfer Abort
The abort runs on the shared scheduler thread and deliberately performs no
disk I/O.
The on-disk streams (
out/checksumOut) are written by the receive threadwithout holding the
BlockReceivermonitor. Flushing these streams fromthe scheduler thread could race with those writes and potentially corrupt the
replica.
Instead, the scheduler:
Data already acknowledged to the client is already durable on disk. The
replica remains in the RBW (Replica Being Written) state, allowing the
NameNode's existing lease-recovery / block-synchronization path to finalize
the block without data loss.
Scheduler Design
The timeout checks use a shared
ScheduledThreadPoolExecutorwith 2 daemonthreads.
The scheduler:
remove-on-cancelso per-block checks cancelled when a transfercompletes do not accumulate in the delay queue.
The last-packet timestamp is updated only when the timeout feature is enabled,
avoiding unnecessary atomic writes on the hot path when the feature is
disabled.
Client Socket Timeout Considerations
Because a stall is declared after no packet is received for
timeout / 2,the DataNode timeout must be configured comfortably larger than the client
socket read timeout.
A healthy idle
hflush/hsyncstream still sends heartbeat packets roughlyevery half of the socket timeout.
The DataNode logs a warning if the configured transfer timeout is not
sufficiently larger than the client socket timeout.
Configuration
dfs.datanode.last.packet.receive.timeout.ms00disables the feature.A typical enabled value is:
This corresponds to a 10-minute timeout.
The change is fully backward compatible and disabled by default.
Testing
TestBlockReceiverLastPacketTimeoutVerifies the DataNode-side scheduler wiring:
remove-on-cancelis configured correctly.TestBlockReceiverTransferTimeoutRuns an end-to-end test using
MiniDFSClusterand verifies that a genuinelystuck client is deterministically aborted.
The test uses a single DataNode with DataNode replacement disabled so that a
broken pipeline cannot be silently recovered.
It also verifies that healthy streams are never falsely aborted, including:
when the total transfer time exceeds the timeout
How was this patch tested?
Tested by newly added unit test
For code changes:
declared according to the connector-specific documentation? Note: Automated CI
testing doesn't cover all cases so manual testing with cloud storage is still
required.
LICENSE,LICENSE-binary,NOTICE-binaryfiles?AI Tooling
If an AI tool was used:
where is the name of the AI tool used.
https://www.apache.org/legal/generative-tooling.html