HDDS-16145. Fix race between DeletedBlockLogStateManager transaction removal and SCM HA buffer flush - #11188
Open
priyeshkaratha wants to merge 1 commit into
Open
HDDS-16145. Fix race between DeletedBlockLogStateManager transaction removal and SCM HA buffer flush#11188priyeshkaratha wants to merge 1 commit into
priyeshkaratha wants to merge 1 commit into
Conversation
…removal and SCM HA buffer flush
priyeshkaratha
marked this pull request as ready for review
September 2, 2026 08:29
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
DeletedBlockLogStateManagerImpl#removeTransactionsFromDBmarks a batch of transaction IDs as hidden (deletingTxIDs), removes each one from the SCM HA transaction buffer, and writes an updated summary — as threeseparate calls, each independently protected only for its own duration by
SCMHADBTransactionBuffer's internal read lock.A concurrent
flush()(holding the buffer's write lock, e.g. triggered by the periodic flush monitor, a Ratis checkpoint download, or a leader transfer) can land in the gap between these calls. If it lands after atxID has been marked hidden but before it has actually been removed from the buffer,
flush()callsonFlush(), which resetsdeletingTxIDs— while the transaction's row is still durably present in the deleted-blocks table. The row is then re-exposed togetReadOnlyIterator()(the deletion scanner) even though its removal is still in progress, which can cause the transaction to be processed again and its summary count to be double-decremented.This PR adds
lock()/unlock()toSCMHADBTransactionBuffer(backed by the buffer's existingReentrantReadWriteLockread lock) and hasremoveTransactionsFromDBacquire it across the entire mark-remove-summary sequence. Since
flush()takes the write lock, it cannot proceed until the whole sequence has released the read lock, so it can only observe the sequence fully completed or not yet started — never midway through.What is the link to the Apache JIRA
HDDS-16145
How was this patch tested?
Added testcases