Skip to content

HDDS-16145. Fix race between DeletedBlockLogStateManager transaction removal and SCM HA buffer flush - #11188

Open
priyeshkaratha wants to merge 1 commit into
apache:masterfrom
priyeshkaratha:HDDS-16145
Open

HDDS-16145. Fix race between DeletedBlockLogStateManager transaction removal and SCM HA buffer flush#11188
priyeshkaratha wants to merge 1 commit into
apache:masterfrom
priyeshkaratha:HDDS-16145

Conversation

@priyeshkaratha

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

DeletedBlockLogStateManagerImpl#removeTransactionsFromDB marks a batch of transaction IDs as hidden (deletingTxIDs), removes each one from the SCM HA transaction buffer, and writes an updated summary — as three
separate calls, each independently protected only for its own duration by SCMHADBTransactionBuffer's internal read lock.

A concurrent flush() (holding the buffer's write lock, e.g. triggered by the periodic flush monitor, a Ratis checkpoint download, or a leader transfer) can land in the gap between these calls. If it lands after a
txID has been marked hidden but before it has actually been removed from the buffer, flush() calls onFlush(), which resets deletingTxIDs — while the transaction's row is still durably present in the deleted-blocks table. The row is then re-exposed to getReadOnlyIterator() (the deletion scanner) even though its removal is still in progress, which can cause the transaction to be processed again and its summary count to be double-decremented.

This PR adds lock()/unlock() to SCMHADBTransactionBuffer (backed by the buffer's existing ReentrantReadWriteLock read lock) and has removeTransactionsFromDB acquire it across the entire mark-remove-
summary sequence. Since flush() takes the write lock, it cannot proceed until the whole sequence has released the read lock, so it can only observe the sequence fully completed or not yet started — never midway through.

What is the link to the Apache JIRA

HDDS-16145

How was this patch tested?

Added testcases

@priyeshkaratha
priyeshkaratha marked this pull request as ready for review September 2, 2026 08:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant