Skip to content

[client] Fix unbounded growth of stale partition entries in writer metadata cache#3367

Open
lilei1128 wants to merge 1 commit into
apache:mainfrom
lilei1128:fix-clean
Open

[client] Fix unbounded growth of stale partition entries in writer metadata cache#3367
lilei1128 wants to merge 1 commit into
apache:mainfrom
lilei1128:fix-clean

Conversation

@lilei1128
Copy link
Copy Markdown

Purpose

Linked issue: close #xxx

When partitions are continuously created and dropped, physical table path
entries in RecordAccumulator.writeBatches and Cluster.partitionsIdByPath
were never removed, causing unbounded memory growth and CPU waste from
iterating over stale entries in ready() and drain() hot loops.

Brief change log

Fix 1 (RecordAccumulator): introduce markPathsAsStale() and
removeStalePathIfEmpty() with a dedicated staleLock so that the
check-then-remove is atomic with respect to concurrent append() calls.
New appends to stale paths are rejected immediately.

Fix 2 (MetadataUtils): during partial metadata updates, remove stale
partition entries for any table whose partition list was refreshed, so
that dropped partitions no longer linger in partitionsIdByPath.

Fix 3 (Sender): call cleanupStaleWriteBatches() on every sendWriteData()
cycle to mark paths absent from the cluster as stale and remove them once
their deques are fully drained.

Tests

  • RecordAccumulatorTest.testMarkAndRemoveStalePathAfterDrain: verifies
    that append() is rejected after markPathsAsStale(), and that
    removeStalePathIfEmpty() succeeds once the deque is drained.
  • RecordAccumulatorTest.testRemoveStalePathReturnsFalseWhenDequeNonEmpty:
    verifies that removeStalePathIfEmpty() returns false and preserves the
    path when pending batches remain.
  • SenderTest.testSenderCleansUpStaleWriteBatchesAfterMetadataUpdate:
    verifies end-to-end that runOnce() removes a stale path from
    writeBatches after the cluster no longer contains it and its deque
    has been drained.

API and Format

Documentation

@lilei1128
Copy link
Copy Markdown
Author

close #3362

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant