Skip to content

[#2790] fix(server): Return NO_REGISTER when shuffle metadata is missing - #2791

Open
ConeyLiu wants to merge 3 commits into
apache:masterfrom
ConeyLiu:fix/missing-shuffle-result-no-register
Open

ConeyLiu wants to merge 3 commits into
apache:masterfrom
ConeyLiu:fix/missing-shuffle-result-no-register

Conversation

@ConeyLiu

Copy link
Copy Markdown

What changes were proposed in this pull request?

Return NO_REGISTER instead of a successful empty bitmap when shuffle registration metadata is missing.

Also prevent shuffle result queries from recreating an expired application.

Why are the changes needed?

After an application is removed, a task retry may query an old shuffle and incorrectly receive an empty bitmap with SUCCESS, potentially causing data loss.

Fix: #2790

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Added tests for expired applications, missing shuffles, and valid empty shuffle results.

@ConeyLiu

Copy link
Copy Markdown
Author

@roryqi @zuston @wForget pls help to review this when you have a time.

@github-actions

github-actions Bot commented Sep 18, 2026

Copy link
Copy Markdown

Test Results

 3 791 files   -  8   3 791 suites   - 8   7h 32m 53s ⏱️ - 17m 31s
 1 266 tests ± 0   1 254 ✅ ± 0  11 💤 ±0  0 ❌ ±0  1 🔥 ±0 
19 080 runs   - 12  19 030 ✅  - 13  48 💤 ±0  0 ❌ ±0  2 🔥 +1 

For more details on these errors, see this check.

Results for commit 888147d. ± Comparison against base commit b9b5d6d.

♻️ This comment has been updated with latest results.

@zuston zuston left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice catch! overall lgtm to me, left minor comment. BTW, please fix the relevant failure CI tests. @ConeyLiu

Please take a look @xianjingfeng .

public byte[] getFinishedBlockIds(
String appId, Integer shuffleId, Set<Integer> partitions, BlockIdLayout blockIdLayout)
throws IOException {
refreshAppId(appId);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this logic of refreshAppId is missing?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for pointing this out. refreshAppId was removed intentionally because it may recreate an expired application via computeIfAbsent.

However, the timestamp refresh should be preserved for valid requests. I have added taskInfo.setCurrentTimes(...) after validating the application, shuffle, and partitions, so valid queries still refresh the application while invalid queries cannot recreate or keep it alive.

@zuston
zuston requested a review from xianjingfeng September 18, 2026 03:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Missing shuffle metadata may be returned as a successful empty bitmap

2 participants