Conversation
Test Results 3 791 files - 8 3 791 suites - 8 7h 32m 53s ⏱️ - 17m 31s For more details on these errors, see this check. Results for commit 888147d. ± Comparison against base commit b9b5d6d. ♻️ This comment has been updated with latest results. |
There was a problem hiding this comment.
nice catch! overall lgtm to me, left minor comment. BTW, please fix the relevant failure CI tests. @ConeyLiu
Please take a look @xianjingfeng .
| public byte[] getFinishedBlockIds( | ||
| String appId, Integer shuffleId, Set<Integer> partitions, BlockIdLayout blockIdLayout) | ||
| throws IOException { | ||
| refreshAppId(appId); |
There was a problem hiding this comment.
this logic of refreshAppId is missing?
There was a problem hiding this comment.
Thanks for pointing this out. refreshAppId was removed intentionally because it may recreate an expired application via computeIfAbsent.
However, the timestamp refresh should be preserved for valid requests. I have added taskInfo.setCurrentTimes(...) after validating the application, shuffle, and partitions, so valid queries still refresh the application while invalid queries cannot recreate or keep it alive.
What changes were proposed in this pull request?
Return
NO_REGISTERinstead of a successful empty bitmap when shuffle registration metadata is missing.Also prevent shuffle result queries from recreating an expired application.
Why are the changes needed?
After an application is removed, a task retry may query an old shuffle and incorrectly receive an empty bitmap with
SUCCESS, potentially causing data loss.Fix: #2790
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Added tests for expired applications, missing shuffles, and valid empty shuffle results.