[SPARK-41246][CORE] Fail fast on RDD id overflow; parse negative block ids as safety net - #57893
Open
giriprakash wants to merge 3 commits into
Open
[SPARK-41246][CORE] Fail fast on RDD id overflow; parse negative block ids as safety net#57893giriprakash wants to merge 3 commits into
giriprakash wants to merge 3 commits into
Conversation
uros-b
approved these changes
Aug 10, 2026
Member
|
Thank you @giriprakash! |
…gative RDD ids are handed out by a 32-bit counter that never resets while the application runs, so an application creating more than Int.MaxValue RDDs wraps around to negative ids. RDDBlockId.name already renders those with a leading minus sign, but BlockId.apply only accepted digits, so it could not parse its own output and every path reconstructing a BlockId from its name (such as UpdateBlockInfo.readExternal) failed with UnrecognizedBlockId until the application was restarted. Allow an optional minus sign on the RDD id group so the name round-trips. The split index stays non-negative and no other block id pattern changes.
giriprakash
force-pushed
the
SPARK-41246-negative-rdd-block-id
branch
from
August 10, 2026 17:17
af0eb74 to
d833b81
Compare
Prevent silent AtomicInteger wrap past Int.MaxValue by throwing from newRddId before a negative id is minted. Keep BlockId's optional-minus RDD regex as a safety net for pre-upgrade cached blocks and tests.
giriprakash
changed the title
cloud-fan
approved these changes
Aug 11, 2026
cloud-fan
left a comment
cloud-fan
left a comment
Contributor
There was a problem hiding this comment.
0 blocking, 0 non-blocking, 1 nit.
The overflow and compatibility behavior is sound; only one comment typo needs cleanup.
Nits: 1 minor item (see inline comments).
Verification
Traced RDD construction through RDD.id to SparkContext.newRddId: AtomicInteger allocation remains unique through Int.MaxValue, and every negative return path throws before exposing an id. Also traced RDDBlockId.name through BlockId's signed RDD regex and Int conversion, confirming the full signed Int name representation round-trips.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
SparkContext.newRddIdthrows if the 32-bit counter would wrap to a negative value afterInt.MaxValue, so long-running apps no longer mint negative RDD ids that break BlockManager / other id-keyed state.BlockIdRDD regex accepts an optional minus (rdd_(-?[0-9]+)_...) so names likerdd_-1330910599_36from pre-upgrade caches or tests still parse instead of throwingUnrecognizedBlockId.Under fail-fast, newly created RDDs never get negative ids; the parser change is defense in depth.
Why are the changes needed?
After more than
Int.MaxValueRDDs,AtomicIntegerwraps to negatives. Cached partitions then use names such asrdd_-1330910599_36, andBlockId.applyfailed withUnrecognizedBlockId(e.g. onUpdateBlockInfo). Parsing negatives alone would let the app continue with wrapped ids and cover up overflow elsewhere. Fail-fast makes overflow an explicit error; the regex remains for in-flight / old negative names.Does this PR introduce any user-facing change?
Yes. Creating an RDD after
Int.MaxValuethrows a clearSparkExceptioninstead of wrapping. Restart the application to reset the counter. Positive block-id parsing is unchanged. Javaint id = rdd.id()still compiles.How was this patch tested?
BlockIdSuite: negative RDD block name round-trip (safety net)SparkContextSuite: fail-fast when counter is atInt.MaxValueWas this patch authored or co-authored using generative AI tooling?
Generated-by: Cursor Agent