[SPARK-41246][CORE] Parse RDD block ids whose RDD id overflowed to negative - #57893
Open
giriprakash wants to merge 4 commits into
Open
[SPARK-41246][CORE] Parse RDD block ids whose RDD id overflowed to negative#57893giriprakash wants to merge 4 commits into
giriprakash wants to merge 4 commits into
Conversation
…gative RDD ids are handed out by a 32-bit counter that never resets while the application runs, so an application creating more than Int.MaxValue RDDs wraps around to negative ids. RDDBlockId.name already renders those with a leading minus sign, but BlockId.apply only accepted digits, so it could not parse its own output and every path reconstructing a BlockId from its name (such as UpdateBlockInfo.readExternal) failed with UnrecognizedBlockId until the application was restarted. Allow an optional minus sign on the RDD id group so the name round-trips. The split index stays non-negative and no other block id pattern changes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Allow
BlockId.applyto parse RDD block names whose RDD id is negative, e.g.rdd_-1330910599_36.RDDBlockId.namealready renders negative ids with a leading minus sign. The only change is to make the RDD id regex accept an optional minus so the name round-trips throughBlockId.apply. The split index stays non-negative, and no other block id pattern is changed.Why are the changes needed?
SparkContexthands out RDD ids from a 32-bitAtomicIntegerthat never resets while the application runs. AfterInt.MaxValueRDDs, the counter wraps to negative values. Cached blocks then get names such asrdd_-1330910599_36. Paths that reconstruct aBlockIdfrom its name — includingUpdateBlockInfo.readExternal— failed withUnrecognizedBlockIdbecause the parser only accepted digits. The application stayed up, but queries kept failing until restart.This matches the failure mode described in SPARK-41246.
Does this PR introduce any user-facing change?
Yes. Applications that previously failed with
UnrecognizedBlockIdafter the RDD id counter overflowed can now parse and use those block ids. Behavior for all previously valid (non-negative) RDD block ids is unchanged.How was this patch tested?
Added a unit test in
BlockIdSuitethat:rdd_-1330910599_36rdd_-2147483648_0(Int.MinValue)Please run:
Was this patch authored or co-authored using generative AI tooling?
Generated-by: Cursor Agent