Search before asking
Motivation
For tables with bucket keys, BUCKET mode provides good locality and ordering guarantees, but
it can suffer from severe data skew when the number of buckets is not evenly divisible by the
sink parallelism, or when some buckets are hotter than others. NONE mode avoids skew but
breaks merge-engine semantics because records with the same key may be processed by different
subtasks.
Unlike the existing BUCKET mode (which maps each bucket to exactly one subtask and may leave
subtasks idle), BUCKET_LOAD_BALANCE distributes records by bucket key using an LCM-based
logical slot assignment so that every subtask receives traffic while still guaranteeing that
records with the same bucket key route to the same subtask.
Solution
- Records with the same bucket key always route to the same subtask.
- All subtasks receive data when
numBuckets and numChannels are not evenly divisible.
- Intra-bucket ordering is not preserved (records with different bucket keys in the same
bucket may be processed by different subtasks).
Anything else?
No response
Willingness to contribute
Search before asking
Motivation
For tables with bucket keys,
BUCKETmode provides good locality and ordering guarantees, butit can suffer from severe data skew when the number of buckets is not evenly divisible by the
sink parallelism, or when some buckets are hotter than others.
NONEmode avoids skew butbreaks merge-engine semantics because records with the same key may be processed by different
subtasks.
Unlike the existing
BUCKETmode (which maps each bucket to exactly one subtask and may leavesubtasks idle),
BUCKET_LOAD_BALANCEdistributes records by bucket key using an LCM-basedlogical slot assignment so that every subtask receives traffic while still guaranteeing that
records with the same bucket key route to the same subtask.
Solution
numBucketsandnumChannelsare not evenly divisible.bucket may be processed by different subtasks).
Anything else?
No response
Willingness to contribute