Skip to content

Introduce a new sink distribution mode BUCKET_LOAD_BALANCE for the Flink connector. #4449

Description

@loserwang1024

Search before asking

  • I searched in the issues and found nothing similar.

Motivation

For tables with bucket keys, BUCKET mode provides good locality and ordering guarantees, but
it can suffer from severe data skew when the number of buckets is not evenly divisible by the
sink parallelism, or when some buckets are hotter than others. NONE mode avoids skew but
breaks merge-engine semantics because records with the same key may be processed by different
subtasks.

Unlike the existing BUCKET mode (which maps each bucket to exactly one subtask and may leave
subtasks idle), BUCKET_LOAD_BALANCE distributes records by bucket key using an LCM-based
logical slot assignment so that every subtask receives traffic while still guaranteeing that
records with the same bucket key route to the same subtask.

Solution

  • Records with the same bucket key always route to the same subtask.
  • All subtasks receive data when numBuckets and numChannels are not evenly divisible.
  • Intra-bucket ordering is not preserved (records with different bucket keys in the same
    bucket may be processed by different subtasks).

Anything else?

No response

Willingness to contribute

  • I'm willing to submit a PR!

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions