Skip to content

[python] Reuse gathered window values across aggregations - #9804

Merged
JingsongLi merged 1 commit into
apache:masterfrom
QuakeWang:window-gather
Sep 15, 2026
Merged

JingsongLi merged 1 commit into
apache:masterfrom
QuakeWang:window-gather

Conversation

@QuakeWang

Copy link
Copy Markdown
Member

Purpose

Window joins already deduplicate row IDs before fetching payloads, but each aggregation rebuilds Arrow indices and gathers the same source column again.

Gather each source column once per window and reuse it across aggregations. Release gathered values after processing that window and column, preserving output order, empty/null behavior, output types, and mean precision and overflow handling.

Tests

  • Regression test verifies reduced take calls, interleaved outputs, overlapping windows, and empty/null behavior.
  • All 67 multimodal temporal tests passed.
  • Flake8, license headers, and git diff --check passed.

Gather each source column once per window and reuse it across aggregations, avoiding repeated Arrow takes and index allocations. Preserve aggregation semantics and output order without caching gathered windows.

Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com>

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requirement fit: SUPPORTED. Implementation: CLEAN.

Reviewed 3de2fe3742d4. A window with several aggregations over one source column currently repeats the same Arrow gather. Sharing that gathered selection has a direct path through the public window join and a small implementation scope. Output ordering, null/empty windows and mean precision behavior remain intact.

Validation: all 67 multimodal temporal tests and 48 subtests passed. Current head CI is green. This confirms behavior and gather reuse; it does not quantify end-to-end latency savings.

No actionable implementation regression found in this review.

@JingsongLi
JingsongLi merged commit 763dc4a into apache:master Sep 15, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants