Skip to content

Merge queue times out PRs whose checks are still running, so merging takes many manual attempts and CI is re-run from scratch #26292

Open

Description

@alamb

Describe the bug

The merge queue regularly removes PRs with "removed this pull request from the merge queue due to no response for status checks" even though no check failed.

For example, #26244 is a 2-line change. It was approved on 2026-10-12 at 18:12 UTC. Since then it has been added to the merge queue 7 times by 4 different committers and removed 6 times.

Image

In every case I looked at, checks had already passed and the rest had simply not finished (or started) within the hour. When the PR is added back to the queue, all of the CI runs again from the beginning, and often times out again.

This has been happening on and off for a while (see #23400 from September), but over the last two days (2026-10-13 and 2026-10-14) it affected most PRs that went through the queue.

Problems this causes

1. Merging an approved PR takes a long time and several manual steps

See above. Other recent examples of PRs that timed out in the queue (all eventually passed CI unchanged):

PR Queue attempts Time from first queue to merge People who had to re-queue it
#26251 5 19.6 h 2
#26253 4 19.6 h 2
#25494 3 20.6 h 2
#26132 2 17.3 h 2
#26215 2 11.4 h 2
#25891 2 7.0 h 1
#26110 4 so far still open (first queued 10-09) 1
#25710 2 so far still open 1
#26092, #26096, #26224, #26164, #26010, #24093 1-2 so far each timed out at least once on 10-13 / 10-14

2. It burns a lot of CI capacity, because the tests do eventually finish but have to run again

Each timeout throws away CI work that already completed.

For each timeout, 22 jobs (65 to 88 runner-minutes of work) had already passed when the PR was evicted. When the PR is re-queued, those 22 jobs run again along with everything else.

For the 19 timeouts I checked on 10-13 / 10-14 alone, that is roughly 23 runner-hours of passing CI discarded in about 36 hours. We are already a top ASF consumer of GitHub Actions minutes:

Even worse is that these timeouts also happen precisely when runners are scarce, so the re-runs add load at the worst possible time and make the next timeout more likely.

Expected behavior

A PR whose checks are all passing, just slowly, should merge without anyone having to touch it again. CI work that has already passed for a queue entry should not be discarded and redone.

Additional context

Activity

  1. added
    bugSomething isn't working
    development-processRelated to development process of DataFusion
    on Oct 14, 2026
  2. blaginin commented on Oct 14, 2026

    @blaginin
    Member

    https://github.com/apache/datafusion/actions/runs/38528268442/job/115733699502 - we were waiting 87 minutes for this run to start - even though the action itself took just 10 s 🤯

  3. comphead commented on Oct 14, 2026

    @comphead
    Contributor

    I'm wondering if MQ is shared, because the similar issue has been reported in iceberg-rs repo yesterday apache/iceberg-rust#3425

  4. blaginin commented on Oct 14, 2026

    @blaginin
    Member

    i think it's not about MQ, but instead of ASF runners being overloaded...

  5. comphead commented on Oct 14, 2026

    @comphead
    Contributor

    I feel we need to reduce MQ throughput for DF, Comet is already 5 max builds

            max_entries_to_merge: 5     
            max_entries_to_build: 5
    

    Its already done in #26293

  6. alamb commented on Oct 14, 2026

    @alamb
    ContributorAuthor

    I feel we need to reduce MQ throughput for DF, Comet is already 5 max builds

    Maybe we could in theory also reduce the number of concurrent builds to reduce the load on the runners 🤔 (aka down to 3 or something). but that would in theory reduce the throughput and not reduce the overall demand)

  7. alamb commented on Oct 14, 2026

    @alamb
    ContributorAuthor

    4. I'm wondering if MQ is shared, because the similar issue has been reported in iceberg-rs repo yesterday apache/iceberg-rust#3425

    @comphead I think the issue is that the github runners are shared across apache projects somehow -- so I think we are suffering from long queue wait times as the runner pool is at max capacity

    3. https://github.com/apache/datafusion/actions/runs/38528268442/job/115733699502 - we were waiting 87 minutes for this run to start - even though the action itself took just 10 s 🤯

    Yeah, @blaginin and even worse is if that then kicks the PR out of the merge queue, we have to re-run all CI again!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingdevelopment-processRelated to development process of DataFusion

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions