You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Merge queue times out PRs whose checks are still running, so merging takes many manual attempts and CI is re-run from scratch #26292
The merge queue regularly removes PRs with "removed this pull request from the merge queue due to no response for status checks" even though no check failed.
For example, #26244 is a 2-line change. It was approved on 2026-10-12 at 18:12 UTC. Since then it has been added to the merge queue 7 times by 4 different committers and removed 6 times.
In every case I looked at, checks had already passed and the rest had simply not finished (or started) within the hour. When the PR is added back to the queue, all of the CI runs again from the beginning, and often times out again.
This has been happening on and off for a while (see #23400 from September), but over the last two days (2026-10-13 and 2026-10-14) it affected most PRs that went through the queue.
Problems this causes
1. Merging an approved PR takes a long time and several manual steps
See above. Other recent examples of PRs that timed out in the queue (all eventually passed CI unchanged):
2. It burns a lot of CI capacity, because the tests do eventually finish but have to run again
Each timeout throws away CI work that already completed.
For each timeout, 22 jobs (65 to 88 runner-minutes of work) had already passed when the PR was evicted. When the PR is re-queued, those 22 jobs run again along with everything else.
For the 19 timeouts I checked on 10-13 / 10-14 alone, that is roughly 23 runner-hours of passing CI discarded in about 36 hours. We are already a top ASF consumer of GitHub Actions minutes:
Even worse is that these timeouts also happen precisely when runners are scarce, so the re-runs add load at the worst possible time and make the next timeout more likely.
Expected behavior
A PR whose checks are all passing, just slowly, should merge without anyone having to touch it again. CI work that has already passed for a queue entry should not be discarded and redone.
Additional context
The queue's status check timeout is currently 60 minutes (the GitHub default); the evictions above all happened 60 to 80 minutes after the queue built the entry.
I feel we need to reduce MQ throughput for DF, Comet is already 5 max builds
Maybe we could in theory also reduce the number of concurrent builds to reduce the load on the runners 🤔 (aka down to 3 or something). but that would in theory reduce the throughput and not reduce the overall demand)
4. I'm wondering if MQ is shared, because the similar issue has been reported in iceberg-rs repo yesterday apache/iceberg-rust#3425
@comphead I think the issue is that the github runners are shared across apache projects somehow -- so I think we are suffering from long queue wait times as the runner pool is at max capacity
Activity
blaginin commentedon Oct 14, 2026
https://github.com/apache/datafusion/actions/runs/38528268442/job/115733699502 - we were waiting 87 minutes for this run to start - even though the action itself took just 10 s 🤯
comphead commentedon Oct 14, 2026
I'm wondering if MQ is shared, because the similar issue has been reported in
iceberg-rsrepo yesterday apache/iceberg-rust#3425blaginin commentedon Oct 14, 2026
i think it's not about MQ, but instead of ASF runners being overloaded...
comphead commentedon Oct 14, 2026
I feel we need to reduce MQ throughput for DF, Comet is already 5 max builds
Its already done in #26293
alamb commentedon Oct 14, 2026
Maybe we could in theory also reduce the number of concurrent builds to reduce the load on the runners 🤔 (aka down to 3 or something). but that would in theory reduce the throughput and not reduce the overall demand)
alamb commentedon Oct 14, 2026
@comphead I think the issue is that the github runners are shared across apache projects somehow -- so I think we are suffering from long queue wait times as the runner pool is at max capacity
Yeah, @blaginin and even worse is if that then kicks the PR out of the merge queue, we have to re-run all CI again!