BulkWriter produces IndexError: pop from an empty deque

[mbyrne00]

Determine this is the right repository

• I determined this is the correct repository in which to report this bug.

Happy to provide a PR if it's likely to be merged.

Summary of the issue

Context:
Using BulkWriter to bulk-write ~1M documents to Firestore from a Cloud Run job. Firestore returns retryable errors (contention) for multiple operations within the same batch window.

Expected Behaviour:
BulkWriter retries the failed operations and completes the flush successfully.

Actual Behaviour:
BulkWriter._schedule_ready_retries() crashes with IndexError: pop from an empty deque when ≥2 retries are ready simultaneously. The method interleaves popleft() with retry.retry(self), and retry.retry() re-enters the send loop (set() → _enqueue_current_batch → _ensure_sending → _send_until_queue_is_empty → _schedule_ready_retries), which drains the remaining retries from the deque. The outer loop then calls popleft() on the now-empty deque.

Related: #15385 (open refactor ticket for BulkWriter retries)

API client name and version

google-cloud-firestore 2.25.0

Reproduction steps: code

file: test_bulkwriter_bug.py

import bisect
import concurrent.futures
import datetime
from unittest.mock import patch

import pytest
from google.cloud.firestore import Client
from google.cloud.firestore_v1.bulk_writer import (
    BulkWriterSetOperation,
    OperationRetry,
)


def test_bulkwriter_schedule_ready_retries_reentrancy_bug():
    """BulkWriter crashes when ≥2 retries are ready simultaneously.

    _schedule_ready_retries computes take_until_index=N, then interleaves
    popleft() with retry.retry(self). retry.retry() re-enters the send loop
    via set() → _enqueue_current_batch → _ensure_sending →
    _send_until_queue_is_empty → _schedule_ready_retries, which drains the
    remaining retries. The outer loop then calls popleft() on the empty deque.

    batch_size=1 forces _enqueue_current_batch on every set(), making the
    re-entrant path deterministic with just 2 retries.
    """
    client = Client(project="any-project")
    writer = client.bulk_writer()
    writer.batch_size = 1

    collection = client.collection("test")
    for i in range(2):
        op = BulkWriterSetOperation(
            reference=collection.document(f"doc-{i}"),
            document_data={"i": i},
            merge=False,
            attempts=1,
        )
        bisect.insort(
            writer._retries,
            OperationRetry(
                operation=op,
                run_at=datetime.datetime(2020, 1, 1, tzinfo=datetime.timezone.utc),
            ),
        )

    f = concurrent.futures.Future()
    f.set_result(None)

    with patch.object(writer, "_send_batch", return_value=f):
        with pytest.raises(IndexError, match="pop from an empty deque"):
            writer.flush()

Reproduction steps: supporting files

None required. The test is self-contained — no Firestore emulator, network access, or credentials needed.

Reproduction steps: actual results

IndexError: pop from an empty deque

Full traceback (from _schedule_ready_retries → re-entrant _send_until_queue_is_empty → _schedule_ready_retries):

File "google/cloud/firestore_v1/bulk_writer.py", line 511, in _schedule_ready_retries
    retry: OperationRetry = self._retries.popleft()
IndexError: pop from an empty deque

Reproduction steps: expected results

flush() completes without error. All retryable operations are re-queued and processed.

OS & version + platform

First observed on Cloud Run (GCP, australia-southeast1). Reproduced locally on macOS (Apple Silicon).

Python environment

Python 3.13.7

Python dependencies

google-cloud-firestore   2.25.0

(Only google-cloud-firestore is relevant; the bug is in synchronous retry scheduling logic with no external dependencies.)

Additional context

Root cause: _schedule_ready_retries (line 499 of bulk_writer.py) computes the number of ready retries via bisect, then loops that many times calling popleft() and retry.retry(self) interleaved. retry.retry(self) calls bulk_writer.set() which, if the batch is full, triggers _enqueue_current_batch → _ensure_sending → _send_until_queue_is_empty → _schedule_ready_retries re-entrantly. The inner call drains the remaining retries; the outer loop then calls popleft() on the empty deque.

Suggested fix: Drain all ready retries into a local list before processing:

def _schedule_ready_retries(self) -> None:
    take_until_index = bisect.bisect(
        self._retries, datetime.datetime.now(tz=datetime.timezone.utc)
    )
    ready = [self._retries.popleft() for _ in range(take_until_index)]
    for retry in ready:
        retry.retry(self)

No memory concern — ready holds the same OperationRetry references already in the deque (no data copying). The maximum number of simultaneous retries is bounded by the batch size (default 20).

Related: #15385 — open refactor ticket for BulkWriter retry internals. This bug is a concrete crash caused by the hand-rolled retry logic flagged in that issue.