feat: detect server-side document rejections on a 200 ingest

[acoshift]

The highest-stakes reliability item from the review (#1): close the silent-loss hole where Quickwit returns 200 but dropped documents.

Problem

A Quickwit ingest 200 only means the docs were queued. The response body reports num_ingested_docs / num_rejected_docs, and a parse-rejected document (bad JSON or schema) is dropped while the client still sees 200. Today flush() does io.Copy(io.Discard, resp.Body) and Acks those docs as durable — silent loss (quickwit-oss/quickwit#5356, #3190).

Change

flush() now reads the response (bounded to 1 MiB) and inspects it:

Server response	Verdict
Truncated / unreadable 200	retryable (ambiguous — don't settle)
num_rejected_docs > 0 and num_ingested_docs == 0 (whole batch rejected)	settle tracked items ReasonRejected (terminal), discard fire-and-forget — exact, nothing was ingested
Partial (num_ingested_docs > 0 and num_rejected_docs > 0)	batch still Acked, count surfaced via OnReject(n) + warning — can't attribute per-doc in a coalesced batch
Empty / unparseable / missing fields (older servers)	accepted (lenient — preserves accept-on-200)

Idle-flush makes the common case precise. A low-volume IngestSync document is flushed as its own one-doc batch (from #20), so a rejected doc hits the exact num_ingested_docs == 0 path and the caller gets a precise ReasonRejected verdict — not a guess.

API added

• ReasonRejected (DiscardReason) — terminal: dead-letter and Ack.
• OnReject(func(numRejected int)) — the signal for partial rejections the client can't attribute.
• SetIngestDetailedResponse(true) — opt-in ?detailed_response=true so per-document parse-failure reasons are logged (off by default; adds server cost).

Invariants

Fire-and-forget batching, ordering, the 4xx failure taxonomy, and the no-double-settle guarantee are unchanged — the all-rejected path settles each item exactly once and drops the batch (so it's never retried or double-settled). Lenient parsing means servers that don't return the rejection JSON behave exactly as before.

Honest limitation

Partial rejection within a coalesced batch can't be attributed to specific documents, so those docs are Acked and surfaced only via OnReject/log. One-doc-per-IngestSync + idle-flush avoids this for the typical pub/sub flow; high-volume coalesced batches degrade to count-only reporting.

Tests

7 new (ingest_reject_test.go): all-rejected → ReasonRejected + OnReject; no-rejection → nil; partial → nil + OnReject(count); unknown/missing-fields body → nil; truncated 200 (hijack-close) → retried to success; ?detailed_response=true sent; fire-and-forget all-rejected → OnDiscard + Close returns. Full suite green, go vet clean, race-clean across -count=2 and the reject/truncate tests held over -count=5.

🤖 Generated with Claude Code