feat: add IngestSync for completion-aware ingest

[acoshift]

Problem

Ingest() is fire-and-forget into an in-memory buffer — the HTTP POST to Quickwit happens later in a background worker. A caller consuming a pub/sub subscription has no signal for when the data is durably on the server, so it cannot decide when to Ack():

• Ack right after Ingest() returns → a crash before the worker flushes loses the data (the message was acked, never redelivered).
• Never ack → the subscription stalls and redelivers.

Exactly-once is impossible here (two-generals: there's always a window between the server's 200 and the broker's Ack). The achievable, honest target is at-least-once with caller-side dedup.

Change

Add a completion-aware ingest API that threads a per-item handle through the existing batch pipeline (so batching, 413 auto-reduce, gzip, and connection pooling are all reused):

err := c.IngestSync(ctx, doc)   // blocks until durably accepted, or ctx/drop
switch {
case err == nil:                          msg.Ack()                  // HTTP 200
case isReason(err, quickwit.ReasonEncode): deadLetter(msg); msg.Ack() // poison
default:                                   msg.Nack()                 // transient / not-confirmed
}

New public surface (purely additive; Ingest behavior unchanged):

• IngestSync(ctx, data...) error — blocking; returns nil on HTTP 200, *IngestError{Reason} on a terminal drop, or a wrapped ctx error meaning "not confirmed".
• IngestBatch(data...) *IngestReceipt + Wait(ctx)/Done() — enqueue-now / wait-later.
• IngestError, DiscardReason (ReasonEncode / ReasonBufferFull / ReasonClosed).

How it works

• ingestBuffer is now chan ingestItem{data, raw, ack}. Fire-and-forget keeps ack==nil, so every settle site is a no-op and the hot path stays 0-alloc (+~12ns for the close guard, invisible behind the POST).
• Tracked items are JSON-encoded in the caller's goroutine (synchronous encode errors; the worker never re-encodes). settle(nil) on HTTP 200 is the only success site, and a 200 always drops the chunk from the retry buffer — so no item is ever settled twice, including across the 413 oversize split.
• enqueue is ctx-aware and guarded by an RWMutex so a send cannot race Close()'s channel close and panic. Close() settles undeliverable buffered items with ReasonClosed rather than leaving waiters hanging.
• Encode failures now settle/discard at detection rather than after a 200.

Caller contract

At-least-once at the server-200 boundary. The caller must attach a deterministic doc id and dedup (Quickwit has no native dedup). Pass a deadline ctx inside the subscription's ack-deadline — IngestSync never creates its own timeout. For low-volume topics, lower SetMaxDelay so a lone item isn't held for the flush interval before its first POST (the flush ticker becomes part of Ack latency). Prefer one document per call for a clean message↔verdict mapping.

Tests

10 new tests (ingest_sync_test.go): server-accepts, encode-poison (0 requests sent), ctx-deadline-while-failing, after-close, 413 split settles all items, concurrent calls are batched, close-while-failing settles ReasonClosed, batch receipt, empty/pre-cancelled ctx. Full suite (32 tests) green, go vet clean, race detector clean across repeated runs.

🤖 Generated with Claude Code