Add cleaning export/API streaming for faster collection access

[isTravis]

Three changes so a multi-million-record collection can be read and exported without paging it 1,500 times.

records.ndjson — stream a whole version in one request

New endpoint: GET .../versions/:n/records.ndjson. One JSON object per line, ordered by record id, resumable via ?after=, filterable by ?type=. Paging /records costs a round trip per page purely to re-establish a cursor the server just had — 1,556 requests for 3.1M records, against one here.

Privacy filtering is identical to /records. X-Underlay-Record-Count lets a client verify completeness, which matters because a stream that dies partway cannot signal it — the 200 and headers are long gone.

Streaming export

Export previously built each record type as one in-memory array and joined it, which is why it was capped at 250k: at 3.1M the array exhausts the heap and the join would exceed V8's max string length regardless.

It now emits bounded tar parts. Types too large for one entry split into records/<Type>.0000.ndjson, .0001.ndjson, … at 25k records per part; a type that fits in one part keeps the original records/<Type>.ndjson name, so existing archives are byte-identical.

Cap raised 250k → 2M — now a "don't hand back a multi-GB tarball" guard rather than a memory limit. The SQL explorer keeps its 250k cap, since that one genuinely does hold the version in memory.

Compression

compress() on /api/*, in the app rather than at ingress so local dev matches prod. ~3× on record data.

Two bugs found while testing at 3.1M

The stream wasn't streaming. A single unbounded ORDER BY made Postgres sort every row before returning the first: ~3.5 GB of temp files, 46s to first byte, and ERROR 53100 when temp space ran out under concurrent reads. A cursor bounds the client's memory, not the server's — the original comment claiming otherwise was wrong.

Replaced with a keyset loop of bounded queries. LIMIT changes the plan qualitatively, letting Postgres walk (version_id, record_id) in index order with an incremental sort over each small group of equal ids (85 kB peak, no spill). First byte: 46s → 0.31s.

Also found record_id is not unique within a version, so the batch cursor is the (record_id, hash) pair — advancing on id alone drops or repeats rows when a duplicated id straddles a batch boundary.

gzip never applied to the stream. Hono's compress() skips it for two independent reasons: application/x-ndjson isn't in its compressible-type regex, and it bails on Transfer-Encoding: chunked. The route now compresses itself, via a pull-based ReadableStream so a slow client throttles reads rather than letting batches accumulate.

Verification (local, against 3,113,504 records)

Check	Result
Stream completeness	3,113,504 / 3,113,504 lines, 0 out of order
Unique ids	3,113,427 — the 77-line delta is exactly the duplicate ids
Time to first byte	0.31s (was ~46s)
gzip	4.03× small / 3.01× at 3.1M (5.40 GB → 1.79 GB)
Export split	Route → 25000+25000+16316, total 80,490 exact
Postgres temp spill	0 bytes, 0 errors
Tests	97/97

Docs

llms.txt (bulk-read section + when-to-use-which), /protocol (the four guarantees an implementation must honour), versions API reference, integration table, README.

Includes the ?after= edge case: because ids aren't unique, a break between two lines sharing an id skips the second on resume — same as /records paging, caught by the line-count check.