fix: stream database dumps for large databases (fixes #59)#127
Open
armorbreak001 wants to merge 1 commit intoouterbase:mainfrom
Open
fix: stream database dumps for large databases (fixes #59)#127armorbreak001 wants to merge 1 commit intoouterbase:mainfrom
armorbreak001 wants to merge 1 commit intoouterbase:mainfrom
Conversation
Problem: The original dump implementation loaded ALL data from ALL tables into memory as a single string before creating the Blob response. For large databases (>1GB), this causes: - Out-of-memory errors in Durable Objects (128MB limit) - Blocked Durable Object during long exports - Failed dumps for any database that doesn't fit in memory Solution: Rewrite dump.ts to use streaming: 1. ReadableStream output — memory stays O(batch_size) not O(total_db_size) 2. Batched row fetching via LIMIT/OFFSET (1000 rows/batch) 3. Cooperative multitasking — yields control between batches so other requests on the same Durable Object can be processed 4. Proper SQL value escaping (NULL, bigint, undefined handling) Fixes outerbase#59
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #59 — Database dumps do not work on large databases
The Problem
The original
dump.tsloaded all data from all tables into memory as a single string before creating the Blob response. For databases approaching 1GB+ (the Durable Objects limit):The Solution
Rewrote
src/export/dump.tsto use streaming architecture:SELECT * FROM table(all rows)LIMIT 1000 OFFSET N(batched)Key Changes
ReadableStreamoutput — data streams to client as it's generated, never held in memory all at oncefetchRowsBatches()async generator — fetches 1,000 rows at a time via LIMIT/OFFSETcooperativeYield()— yields control every ~256KB so other requests on this DO can be processedescapeSqlValue()— handles NULL, undefined, bigint (original only handled strings and numbers)Testing
Files Changed
src/export/dump.ts— Complete rewrite of dump logic (160 insertions, 44 deletions)