Skip to content

feat: Add bulk ZIP export (#867)#222

Open
stephanbuettig wants to merge 2 commits intohttptoolkit:mainfrom
stephanbuettig:feature/zip-export
Open

feat: Add bulk ZIP export (#867)#222
stephanbuettig wants to merge 2 commits intohttptoolkit:mainfrom
stephanbuettig:feature/zip-export

Conversation

@stephanbuettig
Copy link
Copy Markdown

Adds ZIP archive export for HTTP exchanges with 37 code snippet formats via @httptoolkit/httpsnippet. Includes format picker panel, Web Worker generation, and safe filename conventions.

Features:

  • ZIP export with selectable snippet formats (37 languages/clients)
  • Format picker with category grouping and popular defaults
  • Web Worker-based generation for non-blocking UI
  • Safe filename conventions matching existing HAR export pattern

New files: snippet-formats registry, export-filenames utility, download helper, zip-metadata model, zip-download-panel component.

Unit tests for snippet-formats and export-filenames included.

Extracted from #219 as requested by @pimterry.

Adds ZIP archive export for HTTP exchanges with 37 code snippet formats
via @httptoolkit/httpsnippet. Includes format picker panel, Web Worker
generation, and safe filename conventions.

Features:
- ZIP export with selectable snippet formats (37 languages/clients)
- Format picker with category grouping and popular defaults
- Web Worker-based generation for non-blocking UI
- Safe filename conventions matching existing HAR export pattern

New files: snippet-formats registry, export-filenames utility,
download helper, zip-metadata model, zip-download-panel component.

Unit tests for snippet-formats and export-filenames included.

Extracted from httptoolkit#219 as requested by @pimterry.
@stephanbuettig
Copy link
Copy Markdown
Author

✅ Manual Test Results — 2026-04-11

Both features were tested against a fresh clone of current upstream (main, commit 23a99520) using npm start.

ZIP Export (this PR)

All runs completed with 0 snippet errors:

Scenario Formats Snippets Errors Time Size
Single request, all formats 37 37 0 87 ms 67 KB
Single large request, all formats 37 37 0 290 ms 156 KB
Single request, 7 formats 7 7 0 45 ms 8.5 KB

The ZIP download panel opens correctly, format selection persists across sessions, and the generated archives are valid and well-structured.

Batch export (PR #223, depends on this PR)

Scenario Formats Snippets Errors Time Size
13 exchanges, 7 formats 7 91 0 254 ms 152 KB
14 exchanges, all formats 37 518 0 2.1 s 795 KB

Both features are production-ready and work correctly on the current upstream codebase. Ready for review and merge.

Copy link
Copy Markdown
Member

@pimterry pimterry left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I haven't gone through everything in detail yet, especially the actual UI component code, and I haven't tested this manually either, but I think this is a good set of bits to start with from a quick review. There's a strong outline here but there's going to be a good bit of work to properly integrate this into the codebase and make it maintainable for the future.

};

// Build extended optGroups with ZIP at the top
const exportOptionsWithZip: _.Dictionary<SnippetOption[]> = {
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You don't need to import the whole of lodash for this one type, you can just use Record<string, SnippetOption[]>.

There are probably old examples of doing that elsewhere here - they predate Record being added to typescript, if you spot any along the way feel free to clean them up 😄.

*
* Contains ALL available HTTPSnippet targets/clients organized by language
* category. The ZIP export pipeline, format picker UI, and batch toolbar
* all consume this registry.
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this here? HTTPSnippet already has a registry, we don't want to duplicate it, since the options available will change and this will get out of date very quickly.

Comment thread src/model/ui/ui-store.ts Outdated

@action.bound
setZipFormatIds(ids: ReadonlySet<string> | string[]) {
this._zipFormatIds = Array.isArray(ids) ? [...ids] : [...ids];
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This ternary does nothing? Both options are the same.

Comment thread src/services/ui-worker-api.ts Outdated
metadata
} as Omit<GenerateZipRequest, 'id'>));
} catch (err) {
// postMessage can throw for unserializable data (MobX proxies, etc.)
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When does this happen? We probably shouldn't catch this - we need to fix that instead. MobX proxies need to be filtered out etc. Otherwise we'll find that the export doesn't work in lots of common cases and data will silently go missing, which is a big problem.

Do you have any examples? Normally it's best to let this fail hard instead - that way it's easy to spot these issues in testing, and they'll show up in the Sentry error reports for debugging & tracking later as well.

Imo it's good to catch errors when they're due to expected issues like bad data or user input or configuration. If it can be due to implementation problems, we need to make sure that's very visible now in testing (crashing the app on purpose) and that we then fix all the problems to handle it.

Comment thread src/services/ui-worker-api.ts Outdated
};

// Safety timeout: if the worker doesn't respond within 5 minutes,
// clean up the listener to prevent memory leaks.
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doesn't really make sense, it doesn't look like handler has any references that would cause leaks here, and this doesn't actually cancel the processing, it leaves it going indefinitely. If there's a real risk it could take this long it's a problem because it will block the worker completely.

That means every other worker operation (like decoding any compressed request or response body) will wait until this is finished. If these are that slow we'll need to implement actual cancellation, look into abort controllers for how signals for that kind of thing can work.

Comment thread src/services/ui-worker-api.ts Outdated
harEntries,
formats,
metadata
} as Omit<GenerateZipRequest, 'id'>));
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why does this use a custom wrapper instead of callApi? I'd much prefer to keep everything using the same abstraction if we can. It doesn't have progress of course, or any abort support, but both could be added there instead of here, and then that would work for all worker calls which would be great.

Comment thread src/services/ui-worker.ts Outdated
),
cookies: [], // Included in headers already
...(postData !== undefined ? { postData } : {})
};
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't we already have this same preprocessing logic in the normal export snippet generation? Is this intentionally different for some reason? I would've expected to just reuse that. Likely to give better results for users, since it guarantees the content you see in the Export card is the same thing per-request you see in the batch zip export.

* - HAR batch: "HTTPToolkit_export_{date}_{count}-requests.har"
* - ZIP archive: "HTTPToolkit_{date}_{count}-requests.zip"
* - Snippet: "{index}_{METHOD}_{STATUS}_{hostname}.{ext}"
*/
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of just referencing the other name patterns elsewhere, we could just move the logic for all of these into here.

expect(result.length).to.equal(1);
});
});

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These tests just assert on a selection of hardcoded data values, which doesn't seem very useful.

I think this fail at least could probably just go away (along with the hardcoded data itself).

It would be useful to have a proper end to end test of zip generation though. We should be able to do that, we do similar testing including worker API calls in https://github.com/httptoolkit/httptoolkit-ui/blob/main/test/unit/workers/worker-decoding.spec.ts.

Doesn't need to cover every possible edge cases there, we just need a basic covering of the overall key flows to make sure the structure and key behaviours work. That's probably just the success case with a couple of examples and an error case (if there are scenarios where we expect this to fail). You can then use fflate in the test to check the expected output appears. No need to test on specific code snippet contents for specific inputs or anything like that (that's covered in a lot of detail already by httpsnippet's own tests) just that the whole flow glues together correctly.

// This is never passed to httpsnippet — it's only used for dropdown rendering.
const ZIP_SNIPPET_OPTION: SnippetOption = {
target: ZIP_ALL_FORMAT_KEY as any,
client: '' as any,
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These any are a bit suspicious. I think we should have some kind of better types that make this work properly, without this and witohut an extra special case functions like getExportFormatKey that just wraps getCodeSnippetFormatKey with one extra condition. We probably want something like export ExportOption = ZipExportOption | SnippetOption somewhere with some kind of discriminated union, and then to change the various references take either ExportOption or SnippetOption as appropriate, and discriminate to make all those types work correctly, without any.

Changes addressing @pimterry's review:

Architecture:
- DI pattern for ZipExportController (testable without module stubs)
- Shared sanitizer: simplifyHarEntryRequestForSnippetExport (single
  source of truth for snippet export sanitization, used by both
  single-export and ZIP-export paths)
- Cooperative cancellation via MessageChannel + yieldToEventLoop()
- callApi abort now rejects immediately (no 5-min hang if worker is
  stuck before first yield); exportAsZip translates AbortError back
  to cancelled response to preserve the public API contract

Bug fixes:
- Cancellation race: abortListener in callApi now calls finalize() +
  reject(AbortError) immediately, matching timeout handler behavior
- Listener ordering: emitter.once registered before worker.postMessage
  to prevent latent race with synchronous worker responses
- Type safety: replaced `undefined as any` with conditional spread in
  buildUltraSafeRequest

Code quality:
- All comments and debug logs translated to English (upstream PR ready)
- formatBytes JSDoc corrected (SI-style labels, not IEC)
- ZIP_DEBUG flag defaults to false

Tests:
- New: snippet-export-sanitization.spec.ts (hop-by-hop headers, empty
  query params, cookie clearing, postData.text preservation)
- New: zip-export-service.spec.ts (stale-run invalidation, reset
  during in-flight run)
- All 808 tests pass, 0 failures
@stephanbuettig
Copy link
Copy Markdown
Author

Hi @pimterry — thanks for the thorough review! I've pushed a new commit (fbd57d4) that addresses all your feedback. Here's a point-by-point walkthrough:


1. Lodash DictionaryRecord<string, …> (http-export-card.tsx)

✅ Removed the lodash import. Using Record<string, SnippetOption[]> now.

2. Hardcoded snippet format registry (snippet-formats.ts)

✅ Completely removed. The ZIP export now derives all format metadata directly from HTTPSnippet's own registry at runtime via availableTargets() / extname(). No hardcoded list that can go stale. The file snippet-formats.ts and its spec are gone; replaced by zip-export-formats.ts which is a thin derivation layer over HTTPSnippet.

3. Redundant ternary in ui-store.ts

✅ Fixed.

4. Silent postMessage serialization catch (ui-worker-api.ts)

✅ Removed the silent catch entirely. Errors now propagate as-is so they surface in testing and Sentry. Good call — silent data loss would have been a real problem in production.

5. Timeout-only "cancellation" without actual worker abort (ui-worker-api.ts)

✅ Replaced with proper cooperative cancellation via MessageChannel. The main thread transfers a MessagePort to the worker; calling port.postMessage({ type: 'abort' }) sets a flag the worker checks every 5 requests (yieldToEventLoop()). The worker responds with { cancelled: true } and stops immediately. The 5-minute timeout is now just a safety net, not the primary mechanism.

Additionally, the AbortSignal handler now rejects the promise immediately (not just sends cancel to the worker), so the main thread is never blocked waiting for a stuck worker. exportAsZip() translates that AbortError back to a { cancelled: true } response to keep the public API contract stable.

6. Custom wrapper instead of callApi (ui-worker-api.ts)

✅ Everything goes through callApi now. I extended it with an optional CallApiOptions<R> parameter supporting signal, onProgress, cancelChannel, and timeoutMs. All existing callers are unaffected (the options arg is optional with no defaults that change behavior).

7. Duplicated preprocessing / sanitization (ui-worker.ts)

✅ Extracted into a shared single-source-of-truth function: simplifyHarEntryRequestForSnippetExport() in src/model/ui/snippet-export-sanitization.ts. Both the single-request Export card path and the batch ZIP export path use the same sanitizer. It handles:

  • Hop-by-hop header removal (connection, keep-alive, transfer-encoding, etc.)
  • HTTP/2 pseudo-header removal (:authority, :method, :path, :scheme)
  • content-length removal (HTTPSnippet recalculates it)
  • Cookie header clearing
  • Empty query-string cleanup
  • postData.text preservation for form-encoded bodies

Dedicated test suite: snippet-export-sanitization.spec.ts.

8. Filename logic consolidation (export-filenames.ts)

✅ Moved all export filename logic (sanitization, Windows reserved names, ZIP path building, collision avoidance) into src/util/export-filenames.ts as a single module. Status-code embedding in filenames (001_200_example-com.sh) is also handled here.

9. Hardcoded format tests → real worker round-trip tests

✅ Deleted snippet-formats.spec.ts. Replaced with a proper end-to-end worker round-trip test suite at test/unit/workers/zip-export.spec.ts (modeled after worker-decoding.spec.ts as you suggested). It covers:

  • Valid ZIP with snippets, HAR, and manifest
  • Cancellation mid-flight (AbortController)
  • Header sanitization verification (no content-length / pseudo-headers in output)
  • Partial failures (bogus format → error in manifest, not overall rejection)
  • Empty request / empty format rejection
  • Error records with full context (entryIndex, method, url, status)
  • Status code in filenames
  • Body truncation with snippetBodySizeLimit
  • Form-encoded postData.text preservation
  • clj-http crash recovery via ultra-safe retry (nested-null and top-level-null JSON bodies)

All 808 tests pass, 0 failures.

10. Type safety — ExportOption discriminated union (http-export-card.tsx)

✅ Removed the any casts. The ZIP option is now cleanly separated — getExportFormatKey() is gone; the ZIP button is handled as its own code path, not shoehorned into the snippet option type system.


Additional improvements in this commit:

  • All comments and debug logs translated to English (was German in the first push)
  • ZIP_DEBUG flag defaults to false
  • formatBytes JSDoc corrected
  • Listener ordering fix: emitter.once registered before worker.postMessage (latent race)
  • Type safety: replaced undefined as any with conditional spread in buildUltraSafeRequest

Let me know if anything needs further adjustment!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants