Skip to content

Add native HTTP/S input support (--url) #171

@vmvarela

Description

@vmvarela

Description

Add a --url (alias -u) flag to fetch data directly from HTTP/HTTPS URLs without requiring curl.

# Fetch CSV directly from URL
sql-pipe --url https://example.com/data.csv 'SELECT * FROM t'

# Fetch JSON from API
sql-pipe --url "https://api.github.com/repos/owner/repo/issues" -I json 'SELECT * FROM t'

# With custom headers
sql-pipe --url https://api.example.com/data.json -H "Authorization: Bearer token" 'SELECT * FROM t'

Motivation

The README already shows 6+ examples using curl | sql-pipe. Baking HTTP fetching into sql-pipe eliminates the curl dependency (especially beneficial for Windows users), reduces pipe ceremony, and enables automatic format detection via Content-Type response headers.

Acceptance Criteria

  • --url <url> (alias -u) flag is parsed in args.zig
  • Fetch data from HTTP/HTTPS URLs using Zig's std.http.Client
  • Auto-detect input format from Content-Type header (fallback to URL extension)
  • Support -H / --header flag for custom HTTP headers (e.g., Authorization)
  • Support --timeout flag for request timeout (default: 30s)
  • Follow redirects (up to 5)
  • Handle TLS certificates (system cert store on macOS/Windows, /etc/ssl/certs on Linux)
  • Error messages for network failures, timeouts, non-2xx responses
  • Integration tests (may need a local HTTP server fixture)
  • Help text updated

Implementation Notes

  • Zig 0.16 has std.http.Client — no external HTTP library needed
  • TLS certificate handling is the main challenge across platforms
  • Can be combined with -I for explicit format override
  • Consider caching downloaded content to disk for large files (future enhancement)
  • Estimated effort: 1 day

Metadata

Metadata

Assignees

No one assigned

    Labels

    priority:mediumShould be done soonsize:lLarge — 1 to 2 daysstatus:readyRefined and ready for sprint selectiontype:featureNew functionality

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions