Context
Formal.ai claims sub-10ms p50 proxy overhead. Gate has no published latency numbers. Without benchmarks, Gate cannot compete in enterprise evaluations where latency is a hard requirement.
Proposal
Create a reproducible benchmark suite:
- Benchmark script in
scripts/benchmark.sh or internal/benchmark/
- Scenarios: direct connection vs Gate-proxied for each protocol
- PostgreSQL: simple SELECT, complex JOIN, INSERT batch
- MySQL: equivalent queries
- With and without policy evaluation (0 policies, 1 policy, 10 policies)
- With and without masking enabled
- Metrics: p50, p95, p99 latency; throughput (queries/sec); CPU/memory overhead
- Hardware baseline: document the test hardware (e.g., M4 MacBook Pro, or 4-core EC2 instance)
- Automated: runs in CI on demand (not every push) to catch regressions
- Published: results in README or dedicated benchmarks page
Acceptance Criteria
Context
Formal.ai claims sub-10ms p50 proxy overhead. Gate has no published latency numbers. Without benchmarks, Gate cannot compete in enterprise evaluations where latency is a hard requirement.
Proposal
Create a reproducible benchmark suite:
scripts/benchmark.shorinternal/benchmark/Acceptance Criteria