Rate limiting and circuit breaker for transaction submission

[tac0turtle]

Summary

Add backpressure mechanisms to the transaction submission path to handle DA layer degradation gracefully.

Parent: #4

Context

celestia-node has zero rate limiting or backpressure on submission — just a mutex serializing broadcasts. Under load or during network issues, this leads to unbounded retries and resource exhaustion.

Requirements

Concurrency limiter

• Semaphore bounding max in-flight transactions (configurable, default: 4)
• Callers block when limit is reached
• Works with multi-account (Multi-account support for transaction submission #5): limit is global across all accounts

Exponential backoff

• On repeated submission failures, increase delay between attempts
• Base: 1s, max: 30s, jitter: +/- 25%
• Reset backoff on successful submission

Circuit breaker

• Track failure rate over a sliding window (e.g., last 20 submissions)
• States: closed (normal) → open (failing, reject immediately) → half-open (probe)
• Open threshold: >50% failure rate over window
• Half-open: allow 1 submission, if success → closed, if fail → open
• When open: return immediate error with "circuit open" context, don't waste gas

Metrics (#7)

• apex_submission_inflight (gauge)
• apex_submission_circuit_state (gauge — 0=closed, 1=half-open, 2=open)
• apex_submission_backoff_seconds (gauge)

[submission.rate_limit]
max_inflight = 4
backoff_base = "1s"
backoff_max = "30s"
circuit_window = 20
circuit_open_threshold = 0.5

References

• Sony's gobreaker or similar for circuit breaker pattern