feat(router): fan-out V2 PubSub Publish() via FanoutWriter

[dlinsley]

Description

Summary

The router's V2 path currently processes envelopes through a single Repeater goroutine calling v2PubSub.Publish() serially. Each Publish() call walks the subscriber tree and invokes every matching subscriber's Set sequentially, so envelopes queue up behind one another even when CPUs are available.

This PR introduces FanoutWriter, which sits between the Repeater and v2PubSub.Publish() as an opt-in feature. The Repeater feeds envelopes into a small buffered channel (2 slots per worker); N worker goroutines drain the channel and each calls Publish() independently. This allows multiple envelopes to be published concurrently — each worker handles a different envelope, and their Publish() calls run in parallel across CPUs. The per-envelope subscriber walk itself is unchanged. This can be activated as an opt-in feature by adding env variable ROUTER_FANOUT_WRITER_ENABLED=true

Concurrent publishing means envelopes may be delivered to subscribers in a different order than they were received. loggregator-release already makes no guarantees for envelope ordering — the diode buffer is lossy by design, and subscribers are expected to handle out-of-order delivery.

Concurrency concerns to be aware of:

• Many-to-one go-diode is safe for many writers and a single reader
• v2PubSub.Publish() is safe to be called by many go routines
• server.v2.Repeater is still a single go routine but no longer blocked by v2PubSub.Publish() on each Envelope. It now just forwards envelope pointers from the Many-to-one go-diode to a small buffered channel.

Changes

• fanout_writer.go — new FanoutWriter type with Start/Stop lifecycle, a Write method satisfying the Writer function signature, and a WithBufferSize option.
• router.go / config.go — FanoutWriter is opt-in via two new env vars:
  • ROUTER_FANOUT_WRITER_ENABLED — enables the feature (default: false)
  • ROUTER_PUBLISH_WORKERS — number of concurrent workers (default: GOMAXPROCS)
• router_test.go - extended for when FanoutWriter is enabled
• fanout_writer_test.go — Ginkgo specs covering delivery correctness and clean Stop behaviour.
• pubsub_benchmark_test.go — extended with three new benchmarks (BenchmarkDopplerRouterNoOp, BenchmarkDopplerRouterParallelNoOp, BenchmarkDopplerRouterFanout) and a second fixture PubSub using SleepSetter (1 µs time.Sleep per subscriber) in an attempt to model realistic per-subscriber cost.

Benchmark results

20k subscribers, 1 µs cost each, GOMAXPROCS swept via -cpu 1,2,4,8,16 (n=10, benchstat):

                 │ bench_orig.txt │          bench_fanout.txt           │
                 │     sec/op     │   sec/op     vs base                │
DopplerRouter        5.015m ±  4%   4.426m ± 8%  -11.74% (p=0.000 n=10)
DopplerRouter-2      172.0µ ± 25%   115.2µ ± 9%  -33.02% (p=0.000 n=10)
DopplerRouter-4     163.90µ ±  6%   30.67µ ± 3%  -81.29% (p=0.000 n=10)
DopplerRouter-8    166.221µ ± 13%   7.487µ ± 8%  -95.50% (p=0.000 n=10)
DopplerRouter-16   167.942µ ±  6%   1.793µ ± 2%  -98.93% (p=0.000 n=10)
geomean              330.6µ         46.17µ       -86.03%

The original serializes Publish() calls, so adding CPUs gives no benefit (times plateau around 164–172 µs). With FanoutWriter, independent envelopes are published in parallel: at 16 CPUs it is ~94× faster

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

Testing performed?

• Unit tests
• Integration tests
• Acceptance tests

Checklist:

• This PR is being made against the main branch, or relevant version branch
• I have made corresponding changes to the documentation
• I have added testing for my changes

If you have any questions, or want to get attention for a PR or issue please reach out on the #logging-and-metrics channel in the cloudfoundry slack