Pre-serialize pub/sub signals once across fan-out (flag-gated, default off)#2485
Draft
thjaeckle wants to merge 3 commits into
Draft
Pre-serialize pub/sub signals once across fan-out (flag-gated, default off)#2485thjaeckle wants to merge 3 commits into
thjaeckle wants to merge 3 commits into
Conversation
… off) Pekko Artery re-serializes each published signal once per subscriber node (prod: ~4.74x fan-out; the identical DittoHeaders dominate toBinary CPU, which is ~21% of things-service CPU). Add an opt-in path that serializes a published signal once and reuses the bytes across all remote destinations: - SignalBytesHolder: serialize-once memoizer shared across a publication's fan-out envelopes (single volatile record, benign race). - PreSerializedPublishSignal: wire-only transport envelope (shared holder + per-destination groups); never delivered to a Subscriber as-is. - PreSerializedPublishSignalSerializer: custom length-prefixed frame reusing CborJsonifiableSerializer for the signal payload; fromBinary rebuilds a plain PublishSignal so the Subscriber is unchanged. Registered in ditto-pekko-config.conf (id 656329406). - Publisher builds one shared holder per publication and sends the envelope when enabled and the signal fans out to more than one destination; AbstractSubscriber converts a locally-delivered envelope back to a PublishSignal. - New config flag ditto.pubsub.pre-serialize-fanout-enabled (default false, env DITTO_PUBSUB_PRE_SERIALIZE_FANOUT_ENABLED). Micro-benchmark (fan-out 5, 50 read-granted subjects): 131us -> 25us per publication (5.18x less serialization work). Default off; requires a two-step decode-first rollout before enabling fleet-wide (documented in reference.conf). When enabled, fan-out destinations share one serialization span (trade-off of serializing once) -- also noted in reference.conf. Tests: unit round-trip (both paths + memoization + headers/readGrantedSubjects survival + concurrent holder access), SerializationExtension registration + round-trip, 2-node end-to-end with the flag on (remote + local delivery), and PubSubFactoryTest unchanged with the flag off. Full reactor build (compile + checkstyle + license + japicmp) green. Reviewed with the workflow code-review; fixed the one confirmed correctness issue (guard the deserialized-payload cast against a non-Signal result instead of an opaque ClassCastException) and added the fan-out>1 threshold. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Expose the ditto.pubsub.pre-serialize-fanout-enabled setting (default false) for the three services that run a pub/sub Publisher (things, policies, connectivity) via DITTO_PUBSUB_PRE_SERIALIZE_FANOUT_ENABLED, mirroring the existing policiesEnforcer settings. Lets operators toggle the optimization per the required decode-first, enable-fleet-wide rollout. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Address code-review findings on the flag-gated pre-serialize fan-out: - PreSerializedPublishSignalSerializer: pin BIG_ENDIAN byte order so the Artery (little-endian) and byte[] (big-endian) paths interoperate; validate every wire length prefix against remaining bytes before allocating (no OOM/NegativeArraySize on a truncated/corrupt/misrouted frame); propagate the inner serializer's NotSerializableException for an unknown inner manifest instead of throwing IllegalArgumentException on the Artery decoder thread; wrap parse failures as a clean NotSerializableException; UTF-8-encode each string exactly once (encode-once EncodedFrame) instead of re-encoding in both size and write passes. - Publisher: only build the shared holder / envelope when the fan-out reaches at least one remote node, so purely-local (single-node/dev) fan-out no longer allocates a holder + envelopes for payloads that are never serialized. - Tests: cover unknown inner manifest, truncated frame and corrupt length prefix. - Docs: add a 3.9.4 release-notes draft with a two-phase safe-activation guide (upgrade whole fleet with flag off, then enable on the publishing services). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Member
Author
|
System tests:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Introduce an opt-in optimization that serializes a published signal once and reuses those bytes across all
remote fan-out destinations, instead of letting Pekko Artery re-serialize the identical payload once per
destination. The optimization is flag-gated behind
ditto.pubsub.pre-serialize-fanout-enabledand disabled bydefault.
Why
Under high fan-out load, cluster-internal serialization of published signals is the dominant CPU cost on the
thingsservice (consistently the #1 hotspot, ~25% of CPU, in production JFR profiles). When a single publishedsignal fans out to N subscriber nodes, Artery serializes the same payload N times.
How
PreSerializedPublishSignaltransport envelope carrying a shared, lazily-computedSignalBytesHolder. Thefirst Artery encoder thread serializes the payload once; every subsequent remote destination reuses the cached
bytes. Only the small per-destination routing information differs between envelopes.
PreSerializedPublishSignalSerializer(own serializer id) reconstructs a plainPublishSignalonthe receiving side, so the
Subscriberactor is unchanged.Publisheronly uses the envelope when the fan-out reaches at least one remote node; purely node-localfan-out keeps using the in-memory signal (nothing is serialized there).
things,policies,connectivity). Thesubscriber-only services (
gateway,thingssearch) do not read it, but must run this version so they candeserialize incoming envelopes.
Safety / rollout
The envelope relies on a new serializer; a node that does not have it registered would drop the fan-out message.
It is therefore only safe to enable after the whole cluster runs this version. A two-phase rollout is
documented in the (draft) 3.9.4 release notes: upgrade the entire fleet with the flag off, then enable it on the
publishing services. Because the default is off, upgrading is a no-op until the flag is explicitly turned on.
Serializer hardening
byte[]serialization paths interoperate.misrouted frame fails cleanly instead of raising
OutOfMemoryError/NegativeArraySizeExceptionon the Arteryinbound thread.
serializer's
NotSerializableException— consistent with the regularPublishSignalpath — instead of throwingon the decoder thread. Strings are UTF-8-encoded exactly once.
Tests
Serializer round-trip (both entry points), holder thread-safety / serialize-once, serialization-extension
registration, an end-to-end fan-out across two clustered actor systems, and the malformed / truncated /
unknown-manifest failure cases.