Skip to content

fix(archiver): batch multi-key point reads into one LMDB round trip - #25282

Open
spalladino wants to merge 1 commit into
spl/kv-store-read-only-txfrom
spl/batched-kv-gets
Open

fix(archiver): batch multi-key point reads into one LMDB round trip#25282
spalladino wants to merge 1 commit into
spl/kv-store-read-only-txfrom
spl/batched-kv-gets

Conversation

@spalladino

@spalladino spalladino commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

Stack

Part of A-1817. Bottom to top, each PR targets the branch below it:

Context

Every getAsync on the lmdb-v2 kv-store costs a full JS→C++ msgpack round trip, and the archiver has hot paths doing one point read per tx effect or per log: getBlock awaited a sequential read per tx (120-tx block = 120 serialized round trips), and getNoteHashesAndNullifiers (the includeEffects path of getLogsByTags) issued one message per tx hash. The wire protocol's GET request already accepts an array of keys and the native handler returns one slot per key in order — nothing on the TS side used it.

Approach

  • ReadTransaction.getMany(keys, { chunkSize }) sends the keys in GET messages of at most chunkSize keys (default DEFAULT_GET_CHUNK_SIZE = 1024), which bounds message size and how long one read holds its transaction; callers can override per call; WriteTransaction overrides it, resolving each key against the pending batch first (pending writes and removes behave exactly like N sequential get calls) and batch-fetching only the rest. The single-key get now shares the same pending-lookup helper.
  • AztecAsyncMap.getManyAsync(keys) is added to the interface; lmdb-v2 maps use the batched read, other backends (lmdb-v1, sqlite-opfs, indexeddb, lmdb-v2 multimap) use a Promise.all of getAsync.
  • Converted the archiver's N-point-read loops: getBlock's tx-effect loop, getNoteHashesAndNullifiers, and the per-block log reads behind getPrivateLogsForBlock/getPublicLogsForBlock. Warn messages, missing-key behavior, and output order are unchanged.

Measured (interleaved A/B, 3 runs each): getBlock on a 120-tx block 80–84ms → 64–71ms (~17% faster; most of the win is not interleaving deserialization between sequential awaits). getNoteHashesAndNullifiers with 100 hashes was already pipelined via Promise.all, so it drops 100 messages to 1 with a small wall-clock gain (~7%).

API changes

AztecAsyncMap gains getManyAsync(keys: K[], opts?: { chunkSize?: number }): Promise<(V | undefined)[]>; any external implementations of the interface need the method (a Promise.all over getAsync is a valid implementation).

@github-actions github-actions Bot added the port-to-next Forward-port this merged PR into next label Aug 21, 2026
@spalladino
spalladino force-pushed the spl/batched-kv-gets branch from 38e30f9 to 66b3842 Compare August 25, 2026 19:34
@spalladino
spalladino changed the base branch from merge-train/spartan-v5 to spl/kv-store-read-only-tx August 25, 2026 19:34
@spalladino
spalladino force-pushed the spl/batched-kv-gets branch from 66b3842 to 155fde8 Compare August 25, 2026 19:44
@spalladino spalladino removed the port-to-next Forward-port this merged PR into next label Aug 25, 2026
@spalladino
spalladino force-pushed the spl/batched-kv-gets branch from 155fde8 to e99d252 Compare August 25, 2026 21:13
…d trip

The lmdb-v2 wire protocol always supported multi-key GET, but every
getAsync sent a single-key message. Add getMany to the read and write
transactions (overlaying pending batch state) and getManyAsync to
AztecAsyncMap, and use it where the archiver did one point read per tx
effect or per log: getBlock, getNoteHashesAndNullifiers, and
per-block log reads.
@spalladino
spalladino force-pushed the spl/batched-kv-gets branch from e99d252 to c4e8a6c Compare August 25, 2026 21:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant