WIP Shadow test ai - #1803
Draft
lispc wants to merge 98 commits into
Draft
Conversation
…d in current code base)
Co-authored-by: georgehao <hongfan@scroll.io>
…ternal to develop baseline
…SONS_LEARNED.md, update for v0.9.0 S3 paths and S3 digest extraction
…er to test infra, update docs for S3 digests - Deploy ZkEvmVerifierPostFeynman with S3 digests and protocolVersion=10 - Move shadow-only verifier wrapper out of scroll-contracts submodule into tests/shadow-testing/contracts/ - Update GUIDE/TROUBLESHOOTING/README: prefer S3 digest files, remove proof-extraction fallback - Prover-up script: asset detours for v0.9.0 S3 layout - fetch-l2-blocks: store full header/transactions for coordinator chunk-task generation
- libzkp: add openvm-sdk/openvm-stark-sdk, input_commits, batch_root_verifier_vk loading - prover-bin: adapt dumper and universal handler to v0.9.0 ProverConfig/StarkProof - common/message: introduce OpenVMStarkProof with UserPvsProof, switch chunk/batch to StarkProof - coordinator: update proof receiver, mock verifier and tests for StarkProof - rust-toolchain: bump to nightly-2025-11-20 + rust-src - regenerate Cargo.lock to resolve v0.9.0 dependency graph
- prover.rs: load child circuit handler and call enable_deferral for Batch/Bundle tasks - UniversalHandler: expose enable_deferral wrapper around Prover::enable_deferral - CircuitConfig: add child_circuit_vks for chunk/batch child lookup - 04-prover-up.sh: populate child_circuit_vks from openVmVk.json
…rifier, add v0.9.0 lessons
formatProverTask/applyUniversal/InsertProverTask failures after Update*Attempts previously refunded only active_attempts, permanently burning total_attempts. After SessionAttempts (5) such tasks became invisible to GetUnassigned* (total_attempts < maxAttempts) with no prover_task row, so the cron timeout checker could never mark them failed either — silent starvation (shadow-testing Trap 22/24). Add RefundAttemptsByHash (chunk/batch/bundle orm) and recoverAttempts: when the charge happened in this Assign call (hasAssignedTask == nil), refund both counters and reset proving_status to unassigned; re-polls of already-assigned tasks keep the old active-only recovery. Also cover the previously unrecovered fixCompatibility failure sites in batch/bundle assign.
- lib/: scripts and configs shared by both modes (anvil/verifier/prover/ relayer setup, sync-queue-hashes, anvil-utils) - follow/: follow-mode scripts, configs, Makefile, GUIDE, TROUBLESHOOTING (poll-sync follow of mainnet tip, finalize on Anvil fork) - snapshot/: snapshot-replay scripts, configs, Makefile, GUIDE, TROUBLESHOOTING (historical fork + fixed bundle range) - docs/COMMON-TROUBLESHOOTING.md: mode-independent traps; per-mode TROUBLESHOOTING files carry the mode-tagged traps verbatim - root Makefile becomes a thin dispatcher; README rewritten as mode chooser - 10-follow-up.sh: drop follow-run.env on --reset so a fresh run gets a fresh report window; assert postgres log_statement='mod' on every setup (also in 02-prepare-db.sh) for write attribution - update AGENTS.md references and .gitignore for per-mode config dirs
- l2_config.disable_l2_watcher: skip the L2 missing-blocks fetch loop. In shadow mode the DB is populated by import/poll-sync; an empty l2_block table previously made the watcher crawl from genesis (shadow Trap 26). Default false keeps production behavior. - sender_config.chain_nonce_only: initialize the sender nonce from the chain pending nonce only, ignoring pending_transaction rows, which are not chain/fork-scoped and poisoned nonces on fresh forks (shadow Traps 7/27). Default false keeps max(db+1, chain) behavior. - estimategas: set an explicit 30M gas cap in the estimation CallMsg instead of 0; Anvil rejects fee-capped eth_estimateGas with Gas=0 (shadow Trap 9). The cap only bounds the binary search, estimates are unchanged. - l2_relayer: move full-calldata dumps on commit/finalize send errors from Error to Debug (was flooding logs every retry tick), and log a loud Error when a bundle is marked RollupFinalizeFailed — status 7 is never retried by ProcessPendingBundles and needs manual reset.
- relayer.json.template: disable_l2_watcher + chain_nonce_only - 10-follow-up.sh --reset: TRUNCATE pending_transaction (stale rows poison sender nonces and can replay old calldata on a fresh fork) - mark Traps 26/7/9 as fixed upstream; document the new config knobs in follow/GUIDE.md
…hase - 10-follow-up.sh: FOLLOW_FORK_HOURS_BACK (default 5) — fork at tip - hours*300 blocks so the run opens with a real backlog (committed-but-not-finalized at fork + everything produced since); lastFinalized/lastCommitted now read AT the fork block; 0 = old fork-at-tip behavior. follow-run.env records FORK_HOURS_BACK and MAINNET_BUNDLE_TIP_AT_START. - generate-catchup-report.py: split report into catch-up phase (fork finalized index climbing to the recorded mainnet bundle tip) and steady-state phase; use the max_finalized_bundle watermark for finalized counts; backward-compatible with old env files. - docs: follow README/GUIDE + AGENTS.md describe catch-up-then-follow.
…over The impersonated eth_sendTransaction for updateVerifier could still be in the txpool when addProver signed with the same owner nonce, failing with 'replacement transaction underpriced' (hit twice at bring-up). Poll for the receipt before continuing.
…rover in follow-up (traps 28/29)
… blob-base-fee); accept Fusaka-era fee floor
Simulate a production zk upgrade with continuous finalization on a single fork/DB: Phase 1 runs the production stack against the fork's untouched production MVRV verifier, then 20-upgrade.sh cuts over to the new stack (coordinator assets swap + restart, prover restart, new wrapper registered via the genuine updateVerifier path so the old wrapper moves to legacyVerifiers). Bundles proven pre-boundary keep the old verifier (mixed mode); everything at/after the boundary batch is reset and re-proven. - follow/scripts/20-upgrade.sh: cutover orchestrator (boundary computation with hard-cutover fallback, in-flight task reset, genuine registration, MVRV routing verification, boundary recorded in follow-run.env) - 10-follow-up.sh: --skip-verifier for Phase 1 (use the fork's production verifier; assert routing, code presence, protocolVersion) - lib/03-deploy-verifier.sh: --start-batch and --genuine-register (real updateVerifier instead of forcing the latestVerifier storage slot) - lib/04-prover-up.sh: PROVER_BIN and ASSETS_DIR env overrides so phases can point at binaries/assets from different checkouts (e.g. a develop worktree for Phase 1) - lib/01-setup-anvil.sh: --skip-verifier flag; fix the fallback that registered a codeless wrapper because cast code returns 0x for empty accounts - monitor-catchup.py: verifier drift check follows MVRV routing (getVerifier(lastFinalized+1)), works across the two-wrapper boundary - generate-catchup-report.py: split metrics pre/post upgrade - Makefile: follow-old / follow-upgrade targets - docs: mid-run upgrade section in follow/GUIDE.md incl. how to determine the production zk stack; new docs/bundle-digest-encoding.md (v0.8.0 S3 digests are Montgomery, v0.9.0+ canonical); Traps 30-34 in the TROUBLESHOOTING files; AGENTS.md updates
…eration - Pin scroll-zkvm-* to zkvm-prover master bf887150 (halo2-gpu support): the SNARK (bundle) aggregation prover can now run on GPU via the new `halo2-gpu` cargo feature (24 GB-class GPUs), exposed as the `prover_halo2gpu` target in zkvm-prover/Makefile. - Prover now downloads agg_vk.bin alongside app.vmexe/openvm.toml: without the pre-built aggregation VK the prover derives it from the SDK, which is slow and allocates GPU memory that is never reclaimed (the issue the upstream commit fixed). agg_vk.bin must exist on S3 next to each circuit. - Cargo.lock: revert revm-family resolution drift introduced by `cargo update -p scroll-zkvm-*` (alloy-evm back to revm 30.1.1, revm-primitives edges back to the scroll-v91 fork) so the [patch] fork resolution keeps compiling. See follow/TROUBLESHOOTING.md Trap 32. - Docs: new-stack checklist in follow/GUIDE.md mid-run upgrade section, Trap 32 (revm drift) + Trap 33 (agg_vk.bin / batch_root_verifier_vk mandatory assets) in follow/TROUBLESHOOTING.md, AGENTS.md build command and S3 asset notes.
…_vk.bin
libzkp/coordinator:
- Batch-proof verification now reads the batch circuit's agg_vk.bin from
the assets dir instead of a separate batch_root_verifier_vk file (same
content, one less artifact to track).
Upgrade-run fixes learned from the first live 20-upgrade.sh run:
- 20-upgrade.sh reset SQL: also reset rows already at proving_status=1 —
their ready flags (batch_proofs_status/chunk_proofs_status) and proof
blobs can still reference old-circuit proofs — and DELETE stale assigned
prover_task rows while provers are stopped. Previously the new
coordinator rebuilt poisoned tasks from old-format proofs on every poll
and the priority dispatcher starved all task types (Trap 35).
- 03-deploy-verifier.sh: accept foundry >= 1.6 'cast receipt' status
format ('1 (success)') so a successful updateVerifier is not misreported
as reverted (Trap 36).
Docs:
- TROUBLESHOOTING: Trap 34 (stale ProverProofInvalid rows silently starve
batch assignment; sweeper does not clear prover_task), Trap 35
(post-upgrade leftover state), Trap 36 (cast status format); Trap 33
rewritten for the agg_vk.bin-only asset story.
- GUIDE: upgrade notes — aggressive reset rationale, slow first chunk
task generation (per-block witness fetching) is normal.
- AGENTS.md: coordinator reads batch circuit's agg_vk.bin.
…_hook_commit Two crashes hit the first halo2-gpu bundle proving on the shadow fork (both provers died at the SNARK stage, peak 22.9 GiB): 1. quotient.cu cudaErrorInvalidConfiguration: UniversalHandler::agg_vk() built the child's full GPU aggregation prover (~5.7 GiB/circuit, never reclaimed by the VPMM pool) just to read the VK, starving the SNARK phase until quotient chunking computed batch_size = 0. Now uses Prover::load_agg_vk() which reads the pre-built agg_vk.bin asset (matches zkvm-prover bf887150's VRAM-budgeting guidance). 2. 'def_hook_commit must be defined': the bundle verify circuit's hook commit comes from the batch child SDK's deferral prover, which used to be initialized as a side effect of sdk.agg_vk(). do_prove now runs batch.enable_deferral(chunk) before bundle.enable_deferral(batch) for bundle tasks, mirroring the zkvm integration tester. Also: generate-catchup-report.py accepts +0000 timezone offsets. Verified end-to-end on the mid-run upgrade shadow fork: bundles 18246/18247 (end_batch 518861/518862 >= upgrade boundary) were re-proven with the halo2-gpu stack and finalized through the newly-registered wrapper; older bundles finalized through the legacy wrapper. Documented as Trap 38; also adds Trap 37 (stale embedded libzkp.so after coordinator rebuild).
… VRAM starvation Fix 1 (load_agg_vk from file) removed the 5.7 GiB/circuit agg-VK derivation, but a prover process that served chunk/batch tasks before a bundle task still crashed at the halo2_outer SNARK stage with quotient.cu cudaErrorInvalidConfiguration (peak 22.8 GiB on 24 GB cards): every circuit's GPU SDK stays resident once lazily built, and the VPMM pool never returns pages, so the quotient chunking heuristic (raw cudaMemGetInfo) computed batch_size = 0. do_prove now calls Prover::reset() on the child/grandchild handlers right after enable_deferral (+ file-based agg_vk), mirroring the zkvm integration tester: the parent captures agg_vk / cached commit / def_hook_commit by value, so the child SDK is no longer needed and its GPU proving keys are released before the parent's STARK/SNARK phase. Verified live on the mainnet shadow fork (mid-run upgrade test): bundle 18394 was proven by a mixed-type process (same process had proved chunks minutes earlier) and finalized via the new wrapper — halo2_outer create_proof live peak 8.6 GiB vs 22.8 GiB pre-fix.
…ssons - 03-deploy-verifier.sh: cast 1.7 prints 'true'/'false' for receipt status — the genuine-register path misread a successful updateVerifier as reverted and aborted 20-upgrade.sh mid-cutover. Also document the manual recovery (finish steps i/j/k, do NOT re-run the script). - Trap 38: Symptom 3/Cause 3 (mixed task types re-create VRAM starvation post-Fix-1), per-GPU supported_proof_types isolation workaround, Fix 3 (child SDK reset after deferral, verified on bundle 18394), upstream pool-free-bytes budgeting suggestion, and crash-recovery corollaries (reset active_attempts, wipe the local SDK db before restart, cleanup ordering). - Trap 36: note the cast 1.7 output drift.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.