fix(zero): size-aware predicate move timeout and rebalancer backoff

[matthewmcneely]

Summary

Zero cancels every predicate move at a hardcoded 120 minutes (predicateMoveTimeout in dgraph/cmd/zero/tablet.go), so moves of large tablets can never complete. The report in #9784 is a 20GB predicate that needed about 16 hours. This constant has been raised once before for the same reason, from 20 minutes to 2 hours in #6388, so any fixed wall-clock value is eventually wrong for someone's data size.

The failure mode is worse than a cancelled move. While a move runs, Zero aborts every commit on the predicate (blockTablet, enforced in oracle.go). On timeout, all streamed work is discarded, and the auto-rebalancer deterministically re-picks the same largest tablet every --rebalance_interval (default 8m). A cluster with one oversized predicate ends up in a permanent loop: two hours of write aborts and streaming, cancel, discard, repeat.

Changes

• Scale the move timeout with tablet size. moveTimeout() grants at least size / 256 KiB-per-second, using the larger of on-disk and uncompressed bytes, with the previous 2h constant as the floor. Tablets under ~1.8GiB behave exactly as before; the 20GB tablet from Unblock large (>15GB) predicate moves by removing hardcoded 2-hour timeout #9784 gets ~23h.
• Back off after a failed move. A failure that ran at least a minute puts the tablet on a cooldown that doubles per consecutive failure (1h up to 24h) and is never shorter than the failed attempt itself. chooseTablet skips tablets on cooldown, a successful move clears the state, and manual moves via /moveTablet bypass the backoff.
• Log the computed timeout in the move announcement line so operators can see the budget each move was given.

A configurable flag was considered and dropped. A floor-style flag reads ambiguously against a "timeout" name, and a cap-style flag recreates the original bug; the size-derived value needs no configuration.

Why 256 KiB/s

The receive path applies the whole stream through serial 32MB Raft proposals (batchAndProposeKeyValues in worker/predicate_move.go): each chunk is replicated to quorum, written to the Raft WAL, and fully applied before the next is proposed. Sustained rates in the field are therefore low; the #9784 report works out to ~0.35 MB/s on-disk. The floor rate is deliberately conservative. The throughput problem itself is out of scope here and tracked in #9790 (pipeline the receive path) and #9791 (two-phase move to shrink the commit-abort window).

Testing

• New unit tests for the timeout computation and backoff bookkeeping in dgraph/cmd/zero/tablet_test.go (TestMoveTimeout, TestMoveCooldown, TestMoveBackoff).
• New end-to-end test: TestLargePredicateMove in systest/predicate-move runs the full scenario against a real two-group cluster. It loads several GiB of incompressible data (self-calibrating the size to the host's ingest rate so the move reliably outlasts the kill point), kills the destination Alpha mid-stream, asserts the failed move triggers the rebalancer backoff and leaves the source group intact, then restarts the Alpha, retries, and asserts the move completes with both announcements carrying the size-scaled timeout. It sits behind the largemove build tag, which no CI workflow compiles; run it manually per the package README (a run loads GiBs of data and takes 15+ minutes).
• Verified on GCE (c3-standard-8, Ubuntu 24.04, pd-ssd): PASS in 16m at the default MOVE_TEST_GB=8. Zero announced timeout: 9h11m59s for the 8.68GB tablet on both attempts, the mid-stream kill produced the expected failure plus the "Skipping automatic rebalancing" backoff line, and the retry completed with all 131,296 triples intact on the destination. Observed move throughput was ~19 MiB/s against 42-68 MiB/s raw ingest even with single-replica groups, which is the receive-path bottleneck tracked in Predicate moves: pipeline destination Raft proposals and batch KV writes #9790.
• go build, gofmt, and trunk pass.
• Existing move coverage (systest/group-delete TestNodes, systest/integration2 TestUniqueMultipleGroups) moves small tablets, so the 2h floor applies and behavior there is unchanged.

Fixes #9784

🤖 Generated with Claude Code

---

^{Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.}