fix: bound the usage event rewind to prevent unbounded re-aggregation

[waterWang]

Fixes #13906

Problem

UsageManagerImpl.parse() rewinds the aggregation start date to the oldest unprocessed event, but has no bound on how far back it can rewind. If an event cannot be successfully processed, the rewind pins the window to that event's date permanently. Each subsequent run re-aggregates from that date to the present, growing by one aggregation period per run while the job keeps reporting success = 1.

Reported symptoms (4.22.1.0):

• 1,678 consecutive successful jobs whose start_millis never advanced past a single event 71 days earlier
• exec_time up to 42.6 min per hourly run, growing by 24 aggregation periods/day
• cloud_usage at 54.4M rows / 14 GB, mostly duplicate replayed rows
• Not specific to one event type (also Apache CloudStack Usage Server repeatedly reprocesses historical usage from fixed start date and creates duplicate cloud_usage records #13112 on 4.21.0.0)

Fix

Bound the rewind to a 24-hour window. The rewind exists to absorb clock skew between the cloud and usage databases (events created during the previous run window), not to replay the full history. Events older than 24h from the current window start will still be retried, but the aggregation window will not be rewound to them.

startDateMillis = Math.max(oldestEventDate.getTime(), startDateMillis - MAX_EVENT_REWIND_MILLIS);

Testing

• Unit test added? No (existing test class only covers event handlers; parse() has no prior tests)
• Verified the constant is referenced exactly twice (declaration + rewind bound)
• No behavior change for events within the 24h boundary window — they still rewind the window as before

[DaanHoogland]

this hard codes the agregation window to be a day at most. would small installations want for instance a week?

[Alpha162]

Thanks for picking this up, and for the detail in the description.

Two corrections on the figures which I updated on #13906 after restoring a pre-rebuild backup: it was 1,676 consecutive successful jobs pinned to start_date = 2026-06-04 16:50:51, and exec_time peaked at 3,139,799 ms, so 52 minutes. The 54.4M rows and the #13112 cross-reference both hold.

@DaanHoogland's question about a longer window is worth pinning down, because the window length sets a cost. With a permanently unprocessable event, Math.max picks startDateMillis - MAX on every run, so each aggregation period gets re-covered once per run until it drops out of the window: roughly 24x at a day, around 168x at a week. Bounded either way, which is the win here, but the multiplier scales with the bound.

The other thing left over is that a stuck event stays stuck and stays quiet. The job still records success = 1, and usage.sanity.check.interval defaults to NULL. On our cluster the fix was forcing the six affected rows to processed = 1, which took the hourly job from 52 minutes to 2.1 seconds.

None of that argues against this change. Happy to raise the quarantine and surfacing side as a separate issue if you'd rather keep this one focused.