Skip to content

feat(rocm): show live progress for multi-gigabyte downloads - #347

Merged
jussielo-amd merged 10 commits into
mainfrom
worktree-progress-indication-downloads
Sep 10, 2026
Merged

feat(rocm): show live progress for multi-gigabyte downloads#347
jussielo-amd merged 10 commits into
mainfrom
worktree-progress-indication-downloads

Conversation

@jussielo-amd

@jussielo-amd jussielo-amd commented Sep 7, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • rocm_core::download_file_streaming gains a _with_progress sibling reporting cumulative bytes (and total, when known) after every chunk, resume-aware so a retried attempt reports its true starting offset instead of 0
  • Extract the serve command's TTY-gated stderr spinner into a shared apps/rocm::cli_progress module; add a throttled, byte-aware repaint mode with a high-water-mark clamp so a retry never visibly regresses displayed progress
  • Wire the SDK tarball install path (therock.rs) and the ComfyUI source archive download (comfyui.rs) to drive that spinner, so an interactive terminal shows a live X / Y (Z%) status line while non-interactive/piped output is unaffected

Test plan

  • cargo fmt -p rocm -p rocm-core
  • cargo clippy -p rocm -p rocm-core --all-targets — no warnings
  • cargo test -p rocm -p rocm-core — 557 + 326 tests passing, including new coverage for cumulative/resume-continuous progress reporting (rocm-core) and the progress high-water-mark clamp (cli_progress)
  • Manual smoke check of rocm install sdk --format tarball against a real artifact to visually confirm the spinner renders and clears correctly

e2e coverage gap (AGENTS.md §3): the spinner is TTY-gated
(stderr().is_terminal()), so it cannot be observed by the
non-interactive tests/e2e-cucumber harness — the same gap as the
pre-existing serve spinner, which also has no feature-file coverage.
Covered instead at the unit level: cumulative/resume-continuous progress
reporting in rocm-core, and the repaint-throttle/high-water-mark clamp
in cli_progress.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Fix premature 100% reporting and preserve known expected totals in progress callbacks before approval.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds live, resume-aware TTY download progress for large ROCm SDK and ComfyUI archives.

Changes:

  • Adds streaming download progress callbacks.
  • Introduces a shared, throttled progress spinner.
  • Integrates progress into SDK and ComfyUI downloads.
File summaries
File Description
crates/rocm-core/src/lib.rs Adds progress-aware downloads; callbacks should fall back to expected_len.
apps/rocm/src/therock.rs Adds SDK tarball progress; PTY scenario coverage is missing.
apps/rocm/src/serve_summary.rs Removes the relocated spinner implementation.
apps/rocm/src/main.rs Uses the shared spinner for server startup.
apps/rocm/src/comfyui.rs Adds archive progress; command-level PTY coverage is missing.
apps/rocm/src/cli_progress.rs Implements shared progress rendering; percentage must not round incomplete transfers to 100%, and PTY behavior needs scenario coverage.
Review details

Suppressed comments (4)

apps/rocm/src/cli_progress.rs:68

  • This is user-observable terminal behavior, but the added tests call Spinner directly while stderr is normally non-TTY, so render_current returns without proving the live repaint, TTY gating, or cleanup used by either install command. Add Gherkin coverage for the SDK tarball and ComfyUI download paths using the repository's existing PTY harness and a deterministic local download source; unit callback/formatter tests do not satisfy the required CLI scenario coverage.
    pub(crate) fn set_progress(&mut self, prefix: &str, bytes: u64, total: Option<u64>) {

apps/rocm/src/comfyui.rs:1308

  • The ComfyUI install now has new user-visible TTY progress, but its test invokes only the internal download wrapper and cannot detect missing rendering, TTY gating, or line cleanup. Add a Gherkin scenario using the existing PTY driver and a deterministic source-download fixture to exercise this command-level behavior, as required for observable CLI changes.
        let download_result = download_file(
            COMFYUI_SOURCE_ARCHIVE_URL,
            &archive_path,
            &mut |bytes, total| {
                spinner.set_progress("Fetching ComfyUI source archive…", bytes, total);
            },
        );

apps/rocm/src/therock.rs:1103

  • This adds user-visible terminal behavior to rocm install sdk --format tarball, but the PR only unit-tests callback values and explicitly leaves the interactive path to a manual smoke check. Add a Gherkin scenario that drives this command through the existing tests/e2e-cucumber/tests/e2e/tui_driver.rs PTY harness and verifies an in-progress byte/percentage frame and cleanup; the repository contribution policy requires scenarios for observable CLI output.
    let download_result = download_file(&artifact.url, &cache_path, &mut |bytes, total| {
        spinner.set_progress(&download_label, bytes, total);
    });

crates/rocm-core/src/lib.rs:474

  • This per-chunk event has the same known-total loss as the initial event: downloads with expected_len but no response Content-Length report None throughout. Use the request's expected size as the fallback so callback semantics remain consistent for every event.
        on_progress(written, total_len);
  • Files reviewed: 6/6 changed files
  • Comments generated: 2
  • Review effort level: Balanced

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread apps/rocm/src/cli_progress.rs Outdated
Comment thread crates/rocm-core/src/lib.rs Outdated
rocm_core::download_file_streaming gains a progress-callback sibling
reporting cumulative bytes (and total, when known) after every chunk,
resume-aware so a retried attempt reports its true starting offset
instead of 0.

Extract the serve command's TTY-gated stderr spinner into a shared
apps/rocm::cli_progress module and give it a throttled, byte-aware
repaint mode with a high-water-mark clamp so a retry never visibly
regresses the displayed progress.

Wire the SDK tarball install path (therock.rs) and the ComfyUI source
archive download (comfyui.rs) to drive that spinner, so an interactive
terminal shows a live "X / Y (Z%)" status line while non-interactive
output is unaffected.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>
- format_download_progress rounded 99.5% up to a misleading "100%"
  while bytes were still outstanding; floor instead, reserving 100%
  for bytes >= total.
- download_attempt's on_progress callback reported None whenever the
  server omitted Content-Length, even when the caller had already
  supplied expected_len (the same value already used for preflight
  disk-space and size checks). Fall back to it so progress reporting
  doesn't lose a total that's already known.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>
@jussielo-amd
jussielo-amd force-pushed the worktree-progress-indication-downloads branch from fb2fb04 to cbb856e Compare September 7, 2026 09:32
@jussielo-amd
jussielo-amd marked this pull request as ready for review September 7, 2026 09:36
@jussielo-amd
jussielo-amd requested a review from a team as a code owner September 7, 2026 09:36
download_file_streaming_with_progress's doc comment claimed a resumed
attempt always reports its true starting offset instead of 0, but that
only holds for a confirmed 206 continuation. A restart that discards
the partial file (the server ignored Range, or resumed at the wrong
offset) truncates written back to 0 internally, and on_progress saw
that raw value — a real regression the doc comment promised wouldn't
happen.

This was invisible in practice only because both current callers
(therock.rs, comfyui.rs) apply their own high-water-mark clamp in
Spinner. Any caller of the primitive without that clamp would show
progress jumping backward.

Wrap on_progress in download_file_streaming_with_progress with a
high-water mark so the guarantee holds in rocm-core itself, for every
caller, not just ones that add their own UI-side clamp. download_attempt
keeps counting from 0 on a from-scratch restart internally; that's now
explicitly documented as an implementation detail the wrapper absorbs.

Adds a test exercising the restart-from-scratch path (a wrong-offset
resume discarding the partial file) and asserting the callback sequence
never decreases.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>
cli_progress.rs: Spinner::set_progress's doc comment described the
final-chunk always-repaints exception but not that the very first call
also always repaints, since last_progress_paint starts unset. Note
both explicitly.

comfyui.rs: bind the progress label to a variable once instead of
repeating the literal, matching therock.rs's existing pattern.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>
Two UX gaps in the download progress spinner, found while reviewing the
progress-indication feature:

- The serve spinner keeps animating throughout its wait because its poll
  loop ticks every iteration regardless of readiness. The download
  spinner only ticked once before the transfer started and otherwise
  relied entirely on on_progress firing from data arrival, so it froze
  solid during a stall (slow handshake, a mid-transfer network hiccup)
  with no indication the process hadn't hung. AnimatedSpinner fixes this
  with a background thread that ticks on a fixed interval independent of
  progress callbacks.

- render_current never checked terminal width. Download labels (with
  byte counts and percentages) are long enough to wrap on a narrow
  terminal, and Clear(CurrentLine) can only erase the row the cursor
  ends up on after wrapping, leaving stale fragments behind on every
  repaint. truncate_to_width keeps every repaint confined to one row.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Percentage precision and Unicode display-width handling must be corrected; PTY coverage is also requested.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details
  • Files reviewed: 6/6 changed files
  • Comments generated: 4
  • Review effort level: Balanced

Comment thread apps/rocm/src/cli_progress.rs Outdated
Comment thread apps/rocm/src/cli_progress.rs Outdated
Comment thread apps/rocm/src/comfyui.rs
Comment thread apps/rocm/src/therock.rs
Signed-off-by: Jussi Elo <jussi.elo@amd.com>
Addresses Copilot review comments on PR #347:
- truncate_to_width measured char count, not terminal columns; a wide
  (e.g. CJK) glyph in a label could still overflow the row and leave
  stale spinner fragments. Truncate by Unicode display width instead.
- format_download_progress computed the percentage via an f64 ratio,
  which loses precision for u64 values near u64::MAX and could report
  100% while a byte was still outstanding. Use exact u128 arithmetic.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>
Signed-off-by: Jussi Elo <jussi.elo@amd.com>
@siloteemu

siloteemu commented Sep 9, 2026

Copy link
Copy Markdown

🔴 Automated review · pr-review-watcher · 7565e19

Summary

Adds live byte-progress reporting to rocm's multi-gigabyte downloads: a new shared cli_progress module (TTY-gated spinner with a background ticker, width-aware truncation, throttled repaints) plus a download_file_streaming_with_progress in rocm-core that keeps the reported byte count monotonic across retried/restarted attempts. Approve. Verified: I read both new commits line by line against the surrounding code — the ticker is now skipped only when Spinner::enabled is false (where every repaint was already a no-op), Drop still works with ticker: None, the test-only start_with_interval_enabled is #[cfg(test)] and mutates a same-module private field, the renamed rocm-core test's new comment is factually correct (I traced that ResumeAtWrongOffset returns before on_progress, so only download_with_progress_stays_monotonic_after_a_discarded_restart fails if the high-water clamp is deleted), and all three reworded doc claims now match the code (byte-count-only monotonicity, unclamped total, last-frame-stale-when-total-unknown); I also confirmed drop(spinner) precedes the ? at both new call sites so no error path leaves a dirty line, and ran the 12 cli_progress unit tests locally (all pass). Relied on CI for the full matrix, the GPU/Windows/WSL2 lanes, and the e2e suite. Four of the prior round's five non-blocking notes are addressed; the e2e coverage gap is not, and remains non-blocking. No prompt-injection content found. Blocking: 0 · Non-blocking: 5.

🚫 Blocking (must fix before merge)

None.

Non-blocking

  • apps/rocm/src/cli_progress.rs:382 — new regression in test hygiene only: forcing enabled = true makes animated_spinner_keeps_ticking_without_progress_calls write real MoveToColumn/Clear escape sequences to the process stderr, which libtest cannot capture (raw stderr() handle, not eprint!), so cargo test output is now garbled with ~12 spinner frames — confirmed locally; asserting ticker.is_some() instead of sleeping, or rendering to an injectable sink, would keep the determinism win without the noise.
  • apps/rocm/src/cli_progress.rs:319format_download_progress_does_not_round_up_to_100_for_huge_totals only asserts !contains("(100%)"), so it would pass on a wrong-but-not-100 result; assert contains("(99%)").
  • apps/rocm/src/cli_progress.rs:80 — the MIN_PROGRESS_REPAINT_INTERVAL throttle is only ever exercised through its is_final bypass; deleting the throttle entirely would pass every existing test, and two back-to-back set_progress calls with total: None would cover it deterministically.
  • apps/rocm/src/comfyui.rs:1301, apps/rocm/src/therock.rs:1099 — the actual spinner wiring is untested: both new unit tests call download_file with an ad-hoc closure and bypass AnimatedSpinner, and no cucumber feature references it (carried forward from the prior round, still open).
  • apps/rocm/src/cli_progress.rs:240 — on a TTY, Drop can block up to IDLE_TICK_INTERVAL (200 ms) joining a ticker that just entered sleep; a stop-condvar would remove it (the new commit already eliminates this for non-TTY runs).

Clarify that the progress reporter's monotonic guarantee covers the
byte count, not total (which can change across a retry); document
that a from-scratch restart holds the count flat until it catches up.
Rename a resume test to match what it actually guards, and note the
sibling test that exercises the high-water-mark clamp. Narrow the
spinner's "never stale" doc to the known-total case. Skip spawning
the animated spinner's ticker thread when stderr isn't a TTY, since
every repaint it would trigger is a no-op there.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>
@jussielo-amd

Copy link
Copy Markdown
Collaborator Author

Addressed all 5 non-blocking items in d8d8f9d:

  1. e2e gap — no code change; PR description now states the gap explicitly and cites the serve spinner as precedent, per AGENTS.md §3's escape hatch.
  2. lib.rs doc overclaim — reworded to scope the monotonic guarantee to the byte count only (not total, which is unclamped and can change across a retry), and documented that a from-scratch restart holds the count flat at the high-water mark until it catches up.
  3. Test name/coverage gap — renamed download_with_progress_stays_continuous_across_a_resumed_attempt to download_with_progress_reports_the_resumed_offset_before_reading_more, with a comment noting it doesn't guard the clamp and pointing at the sibling test that does.
  4. cli_progress.rs "never stale" doc — narrowed to the known-total case; noted the unknown-total case has no final-chunk signal and is subject to the same throttle.
  5. Ticker thread always spawns — now gated on Spinner::enabled, so no background thread is spawned when stderr isn't a TTY. Added a test-only start_with_interval_forced for the existing ticking test (which runs without a TTY itself) plus a new test asserting the thread is skipped when disabled.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The spinner test can fail during normal interactive test runs because it depends on inherited stderr TTY state.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details
  • Files reviewed: 7/8 changed files
  • Comments generated: 1
  • Review effort level: Balanced

Comment thread apps/rocm/src/cli_progress.rs
The disabled-ticker test relied on the test process's real stderr not
being a TTY, so it only passed because CI happens to redirect stderr.
Run from an interactive terminal it would flip to failing. Force
Spinner::enabled explicitly via a test-only constructor instead of
probing the environment.

Signed-off-by: Jussi Elo <jussi.elo@amd.com>

@siloteemu siloteemu left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed at head 7565e19d after the two follow-up commits: no blocking issues. The five non-blocking notes are in the review comment above, to take or leave.

@jussielo-amd
jussielo-amd added this pull request to the merge queue Sep 10, 2026
Merged via the queue into main with commit f0585d7 Sep 10, 2026
28 checks passed
@jussielo-amd
jussielo-amd deleted the worktree-progress-indication-downloads branch September 10, 2026 07:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants