Skip to content

[DON'T MERGE] Run multi-rank CPU unit tests in CI via LOCAL_SIZE #8381

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
delock wants to merge 3 commits into
base: master
Choose a base branch
from
+19 −2
Open

[DON'T MERGE] Run multi-rank CPU unit tests in CI via LOCAL_SIZE #8381

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
ec752ba
Run multi-rank CPU unit tests in CI via LOCAL_SIZE
delock Sep 1, 2026
f2b7d0b
Disable dist-env reuse for the full multi-rank CPU run
delock Sep 1, 2026
f2793e7
Split CPU unit tests into halves with per-half timeouts
delock Sep 2, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 19 additions & 2 deletions .github/workflows/cpu-torch-latest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,16 @@ jobs:
runs-on: ubuntu-24.04

env:
# The runner is single-socket, so CPU_Accelerator.device_count() reports 1
# NUMA node and the per-device gate in tests/unit/common.py skips every
# multi-rank test. CPU ranks are plain processes over gloo, so advertise 4
# local devices to let world_size<=4 tests run. The test harness re-sets
# LOCAL_SIZE per worker, so this value only affects the launch gate.
LOCAL_SIZE: '4'

@chatgpt-codex-connector chatgpt-codex-connector Bot Sep 1, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Keep CUDA-only distributed tests out of the CPU run

In the cpu-torch-latest job, advertising four devices admits every distributed test with world_size <= 4, not only CPU-safe tests. For example, tests/unit/ulysses_alst/test_ulysses_sp_hf.py:240-263 defines an unguarded two-rank test that creates tensors on cuda:<rank>; because CPU_Accelerator.is_available() returns true, the harness does not skip it, and the all-unit pytest invocation at workflow line 283 will fail on the CPU-only PyTorch installation. Scope this override to an explicitly CPU-compatible subset or add CPU capability checks before enabling the previously skipped tests.

Useful? React with 👍 / 👎.

@chatgpt-codex-connector chatgpt-codex-connector Bot Sep 1, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Add the required sign-off trailer

This is a non-merge commit, but its commit message has no Signed-off-by trailer. Add the author sign-off so the commit satisfies the repository's commit and CI requirements.

AGENTS.md reference: AGENTS.md:L8-L8

Useful? React with 👍 / 👎.

# Multi-rank tests churn mp pools for hours; reused pools eventually hang in
# cleanup and stall workers until the 6h job limit. Fresh pools per test are
# slower but let the suite finish (knob documented in tests/unit/common.py).
DS_DISABLE_REUSE_DIST_ENV: '1'
DEFAULT_TORCH_PRESET: '2.10.0-cpu'
DEFAULT_TRANSFORMERS_SOURCE: 'git'
# Manual PyPI fallback only; scheduled and default manual runs use Git.
Expand Down Expand Up @@ -274,5 +284,12 @@ jobs:
run: |
unset TORCH_CUDA_ARCH_LIST # only jit compile for current arch
cd tests
HF_HOME=/tmp/hf_home/ pytest $PYTEST_OPTS --forked -n 4 unit/ --torch_ver="$TORCH_TEST_VERSION"
HF_HOME=/tmp/hf_home/ pytest $PYTEST_OPTS --forked -m 'sequential' unit/ --torch_ver="$TORCH_TEST_VERSION"
# The suite is split so each half gets fresh xdist workers: multi-rank pool
# teardown eventually wedges a worker, and one 3400-test process never
# reaches its summary inside the 6h job limit. maxfail is raised so every
# failure is listed, and timeout caps each half.
overall=0
timeout 150m pytest $PYTEST_OPTS --maxfail=100000 --forked -n 4 unit/ --ignore=unit/v1 --torch_ver="$TORCH_TEST_VERSION" || overall=$?
timeout 150m pytest $PYTEST_OPTS --maxfail=100000 --forked -n 4 unit/v1 --torch_ver="$TORCH_TEST_VERSION" || overall=$?
HF_HOME=/tmp/hf_home/ pytest $PYTEST_OPTS --forked -m 'sequential' unit/ --torch_ver="$TORCH_TEST_VERSION" || overall=$?
exit $overall
Loading