Skip to content

build: compile the extension modules in parallel (SBD_BUILD_JOBS) - #96

Merged
Sophia Wen (hfwen0502) merged 1 commit into
mainfrom
parallel-build-jobs
Nov 13, 2026
Merged

Sophia Wen (hfwen0502) merged 1 commit into
mainfrom
parallel-build-jobs

Conversation

@hfwen0502

Copy link
Copy Markdown
Member

Each extension module is one large compile of src/sbd/bindings.cpp, and setup.py build_ext built them one at a time, leaving the build host nearly idle. They now compile in parallel.

  • SBD_BUILD_JOBS sets how many modules compile at once. Leave it unset to build all of them in parallel, capped by the CPU count. Set a smaller number, or 1, on a build host or container short of memory: a Thrust module's compile alone peaks near 4 GB. An explicit build_ext -j N still takes precedence, and an invalid value stops the build with an error.
  • Each module builds in its own temp directory. Every module compiles the same source file with different defines, and setuptools would place all their objects at one path. In parallel they would overwrite each other, and a module could silently link another's code, such as a UHF module without -D_UHF.
  • tox.ini passes SBD_BUILD_JOBS through, and INSTALL.md documents it. CI is unchanged and uses the default.

Build time

Host Modules Serial (SBD_BUILD_JOBS=1) Parallel (default)
8× H100 server, NVHPC 5 (CPU, Thrust and OpenMP offload, plus the UHF builds) 617 s 166 s
macOS laptop, clang 2 (CPU and CPU UHF) 24 s 13 s

Checked

  • Each module built in parallel reports the flags of its own build (__sbd_uhf__, __sbd_offload_target__, __sbd_has_evaluator__, __sbd_has_caop__), so none linked another's object.
  • The single-process test suite passes on the cpu, gpu and gpu-omp backends built in parallel.
  • An invalid SBD_BUILD_JOBS (0, x) stops the build with an error.

🤖 Generated with Claude Code

Each module is one large compile of src/sbd/bindings.cpp, so building them one at
a time left the machine nearly idle. They now compile in parallel, all at once by
default up to the CPU count; SBD_BUILD_JOBS sets how many, for build hosts short of
memory. Every module compiles the same source file, so each gets its own temp
directory: in a shared one their objects would overwrite each other and a module
could link another's code. tox passes SBD_BUILD_JOBS through.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@hfwen0502
Sophia Wen (hfwen0502) merged commit 1db6bc2 into main Nov 13, 2026
15 checks passed
@hfwen0502
Sophia Wen (hfwen0502) deleted the parallel-build-jobs branch November 13, 2026 00:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant