Skip to content

Stable GCC/libgomp thread team size across parallel regions - #1397

Open
pascoec wants to merge 2 commits into
devfrom
issue1394
Open

pascoec wants to merge 2 commits into
devfrom
issue1394

Conversation

@pascoec

@pascoec pascoec commented Nov 11, 2026 •

Copy link
Copy Markdown
Collaborator

Closes #1394.

TL;DR

  • GCC builds keep the libgomp thread team across parallel regions instead of ending and re-creating threads whenever the width changes: CKKS and functional bootstrapping 0.85× at 36 threads and 0.86× at 72, lib-benchmark 0.88–0.90×, for 1.2–2.9× the CPU seconds in idle threads. clang builds use libomp's own pool and compile this part out.
  • The default thread count on Linux is the number of cores among the processors the process may run on, read from the CPU topology: one thread per core on machines with simultaneous multithreading, also under taskset or a container CPU set; OMP_NUM_THREADS and SetNumThreads() override it. On a 16-thread laptop whose VM presents 8 cores, the core unit-test suite goes from 17.5 to 3.0 s and the pke suite under load from 1805 to 611 s.
  • Two team-width constants (THREADS_SCALE_TO_POLY 8 → 16, APPROX_CRT_BASIS_SWITCH_WORK_PER_THREAD 2^17 → 2^16) give mid-sized basis switches more threads: BGV and CKKS multiplication with relinearization at depths 8–12 0.87–0.94× under gcc and 0.76–0.82× under clang; nothing changes at 8 threads or fewer.

Every parallel region in the library asks for the number of threads its work can use (towers, digits, rotations), and under libgomp a region narrower than the previous one ends the surplus threads while a wider one creates new ones, at 30 to 50 µs per thread. A CKKS bootstrap runs thousands of regions of alternating width, so a large share of its time at high thread counts went into resizing the team. This PR keeps the team at its recent width, defaults to one thread per core on machines with simultaneous multithreading, widens two team caps that were sized for the resizing behaviour, fixes the error distribution of key-switching key generation in OpenMP builds, and documents thread counts and shared machines.

The PR is two commits, the change (2de94fd) and the review follow-ups (e7576ce); their messages carry the details and their own measurements.

Changes

  • Key-switching key errors. The key generation loops of KeySwitchBV and KeySwitchHYBRID declared the discrete Gaussian generator private(dgg), which default-constructs a per-thread copy with standard deviation 1.0; OpenMP builds therefore sampled key-switching, relinearization and rotation keys with sigma 1.0 instead of the parameters' 3.19 (since v1.4.2.1; builds without OpenMP were correct). firstprivate(dgg) copies the configured generator. Key generation takes 3 to 5% longer and key switching adds the noise it was meant to add.
  • Thread team kept across regions. ParallelControls::GetThreadLimit keeps the calling thread's top-level team at the widest width recently requested; narrower regions run on it, split across the whole team when they have the iterations and otherwise with the idle threads waiting at the closing barrier, and the team shrinks to the widest of the narrower requests after 256 consecutive ones. A site's request is therefore the team's minimum width under libgomp while a wider team is kept. GCC builds only, whose omp.h is libgomp's; libomp keeps its own pool, and clang builds compile the code out. ClampToThreadLimit is the plain clamp for the sites whose count sizes the work split or must be exact (binfhe accumulators and LWE key switch, BV task count). Four regions in three functions that picked numTasks or numTowers by comparing with the thread count now ask for numTasks; the single-key cases of EvalAutomorphismKeyGenAtLevels and MultiEvalAutomorphismKeyGen no longer open a one-thread region (libgomp creates fresh threads for every region nested in one).
  • Default thread count. On Linux, when OMP_NUM_THREADS is unset, the thread limit starts at the number of distinct cores among the processors in the process's affinity mask (core_id and physical_package_id from the CPU topology in sysfs), so taskset, cpusets and hybrid CPUs are counted as they are; Enable() restores it and sets OpenMP's own thread count back to the machine count, GetDefaultThreads() reports it, SetNumThreads() still accepts any count up to the machine's, and OMP_NUM_THREADS is honoured as set. Machines without SMT, and systems without the topology files, keep every thread OpenMP reports.
  • Team widths. THREADS_SCALE_TO_POLY 8 → 16 and APPROX_CRT_BASIS_SWITCH_WORK_PER_THREAD 2^17 → 2^16: the ScaleAndRound overloads that return a native polynomial and the mid-sized ApproxSwitchCRTBasis/ApproxModDown conversions get more threads. No change at 8 threads or fewer.

Measurements

Ice Lake (2 × 36 cores, 2.4 GHz, SMT off), gcc 14, WITH_NATIVEOPT=OFF, libgomp; the kept team against dev 6ff73c6 (before #1393, which is independent of this change):

36 threads, one socket (before → after) 72 threads, both sockets (after / before)
CKKS bootstrap 2^16, 2^15 slots 2.263 → 1.899 s 0.88
SlotsToCoeffs-first, same 1.467 → 1.271 s 0.88
functional bootstrap, 1-bit LUT 0.629 → 0.492 s 0.77
functional bootstrap, 6-bit LUT 2.626 → 2.161 s 0.87
17 bootstrapping rows, geomean 0.854 (0.78–0.97) 0.864 (0.77–0.96)
lib-benchmark, 129 cases, geomean 0.897 0.877
poly-benchmark, binfhe gates 1.00 1.00

At 8 threads the rows are 0.997 and lib-benchmark 0.937 (0.882 on an 8-core i7-9700); at one thread everything is unchanged. The kept team costs CPU time where regions are narrower than it — 1.2 to 1.8 times the CPU seconds at 36 threads and 1.5 to 2.9 times at 72, with equal memory — and one wall-time loss, the in-place addition of a 4-tower ciphertext on a 16-wide team (20.7 → 22.9 µs at 36 threads). The two width changes add BFV MultNoRelin at depth 12 0.87–0.89, BGV MultRelin at depths 10–12 0.87–0.94 and CKKS MultRelin/EvalFastRotation at depth 10 0.96–0.97 on top, at 8 to 10% more CPU time in lib-benchmark.

On the FLEXIBLEAUTO rows of the retuned functional-bootstrapping benchmarks of #1384 (its head applied to dev with #1393 merged, gcc 14, 36 threads on one socket, one process per case, mean of 4 calls, median of two rounds), seconds per call without → with this PR:

row without with speedup
FBT FA 1 / 2 / 4 / 6-bit 0.577 / 1.008 / 1.356 / 2.360 0.433 / 0.836 / 1.114 / 1.891 1.33 / 1.21 / 1.22 / 1.25×
FBT FA 8 / 10 / 12-bit 6.79 / 14.63 / 33.05 6.34 / 13.86 / 32.45 1.07 / 1.06 / 1.02×
FE-FBT 29-bit FA, 1 level (identity … gelu) 0.604–0.689 0.516–0.591 1.12–1.18×
FE-FBT 29-bit FA, 6 levels 0.781–1.130 0.674–1.024 1.10–1.16×
FE-FBT 59-bit FA, 1 level 1.920–2.323 1.642–1.963 1.17–1.20×
FE-FBT 59-bit FA, 6 / 10 levels 4.608–6.255 4.433–6.037 1.02–1.05×

The gain follows the share of time outside the LUT polynomial: largest on small LUTs and one-level FE rows, a few percent where the polynomial dominates. The same trees built with clang 18 link LLVM's libomp, under which the kept team is compiled out; the 1 / 2 / 4 / 6-bit FBT rows there go 0.477 / 0.838 / 1.119 / 2.019 → 0.478 / 0.809 / 1.081 / 1.849 s (1.00 / 1.04 / 1.04 / 1.09×) from the remaining changes. That libomp baseline is 10–17% faster than the libgomp one, and the libgomp build with this PR matches or beats it. lib-benchmark under clang, 10 alternating rounds per case at 36 and 72 threads: BGV and CKKS MultRelin at depths 8–12 0.76–0.82, EvalFastRotation at depths 8–12 0.91–0.94, BFV MultNoRelin at depth 12 0.93–0.95, the small operations (additions, rescaling, relinearization, modulus switching, encryption, decryption, key generation) 1.006 with every case within 0.985–1.028, geomean 0.97 over the 63 multiplication and rotation cases, 1.00 at 8 threads, and no case slower beyond its run-to-run spread. The same tree with the two width constants at their old values measures 1.00 against dev under clang, so under libomp the two constants are the whole effect, and they pay more there than under libgomp.

Unit-test suites on an i7-1270P laptop (12 cores, 16 threads; the WSL2 virtual machine presents 8 cores with two threads each, so the default is 8; gcc 13), wall time, OMP_NUM_THREADS unset unless stated:

suite busy processes dev default (16 threads) this PR default (8) dev at 8 this PR at 16
core 0 17.5 s 3.0 s 4.0 s 5.8 s
core 8 54.5 s 8.5 s 7.4 s 56.4 s
binfhe 0 37.0 s 27.7 s 24.4 s 31.2 s
pke 0 398 s 350 s 345 s 327 s
pke 8 1805 s 611 s 1777 s 902 s

The first column is today's behaviour on any SMT laptop: per-region 16-thread teams on 8 cores. The default of this PR is never worse than it, and the last column shows why the cap matters for the kept team.

Shared machines: both runtimes spin idle threads for a while before sleeping, and a wide team on a loaded machine waits for the scheduler at every region; with a 12-thread job on the same socket the 2^16 bootstrap takes 3.8–4.5 s instead of 1.7 s, which OMP_WAIT_POLICY=passive brings back to 2.2 s. The library cannot set that policy (it is read at runtime start-up), so it is documented, together with keeping OMP_NUM_THREADS one or two below the core count on a dedicated machine.

Every parallel region of the library asks for a number of threads. Under libgomp
a region narrower than the previous one ends the surplus threads and a wider one
creates new threads. At 30 to 50 us per thread, a CKKS bootstrap running thousands
of regions with alternating widths encoutered high overhead. Now,
ParallelControls::GetThreadLimit() keeps the calling thread's top-level team
at the widest width recently requested: a request at least as wide resets
the team to it, a narrower request runs on the current team, and the team
shrinks to the widest of the narrower requests after 256 consecutive ones
(TEAM_SHRINK_REGIONS). Threads without an iteration wait at the region's
closing barrier. The rule exists in GCC builds, whose omp.h is libgomp's
(_LIBGOMP_OMP_LOCK_DEFINED); libomp keeps its own pool across regions, and
clang builds compile it out. The width is tracked per thread and only at
nesting level 0, so regions nested in a user's parallel loop are unchanged.
ClampToThreadLimit is the plain clamp for the sites whose thread count sizes
the work split or must be exact: the binfhe accumulators and LWE key switch,
and the BV key-switch task count. Three regions picked their width with
(numTasks < GetNumThreads()) ? numTasks : numTowers; they now ask for
numTasks. The single-key cases of EvalAutomorphismKeyGenAtLevels and
MultiEvalAutomorphismKeyGen call the generator directly instead of opening a
one-thread region, since libgomp creates fresh threads for every region
nested in a one-thread region.

On Linux, when /sys/devices/system/cpu/smt/active reads 1 and OMP_NUM_THREADS
is not set, the thread limit starts at omp_get_num_procs() / 2, one thread
per core. Enable() restores that default, GetDefaultThreads() reports it,
and SetNumThreads() still accepts any count up to the machine's.
OMP_NUM_THREADS, when set, is taken as is. Other systems, and kernels
without the file, start with every thread OpenMP reports.

THREADS_SCALE_TO_POLY, the team width of the ScaleAndRound overloads that
return a native polynomial, goes from 8 to 16, the width of the basis
switches next to them. APPROX_CRT_BASIS_SWITCH_WORK_PER_THREAD goes from 2^17
to 2^16 inner iterations, so ApproxSwitchCRTBasis and ApproxModDown reach
their 36-thread cap at half the work: conversions of 0.5 to 4.7 million
iterations get twice the threads, for example 8 -> 13 for a 3-tower digit
against 9 towers at N = 2^15 and 11 -> 22 for 4 against 11. Builds limited
to 8 threads or fewer are unchanged.

Ice Lake (2 x 36 cores, 2.4 GHz), gcc 14, the kept team measured against
dev 6ff73c6, 36 threads on one socket, before -> after:
  CKKS bootstrap 2^16, 2^15 slots           2.263 -> 1.899 s
  SlotsToCoeffs-first, same                 1.467 -> 1.271 s
  functional bootstrap, 1-bit LUT           0.629 -> 0.492 s
  functional bootstrap, 6-bit LUT           2.626 -> 2.161 s
  functional bootstrap, 6-bit FIXEDMANUAL   3.155 -> 2.801 s
  17 bootstrapping rows, geomean            0.854
  lib-benchmark, 129 cases, geomean         0.897
72 threads across both sockets: rows 0.864, lib-benchmark 0.877. 8 threads:
rows 0.997, lib-benchmark 0.937 (0.882 on an 8-core i7-9700). One thread:
unchanged. poly-benchmark and binfhe gates: unchanged at every thread count.
The kept team costs CPU time where regions are narrower than it: 1.2 to 1.8
times the CPU seconds at 36 threads and 1.5 to 2.9 times at 72, with equal
memory; the one wall-time loss is the in-place addition of a 4-tower
ciphertext on a 16-wide team, 20.7 -> 22.9 us at 36 threads and 22.4 -> 27.1
us at 72.

The two team widths on top of that, lib-benchmark, before -> after at 36
and 72 threads, gcc 14:
  BFV MultNoRelin depth 12       12.19 -> 10.63 ms    13.08 -> 11.68 ms
  BGV MultRelin depth 10         11.02 -> 10.27 ms    12.52 -> 10.83 ms
  BGV MultRelin depth 12         11.13 -> 10.48 ms    11.50 -> 10.67 ms
  CKKS MultRelin depth 10        10.54 -> 10.11 ms    11.04 -> 10.62 ms
  CKKS EvalFastRotation depth 10  5.10 ->  4.95 ms     5.45 ->  5.26 ms
and clang 18 with libomp, where the kept team compiles out, medians of 10
rounds:
  BGV MultRelin depth 10         10.38 ->  8.35 ms    10.50 ->  8.11 ms
  BGV MultRelin depth 12         10.07 ->  8.11 ms    10.22 ->  8.21 ms
  CKKS MultRelin depth 10        10.12 ->  8.02 ms    10.28 ->  7.94 ms
  BGV EvalFastRotation depth 10   4.52 ->  4.22 ms     4.61 ->  4.24 ms
  CKKS EvalFastRotation depth 10  4.22 ->  3.96 ms     4.27 ->  3.99 ms
  BFV MultNoRelin depth 12       10.42 -> 10.10 ms    10.88 -> 10.74 ms
Other cases within 3% under gcc and 5% under clang; CKKS bootstrapping and
functional bootstrapping unchanged; lib-benchmark CPU time 8 to 10% higher
under gcc.

Unit-test suites on an 8-core, 16-thread laptop (i7-1270P, gcc 13), default
thread count before -> after, that is 16 -> 8 threads, wall time (CPU time):
  core    idle                 17.5 s   (219)  ->   3.0 s   (10)
          8 busy processes     54.5 s   (402)  ->   8.5 s   (32)
  binfhe  idle                 37.0 s   (388)  ->  27.7 s  (176)
          8 busy processes     63.6 s   (453)  ->  67.6 s  (352)
  pke     idle                  398 s  (2405)  ->   350 s (2362)
          8 busy processes     1805 s (11507)  ->   611 s (3774)
The loaded binfhe cell swings by a factor of 2 between rounds on this laptop.
@pascoec pascoec added this to the Release 1.6.2 milestone Nov 11, 2026
@pascoec
pascoec requested a review from yspolyakov November 11, 2026 06:42
@pascoec pascoec self-assigned this Nov 11, 2026
@pascoec pascoec added the optimization Improves performance label Nov 11, 2026
@pascoec pascoec linked an issue Nov 11, 2026 that may be closed by this pull request
…omp team within a lowered limit

The default thread count was omp_get_num_procs() / 2 whenever the kernel
reported simultaneous multithreading as active, which halves the threads of
a process whose affinity mask already holds one thread per core (taskset, a
container's cpuset) and counts the cores of a hybrid CPU wrong.
ParallelControls::DefaultThreads now counts the distinct (package, core)
pairs of the processors in the process's affinity mask from the Linux CPU
topology (sched_getaffinity, core_id and physical_package_id in sysfs), and
falls back to every thread OpenMP reports when the topology is unreadable.
On a 16-thread laptop that presents 8 cores with two threads each: 8;
taskset -c 0-5 (three cores): 3; a mask of six distinct cores: 6;
OMP_NUM_THREADS=16: 16. On a 72-core machine without SMT: 72, 18 under
numactl --cpunodebind=0 (an 18-core node) and 6 under taskset -c 0-5.

TeamWidth clamped the kept team to a lowered thread limit but kept the
widest narrower request recorded before the limit changed, so the shrink
after TEAM_SHRINK_REGIONS narrower regions could restore a width above the
limit. The state is reset to the limit when the team exceeds it.

Enable() set OpenMP's own thread count to the default thread count, while
construction leaves it at the machine thread count, so an application's own
parallel regions changed width after Disable(); Enable(). Enable() now
restores both values of the start state: the library's limit to the default
thread count and omp_set_num_threads to the machine thread count.

UnitTestParallel.cpp pins the clamp, the default thread count with and
without OMP_NUM_THREADS, Enable(), and under libgomp the keep, reset and
shrink of the kept team and the lowered limit.

The documentation of GetThreadLimit and Best_Performance.md state that a
narrower region runs on the kept team and a loop with more iterations than
it requested is split across all of it, so under libgomp a request is the
team's minimum width while a wider team is kept. The comment above the
ring-dimension team widths is removed.
@pascoec pascoec added the bug Something isn't working label Nov 19, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working optimization Improves performance

Projects

None yet

Development

Successfully merging this pull request may close these issues.

speed up openmp performance

1 participant