Repository navigation
Conversation
Every parallel region of the library asks for a number of threads. Under libgomp a region narrower than the previous one ends the surplus threads and a wider one creates new threads. At 30 to 50 us per thread, a CKKS bootstrap running thousands of regions with alternating widths encoutered high overhead. Now, ParallelControls::GetThreadLimit() keeps the calling thread's top-level team at the widest width recently requested: a request at least as wide resets the team to it, a narrower request runs on the current team, and the team shrinks to the widest of the narrower requests after 256 consecutive ones (TEAM_SHRINK_REGIONS). Threads without an iteration wait at the region's closing barrier. The rule exists in GCC builds, whose omp.h is libgomp's (_LIBGOMP_OMP_LOCK_DEFINED); libomp keeps its own pool across regions, and clang builds compile it out. The width is tracked per thread and only at nesting level 0, so regions nested in a user's parallel loop are unchanged. ClampToThreadLimit is the plain clamp for the sites whose thread count sizes the work split or must be exact: the binfhe accumulators and LWE key switch, and the BV key-switch task count. Three regions picked their width with (numTasks < GetNumThreads()) ? numTasks : numTowers; they now ask for numTasks. The single-key cases of EvalAutomorphismKeyGenAtLevels and MultiEvalAutomorphismKeyGen call the generator directly instead of opening a one-thread region, since libgomp creates fresh threads for every region nested in a one-thread region. On Linux, when /sys/devices/system/cpu/smt/active reads 1 and OMP_NUM_THREADS is not set, the thread limit starts at omp_get_num_procs() / 2, one thread per core. Enable() restores that default, GetDefaultThreads() reports it, and SetNumThreads() still accepts any count up to the machine's. OMP_NUM_THREADS, when set, is taken as is. Other systems, and kernels without the file, start with every thread OpenMP reports. THREADS_SCALE_TO_POLY, the team width of the ScaleAndRound overloads that return a native polynomial, goes from 8 to 16, the width of the basis switches next to them. APPROX_CRT_BASIS_SWITCH_WORK_PER_THREAD goes from 2^17 to 2^16 inner iterations, so ApproxSwitchCRTBasis and ApproxModDown reach their 36-thread cap at half the work: conversions of 0.5 to 4.7 million iterations get twice the threads, for example 8 -> 13 for a 3-tower digit against 9 towers at N = 2^15 and 11 -> 22 for 4 against 11. Builds limited to 8 threads or fewer are unchanged. Ice Lake (2 x 36 cores, 2.4 GHz), gcc 14, the kept team measured against dev 6ff73c6, 36 threads on one socket, before -> after: CKKS bootstrap 2^16, 2^15 slots 2.263 -> 1.899 s SlotsToCoeffs-first, same 1.467 -> 1.271 s functional bootstrap, 1-bit LUT 0.629 -> 0.492 s functional bootstrap, 6-bit LUT 2.626 -> 2.161 s functional bootstrap, 6-bit FIXEDMANUAL 3.155 -> 2.801 s 17 bootstrapping rows, geomean 0.854 lib-benchmark, 129 cases, geomean 0.897 72 threads across both sockets: rows 0.864, lib-benchmark 0.877. 8 threads: rows 0.997, lib-benchmark 0.937 (0.882 on an 8-core i7-9700). One thread: unchanged. poly-benchmark and binfhe gates: unchanged at every thread count. The kept team costs CPU time where regions are narrower than it: 1.2 to 1.8 times the CPU seconds at 36 threads and 1.5 to 2.9 times at 72, with equal memory; the one wall-time loss is the in-place addition of a 4-tower ciphertext on a 16-wide team, 20.7 -> 22.9 us at 36 threads and 22.4 -> 27.1 us at 72. The two team widths on top of that, lib-benchmark, before -> after at 36 and 72 threads, gcc 14: BFV MultNoRelin depth 12 12.19 -> 10.63 ms 13.08 -> 11.68 ms BGV MultRelin depth 10 11.02 -> 10.27 ms 12.52 -> 10.83 ms BGV MultRelin depth 12 11.13 -> 10.48 ms 11.50 -> 10.67 ms CKKS MultRelin depth 10 10.54 -> 10.11 ms 11.04 -> 10.62 ms CKKS EvalFastRotation depth 10 5.10 -> 4.95 ms 5.45 -> 5.26 ms and clang 18 with libomp, where the kept team compiles out, medians of 10 rounds: BGV MultRelin depth 10 10.38 -> 8.35 ms 10.50 -> 8.11 ms BGV MultRelin depth 12 10.07 -> 8.11 ms 10.22 -> 8.21 ms CKKS MultRelin depth 10 10.12 -> 8.02 ms 10.28 -> 7.94 ms BGV EvalFastRotation depth 10 4.52 -> 4.22 ms 4.61 -> 4.24 ms CKKS EvalFastRotation depth 10 4.22 -> 3.96 ms 4.27 -> 3.99 ms BFV MultNoRelin depth 12 10.42 -> 10.10 ms 10.88 -> 10.74 ms Other cases within 3% under gcc and 5% under clang; CKKS bootstrapping and functional bootstrapping unchanged; lib-benchmark CPU time 8 to 10% higher under gcc. Unit-test suites on an 8-core, 16-thread laptop (i7-1270P, gcc 13), default thread count before -> after, that is 16 -> 8 threads, wall time (CPU time): core idle 17.5 s (219) -> 3.0 s (10) 8 busy processes 54.5 s (402) -> 8.5 s (32) binfhe idle 37.0 s (388) -> 27.7 s (176) 8 busy processes 63.6 s (453) -> 67.6 s (352) pke idle 398 s (2405) -> 350 s (2362) 8 busy processes 1805 s (11507) -> 611 s (3774) The loaded binfhe cell swings by a factor of 2 between rounds on this laptop.
…omp team within a lowered limit The default thread count was omp_get_num_procs() / 2 whenever the kernel reported simultaneous multithreading as active, which halves the threads of a process whose affinity mask already holds one thread per core (taskset, a container's cpuset) and counts the cores of a hybrid CPU wrong. ParallelControls::DefaultThreads now counts the distinct (package, core) pairs of the processors in the process's affinity mask from the Linux CPU topology (sched_getaffinity, core_id and physical_package_id in sysfs), and falls back to every thread OpenMP reports when the topology is unreadable. On a 16-thread laptop that presents 8 cores with two threads each: 8; taskset -c 0-5 (three cores): 3; a mask of six distinct cores: 6; OMP_NUM_THREADS=16: 16. On a 72-core machine without SMT: 72, 18 under numactl --cpunodebind=0 (an 18-core node) and 6 under taskset -c 0-5. TeamWidth clamped the kept team to a lowered thread limit but kept the widest narrower request recorded before the limit changed, so the shrink after TEAM_SHRINK_REGIONS narrower regions could restore a width above the limit. The state is reset to the limit when the team exceeds it. Enable() set OpenMP's own thread count to the default thread count, while construction leaves it at the machine thread count, so an application's own parallel regions changed width after Disable(); Enable(). Enable() now restores both values of the start state: the library's limit to the default thread count and omp_set_num_threads to the machine thread count. UnitTestParallel.cpp pins the clamp, the default thread count with and without OMP_NUM_THREADS, Enable(), and under libgomp the keep, reset and shrink of the kept team and the lowered limit. The documentation of GetThreadLimit and Best_Performance.md state that a narrower region runs on the kept team and a loop with more iterations than it requested is split across all of it, so under libgomp a request is the team's minimum width while a wider team is kept. The comment above the ring-dimension team widths is removed.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1394.
TL;DR
tasksetor a container CPU set;OMP_NUM_THREADSandSetNumThreads()override it. On a 16-thread laptop whose VM presents 8 cores, the core unit-test suite goes from 17.5 to 3.0 s and the pke suite under load from 1805 to 611 s.THREADS_SCALE_TO_POLY8 → 16,APPROX_CRT_BASIS_SWITCH_WORK_PER_THREAD2^17 → 2^16) give mid-sized basis switches more threads: BGV and CKKS multiplication with relinearization at depths 8–12 0.87–0.94× under gcc and 0.76–0.82× under clang; nothing changes at 8 threads or fewer.Every parallel region in the library asks for the number of threads its work can use (towers, digits, rotations), and under libgomp a region narrower than the previous one ends the surplus threads while a wider one creates new ones, at 30 to 50 µs per thread. A CKKS bootstrap runs thousands of regions of alternating width, so a large share of its time at high thread counts went into resizing the team. This PR keeps the team at its recent width, defaults to one thread per core on machines with simultaneous multithreading, widens two team caps that were sized for the resizing behaviour, fixes the error distribution of key-switching key generation in OpenMP builds, and documents thread counts and shared machines.
The PR is two commits, the change (2de94fd) and the review follow-ups (e7576ce); their messages carry the details and their own measurements.
Changes
KeySwitchBVandKeySwitchHYBRIDdeclared the discrete Gaussian generatorprivate(dgg), which default-constructs a per-thread copy with standard deviation 1.0; OpenMP builds therefore sampled key-switching, relinearization and rotation keys with sigma 1.0 instead of the parameters' 3.19 (since v1.4.2.1; builds without OpenMP were correct).firstprivate(dgg)copies the configured generator. Key generation takes 3 to 5% longer and key switching adds the noise it was meant to add.ParallelControls::GetThreadLimitkeeps the calling thread's top-level team at the widest width recently requested; narrower regions run on it, split across the whole team when they have the iterations and otherwise with the idle threads waiting at the closing barrier, and the team shrinks to the widest of the narrower requests after 256 consecutive ones. A site's request is therefore the team's minimum width under libgomp while a wider team is kept. GCC builds only, whoseomp.his libgomp's; libomp keeps its own pool, and clang builds compile the code out.ClampToThreadLimitis the plain clamp for the sites whose count sizes the work split or must be exact (binfhe accumulators and LWE key switch, BV task count). Four regions in three functions that pickednumTasksornumTowersby comparing with the thread count now ask fornumTasks; the single-key cases ofEvalAutomorphismKeyGenAtLevelsandMultiEvalAutomorphismKeyGenno longer open a one-thread region (libgomp creates fresh threads for every region nested in one).OMP_NUM_THREADSis unset, the thread limit starts at the number of distinct cores among the processors in the process's affinity mask (core_idandphysical_package_idfrom the CPU topology in sysfs), sotaskset, cpusets and hybrid CPUs are counted as they are;Enable()restores it and sets OpenMP's own thread count back to the machine count,GetDefaultThreads()reports it,SetNumThreads()still accepts any count up to the machine's, andOMP_NUM_THREADSis honoured as set. Machines without SMT, and systems without the topology files, keep every thread OpenMP reports.THREADS_SCALE_TO_POLY8 → 16 andAPPROX_CRT_BASIS_SWITCH_WORK_PER_THREAD2^17 → 2^16: theScaleAndRoundoverloads that return a native polynomial and the mid-sizedApproxSwitchCRTBasis/ApproxModDownconversions get more threads. No change at 8 threads or fewer.Measurements
Ice Lake (2 × 36 cores, 2.4 GHz, SMT off), gcc 14,
WITH_NATIVEOPT=OFF, libgomp; the kept team against dev 6ff73c6 (before #1393, which is independent of this change):At 8 threads the rows are 0.997 and lib-benchmark 0.937 (0.882 on an 8-core i7-9700); at one thread everything is unchanged. The kept team costs CPU time where regions are narrower than it — 1.2 to 1.8 times the CPU seconds at 36 threads and 1.5 to 2.9 times at 72, with equal memory — and one wall-time loss, the in-place addition of a 4-tower ciphertext on a 16-wide team (20.7 → 22.9 µs at 36 threads). The two width changes add BFV
MultNoRelinat depth 12 0.87–0.89, BGVMultRelinat depths 10–12 0.87–0.94 and CKKSMultRelin/EvalFastRotationat depth 10 0.96–0.97 on top, at 8 to 10% more CPU time in lib-benchmark.On the FLEXIBLEAUTO rows of the retuned functional-bootstrapping benchmarks of #1384 (its head applied to dev with #1393 merged, gcc 14, 36 threads on one socket, one process per case, mean of 4 calls, median of two rounds), seconds per call without → with this PR:
The gain follows the share of time outside the LUT polynomial: largest on small LUTs and one-level FE rows, a few percent where the polynomial dominates. The same trees built with clang 18 link LLVM's libomp, under which the kept team is compiled out; the 1 / 2 / 4 / 6-bit FBT rows there go 0.477 / 0.838 / 1.119 / 2.019 → 0.478 / 0.809 / 1.081 / 1.849 s (1.00 / 1.04 / 1.04 / 1.09×) from the remaining changes. That libomp baseline is 10–17% faster than the libgomp one, and the libgomp build with this PR matches or beats it. lib-benchmark under clang, 10 alternating rounds per case at 36 and 72 threads: BGV and CKKS
MultRelinat depths 8–12 0.76–0.82,EvalFastRotationat depths 8–12 0.91–0.94, BFVMultNoRelinat depth 12 0.93–0.95, the small operations (additions, rescaling, relinearization, modulus switching, encryption, decryption, key generation) 1.006 with every case within 0.985–1.028, geomean 0.97 over the 63 multiplication and rotation cases, 1.00 at 8 threads, and no case slower beyond its run-to-run spread. The same tree with the two width constants at their old values measures 1.00 against dev under clang, so under libomp the two constants are the whole effect, and they pay more there than under libgomp.Unit-test suites on an i7-1270P laptop (12 cores, 16 threads; the WSL2 virtual machine presents 8 cores with two threads each, so the default is 8; gcc 13), wall time,
OMP_NUM_THREADSunset unless stated:The first column is today's behaviour on any SMT laptop: per-region 16-thread teams on 8 cores. The default of this PR is never worse than it, and the last column shows why the cap matters for the kept team.
Shared machines: both runtimes spin idle threads for a while before sleeping, and a wide team on a loaded machine waits for the scheduler at every region; with a 12-thread job on the same socket the 2^16 bootstrap takes 3.8–4.5 s instead of 1.7 s, which
OMP_WAIT_POLICY=passivebrings back to 2.2 s. The library cannot set that policy (it is read at runtime start-up), so it is documented, together with keepingOMP_NUM_THREADSone or two below the core count on a dedicated machine.