Skip to content

Stable 25.8 Backport of #116836 - jemalloc: lower lg_extent_max_active_fit to default 6, to reduce fragmentation #2538

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
mkmkme wants to merge 1 commit into
base: stable-25.8
Choose a base branch
from
Open

Stable 25.8 Backport of #116836 - jemalloc: lower lg_extent_max_active_fit to default 6, to reduce fragmentation #2538

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 6 additions & 11 deletions contrib/jemalloc-cmake/CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,18 +35,13 @@ if (OS_LINUX)
# MADV_DONTNEED. See
# https://github.com/ClickHouse/ClickHouse/issues/11121 for motivation.
#
# lg_extent_max_active_fit:8 controls how jemalloc reuses memory blocks.
# Value 8 means extents can be reused when their size is up to 2^8 (256) times larger than requested.
# This enables better hot dirty memory reuse for two-level hashtables with 256 sub-hashtables.
# In ClickBench queries (e.g. Q35), these 256 sub-hashtables (16KB each) get reallocated frequently.
# Without this setting, when multiple 16KB blocks coalesce into larger blocks (>64*16KB),
# they can't be reused for new 16KB requests, causing expensive page faults from MADV_DONTNEED.
# Increasing from default 6(64x) to 8(256x) allows these coalesced blocks to be split and reused.
# See: https://github.com/ClickHouse/ClickHouse/pull/80245
# https://github.com/jemalloc/jemalloc/pull/2842
set (JEMALLOC_CONFIG_MALLOC_CONF "percpu_arena:percpu,oversize_threshold:0,muzzy_decay_ms:0,dirty_decay_ms:5000,prof:true,prof_active:false,background_thread:true,lg_extent_max_active_fit:8")
# Use jemalloc's default maximum extent reuse ratio, 2^6 (64).
# Previously increased to 8 for ClickBench Q35, but that can
# significantly increase fragmentation.
# https://github.com/ClickHouse/ClickHouse/pull/116836
set (JEMALLOC_CONFIG_MALLOC_CONF "percpu_arena:percpu,oversize_threshold:0,muzzy_decay_ms:0,dirty_decay_ms:5000,prof:true,prof_active:false,background_thread:true,lg_extent_max_active_fit:6")
else()
set (JEMALLOC_CONFIG_MALLOC_CONF "oversize_threshold:0,muzzy_decay_ms:0,dirty_decay_ms:5000,lg_extent_max_active_fit:8")
set (JEMALLOC_CONFIG_MALLOC_CONF "oversize_threshold:0,muzzy_decay_ms:0,dirty_decay_ms:5000,lg_extent_max_active_fit:6")
endif()
# CACHE variable is empty to allow changing defaults without the necessity
# to purge cache
Expand Down
Loading