Describe the situation
Under amd_msan, clickhouse-client aborts while base64-encoding image output. The test's own checks all pass — it is marked FAIL only because the abort reaches stderr.
04327_png_terminal_mode: [ FAIL ] Reason: having stderror:
stdout: iterm: OK / kitty: OK / sixel: OK / auto: OK / BAD_ARGUMENTS
Two aborts, two call sites, both on a stack buffer encoded passed to libbase64:
Uninitialized bytes in write at offset 29 inside [.., 143)
created by an allocation of 'encoded' #0 DB::Base64WriteBuffer::nextImpl() src/IO/Base64WriteBuffer.cpp:34
Uninitialized bytes in write at offset 17 inside [.., 132)
created by an allocation of 'encoded' #0 DB::writeImageKitty(...) src/Formats/PNGTerminalOutput.cpp:157
Evidence
1. The result follows the CPU. Flags from Available CPU instruction sets in each job's clickhouse-server.log; runner from the job log header.
All 7 failures were on CPUs without AVX512VBMI; all 10 passes were on CPUs with it. Jobs whose S3 artefact was overwritten by a later attempt are excluded, since the CPU flags would belong to a different attempt.
The runner pool is a proxy, not the cause: the four standby runners sampled all lack VBMI; the ephemeral pool draws mixed hardware and its result follows the CPU.
2. Same binary, different machine. Reproduced locally on a VBMI-less CPU (Intel Core 7 240H) with a binary from a failing CI run and with one from a passing run — both abort.
3. The output is byte-for-byte correct. Same machine, same query, comparing the raw PNG against the decoded base64:
raw PNG (terminal_mode=auto) : 84 bytes sha256 672225eaa704acf7
base64 decoded (mode=iterm) : 84 bytes sha256 672225eaa704acf7 -> identical
Nothing is missing. The reported offsets are also exactly where the base64 output begins (iTerm prefix 29 bytes, kitty prefix 17) — no byte written by ClickHouse itself is flagged.
AI-generated analysis
Everything above is measured. What follows is AI-generated analysis and a set of options; it has not been validated by applying any of them.
libbase64 picks its codec at runtime and takes the AVX-512 path only when the CPU has both VL and VBMI — lib/codec_choose.c:
if ((ebx & bit_AVX512vl) && (ecx & bit_AVX512vbmi)) {
codec->enc = base64_stream_encode_avx512;Without VBMI it falls back to the AVX2/SSSE3 codec, whose hand-written SIMD MSan cannot follow, so it believes the output buffer was never written.
Our build configuration matches upstream 26.6 (cmake/sanitize.cmake, the MSan cmake flags, the contrib/base64 submodule commit, both source files, the Dockerfiles). Upstream does not see this because their runners have VBMI.
Options
__msan_unpoison(encoded, encoded_size) after the libbase64 call, at both sites. The project's existing pattern — already used in 107 places, e.g. __msan_unpoison(ciphertext_and_tag, out_len); /// OpenSSL uses assembly which evades msan's analysis. base/base/MemorySanitizer.h makes it a no-op on non-MSan builds. Fixes it everywhere, including upstream.- Pin the msan jobs to VBMI-capable runners. Not a config change: the job uses
FUNC_TESTER_AMD = ["self-hosted", "altinity-on-demand", "altinity-func-tester"], and no existing label describes the CPU, so a new label would have to be added to the machines first. Also shrinks the pool.
broken_tests.yaml entry restricted to msan — stops the noise, hides the cause.no-msan tag on the test — stronger than 3, but drops coverage on the half of the fleet where the test works.
Note Base64WriteBuffer is generic, so any future caller that base64-encodes output hits the same false positive on VBMI-less hardware.
Describe the situation
Under
amd_msan,clickhouse-clientaborts while base64-encoding image output. The test's own checks all pass — it is marked FAIL only because the abort reaches stderr.Two aborts, two call sites, both on a stack buffer
encodedpassed to libbase64:Evidence
1. The result follows the CPU. Flags from
Available CPU instruction setsin each job'sclickhouse-server.log; runner from the job log header.hetznerhetznerhetznerstandbystandbystandbystandbyhetznerhetznerhetznerhetznerhetznerhetznerhetznerhetznerhetznerhetznerAll 7 failures were on CPUs without AVX512VBMI; all 10 passes were on CPUs with it. Jobs whose S3 artefact was overwritten by a later attempt are excluded, since the CPU flags would belong to a different attempt.
The runner pool is a proxy, not the cause: the four
standbyrunners sampled all lack VBMI; the ephemeral pool draws mixed hardware and its result follows the CPU.2. Same binary, different machine. Reproduced locally on a VBMI-less CPU (Intel Core 7 240H) with a binary from a failing CI run and with one from a passing run — both abort.
3. The output is byte-for-byte correct. Same machine, same query, comparing the raw PNG against the decoded base64:
Nothing is missing. The reported offsets are also exactly where the base64 output begins (iTerm prefix 29 bytes, kitty prefix 17) — no byte written by ClickHouse itself is flagged.
AI-generated analysis
libbase64 picks its codec at runtime and takes the AVX-512 path only when the CPU has both VL and VBMI —
lib/codec_choose.c:Without VBMI it falls back to the AVX2/SSSE3 codec, whose hand-written SIMD MSan cannot follow, so it believes the output buffer was never written.
Our build configuration matches upstream 26.6 (
cmake/sanitize.cmake, the MSan cmake flags, thecontrib/base64submodule commit, both source files, the Dockerfiles). Upstream does not see this because their runners have VBMI.Options
__msan_unpoison(encoded, encoded_size)after the libbase64 call, at both sites. The project's existing pattern — already used in 107 places, e.g.__msan_unpoison(ciphertext_and_tag, out_len); /// OpenSSL uses assembly which evades msan's analysis.base/base/MemorySanitizer.hmakes it a no-op on non-MSan builds. Fixes it everywhere, including upstream.FUNC_TESTER_AMD = ["self-hosted", "altinity-on-demand", "altinity-func-tester"], and no existing label describes the CPU, so a new label would have to be added to the machines first. Also shrinks the pool.broken_tests.yamlentry restricted to msan — stops the noise, hides the cause.no-msantag on the test — stronger than 3, but drops coverage on the half of the fleet where the test works.Note
Base64WriteBufferis generic, so any future caller that base64-encodes output hits the same false positive on VBMI-less hardware.