[MOD-17526][MOD-17527] Make SQ8 metadata exact and fix the symmetric L2 formulation - #1011

Open

dor-forer wants to merge 17 commits into

mainfrom

dor-forer-MOD-17526-sq8-exact-metadata

Commits on Aug 10, 2026

Make SQ8 metadata exact and fix the symmetric L2 and IP formulations

dor-forer
and
claude
committed
6a4803a
Update the SQ8 test fixtures for the exact integer metadata

dor-forer
and
claude
committed
2b84cc2
Give the query-metadata assertions their own baseline

dor-forer
and
claude
committed
50a395e
Sum the quantized bytes in the reference quantizer too

dor-forer
and
claude
committed
e405dd7
Fix the NaN-to-uint8 UB, and pin all four defects with regressions

dor-forer
and
claude
committed
5ad4f03
Loosen a sanity bound to the quantization residue

dor-forer
and
claude
committed
a82713b

Commits on Aug 12, 2026

fix(uint8): widen and unsign the integer accumulators
dor-forer
committed
b711c0c
refactor(sq8): drop MAX_EXACT_DIM
dor-forer
committed
43250bf
perf(sq8): avoid a libm round call per element when quantizing
dor-forer
committed
c090e15
fix(sq8): make the symmetric L2 cancellation exact
dor-forer
committed
f6780aa
perf(sq8): force-inline the reconstruction helpers
dor-forer
committed
6292913
test(sq8): add regression tests for the exact-metadata contract
dor-forer
committed
983c742
style: drop the blank line left by removing the MAX_EXACT_DIM assert
dor-forer
committed
bfbcd90
chore: remove .sites.txt
dor-forer
committed
2b66b5f
test: free the preprocessor in QuantizationHandlesNonRepresentableRange
dor-forer
committed
9594c8a

Commits on Aug 13, 2026

fix(sq8): close the two remaining paths to NaN in quantize

dor-forer
and
claude
committed
6c8d6e0
docs(sq8): state the quantize numerics once instead of per line

dor-forer
and
claude
committed
729f621