Repository navigation
fix(z-image): dequantize GGUF pad tokens stored as a 2-D row - #9942
Open
Pfannkuchensack wants to merge 1 commit into
Open
Pfannkuchensack wants to merge 1 commit into
Pfannkuchensack wants to merge 1 commit into
Conversation
GGUF files that store x_pad_token/cap_pad_token as [dim, 1] kept them packed, so every generation failed in torch.where.
5 of 7 tasks
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Z-Image GGUF transformers whose pad tokens are stored as a 2-D row (
x_pad_token/cap_pad_tokenwith GGUF shape[3840, 1], e.g.z-image-turbo-Q4_K_S.gguf) could not generate at all: every run failed in diffusers'_prepare_sequencewithTypeError: Multiple dispatch failed for 'torch._ops.aten.where.self' … GGMLTensor. The GGUF key converter only dequantized pad tokens that were 1-D, so the 2-D ones stayed packed. Pad tokens of any single-row shape are now dequantized and reshaped to(1, dim); a pad token that is not a single row is refused with a clear error instead of being reshaped.Related Issues / Discussions
Found while testing Z-Image ControlNets on a GGUF base.
QA Instructions
RTX 4090,
z-image-turbo-Q4_K_S.gguf+ Qwen3-4B + FLUX VAE, 8 steps: before, every generation failed with the error above (768×1344 and 1024², with and without ControlNet); after, both sizes generate normally.Tests: a CPU end-to-end test writes a tiny GGUF with the released file's layout (Q8_0 layers, BF16 refiners/embedders, F32 norms, pad tokens F32
[dim, 1]), loads it through the real GGUF loader and compares the forward with a dense reference (fails with the same TypeError before the fix; zeroed or scrambled pad tokens also fail it); unit tests for 1-D,(dim,)and(1, dim)pad tokens and the refusal.pytest tests/backend/model_manager/load -k "z_image or gguf"andtests/backend/quantization/gguf: passed. Ruff clean.Review
No material findings. Noted for follow-up, unchanged here: the Z-Image GGUF loader does not unpack small GGML tensors at load (other GGUF loaders do), and splitting the fused qkv keeps those weights dequantized in memory.
Checklist
What's Newcopy (if doing a release after this PR)Need help on this PR? Tag
@codesmith-botwith what you need.