Repository navigation
refactor(sampling): share one schedule-shift fit across flow-matching families - #9938
Conversation
… families FLUX.1, Krea-2 Raw, Qwen-Image, Qwen-Image-2.1 and Z-Image evaluate their resolution-dependent mu through ShiftFit; checkpoint scheduler configs load through load_scheduler. Recorded mu and sigma schedules match the previous per-family code bit for bit.
LTX-2 holds its fit at both anchors through ShiftFit. mu between the anchors moves by at most one ulp; float32 sigmas are unchanged.
Pfannkuchensack
left a comment
There was a problem hiding this comment.
Approving. One request before merge: please restore one _run_diffusion-driven Qwen-Image explicit-shift test (see inline).
Old vs. new numerics
I compared the merge base (b3c6364) with this head in separate processes, comparing values as float.hex on CPU:
- FLUX.1: mu for 1–70,000 tokens.
get_scheduleat 32 token counts × 12 step counts, plus every token count from 1 to 20,000 at 30 steps. - Z-Image: shift for 1–70,000 tokens and the
_get_sigmasgrid. Explicit shifts 0, 0.5, 1, 3, 25.28 and 103777. The real_run_diffusionfrom 256 to 4096 px, with the entry sigma read back from the img2img blend. - Krea-2 Raw and Qwen-Image: the real
_run_diffusionwith the realFlowMatchEulerDiscreteScheduler. Five configs: noscheduler/folder, an empty folder, a full checkpoint config, a custom config withshift_terminal, and a config with missing keys. 15 sizes from 16² to 4096², at two step counts. Krea-2 with shift None and 0; Qwen-Image with shift None, 0.7 and 3.0. - Qwen-Image 2.1: mu for 1–70,000 tokens, and
build_sigmasfor Base, Turbo and None. - LTX-2: mu for 1–70,000 tokens.
build_sigmasfor every token count from 1 to 20,000 at 8, 30 and 50 steps, and the refine schedules.
Everything is bit-identical except LTX-2's mu. There, 618 token counts strictly between 1024 and 4096 move by 1 ulp; nothing at or outside the anchors moves, and every float32 LTX-2 sigma schedule is unchanged. That matches the description. The holds from #9887, #9892 and #9902 are kept exactly.
The tests pin the numbers well. These changes all make the focused tests fail:
- changing any family's anchor or shift constant
- turning off
hold_below_baseor removing the upper hold - making
from_scheduler_configread the wrong key - making
load_schedulerignore the checkpoint config - making Krea-2 or Qwen-Image fall back to the stock fit
Locally, ruff check, ruff format --check and the focused tests pass.
Conflict in settings.ts
The conflict with #9934 is textual only: #9934 added the shiftedSigmas helper right above the comment you changed. Keep main's block as is and take your `z_image_denoise.Z_IMAGE_SHIFT_FIT` comment. With that resolution:
- the focused tests,
test_upscale_index_clipped_starts.pyandtest_frontend_graph_node_types.pypass on the merged tree; - nothing on main still references a removed helper;
- the Krea-2 and Qwen-Image comments #9934 added there still point to the right files.
Nits
- Compatibility note. Please add a line under Compatibility / Rollout for the removed backend helpers. None of them are in
invocation_api, and anything that still uses them fails loudly, but custom-node authors should know:flux.sampling_utils.time_shiftkeeps its name but changes signature from(mu, sigma, t)to(shift, t).get_scheduledropsbase_shift/max_shift.- Removed:
get_lin_function,MU_FIT_*, Krea-2'scalculate_shiftandKREA2_*shift constants, LTX-2'scalculate_shiftandLTX2_{BASE,MAX}_*, and Qwen-Image 2.1'scalculate_mu.
- PR description. Since #9934, the follow-up ("the frontend Upscale feature keeps its own copies of the Z-Image and Krea-2 shift") also covers Qwen-Image's
qwenImageSigmas.
…eduler Restores the run-to-scheduler helper so a node-level test checks that the shift input reaches set_timesteps as ln(shift).
Summary
Several flow-matching families compute their resolution-dependent schedule shift (mu) with the same two-anchor line fit:
Each family had its own copy, in backend helpers, invocation methods or inline code, and the "hold the shift at the end of its fit" fix had to land in every copy separately.
invokeai/backend/rectified_flow/schedule_shift.pynow owns:ShiftFit), including holding at the upper anchor and, for LTX-2, at both anchors;load_scheduler). Keys a checkpoint omits take diffusers' defaults, as before.Each family now declares only its anchors and form. The per-family helpers are removed. Qwen-Image's node gets a small
_scheduleseam that owns its config precedence, explicit-shift override and token count.CogView4 (square-root form), FLUX.2 (fixed mu), Anima, Wan, SD3 and ERNIE are unchanged.
Commits:
QA Instructions
uv run --no-sync pytest tests/backend/rectified_flow tests/backend/flux tests/backend/ltx2 tests/backend/qwen_image_2_1 tests/app/invocations/test_krea2_denoise.py tests/app/invocations/test_qwen_image_denoise.py tests/app/invocations/test_z_image_schedule_shift.py tests/app/invocations/test_denoise_noise_inputs.pypasses.-n 4): 15002 passed. The failures were the 112 model-identification and model-path tests, which need git-lfs fixtures that were not installed on the test machine, and fail the same way onmain.Equivalence check. These results are one-off evidence, not a committed test:
FlowMatchEulerDiscreteScheduler.scheduler_config.json, both complete and missing keys. Qwen-Image explicit shifts 0.7 and 3.0. Z-Image explicit shifts 0, 0.5, 3 and 25.28.float.hex) after the refactor. A separate comparison against a reimplementation of the previous code confirmed mu for every token count from 1 to 70000, and FLUX schedules for steps up to 50.calculate_shiftdoes. LTX-2 used to evaluatex·slope + base − slope·base_seqleft to right.Review
Resolved:
ATEN_CPU_CAPABILITY=default._run_diffusionto the scheduler, so droppingself.shiftat the call site fails a test.Remaining follow-up: the frontend Upscale feature keeps its own copies of the Z-Image, Krea-2 and Qwen-Image shift (
qwenImageSigmas) for its start-index clipping.Compatibility / Rollout
invocation_api, and remaining callers fail loudly, but custom nodes that import them need updating:flux.sampling_utils.time_shiftkeeps its name, but its signature changes from(mu, sigma, t)to(shift, t).get_scheduledrops itsbase_shift/max_shiftarguments.get_lin_function,MU_FIT_*, Krea-2'scalculate_shiftandKREA2_*shift constants, LTX-2'scalculate_shiftandLTX2_{BASE,MAX}_*, Qwen-Image 2.1'scalculate_mu.Checklist
What's Newcopy (if doing a release after this PR)Need help on this PR? Tag
@codesmith-botwith what you need.