Repository navigation
fix(deps): update dependency transformers to v5.22.1 - #59
Merged
Merged
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #59 +/- ##
=======================================
Coverage 82.72% 82.72%
=======================================
Files 71 71
Lines 3218 3218
=======================================
Hits 2662 2662
Misses 556 556 🚀 New features to boost your workflow:
|
renovate
Bot
force-pushed
the
renovate/transformers-5.x
branch
2 times, most recently
from
October 16, 2026 15:00
3b0baea to
5b81a43
Compare
renovate
Bot
changed the title
renovate
Bot
changed the title
renovate
Bot
force-pushed
the
renovate/transformers-5.x
branch
2 times, most recently
from
October 26, 2026 22:05
8306b4a to
846d7b0
Compare
renovate
Bot
changed the title
renovate
Bot
force-pushed
the
renovate/transformers-5.x
branch
from
October 29, 2026 10:27
846d7b0 to
8657295
Compare
renovate
Bot
changed the title
renovate
Bot
force-pushed
the
renovate/transformers-5.x
branch
from
October 29, 2026 19:03
8657295 to
2b7b4fb
Compare
renovate
Bot
changed the title
renovate
Bot
changed the title
renovate
Bot
force-pushed
the
renovate/transformers-5.x
branch
2 times, most recently
from
November 2, 2026 01:56
7eedaea to
04af6c2
Compare
renovate
Bot
force-pushed
the
renovate/transformers-5.x
branch
from
November 2, 2026 09:30
04af6c2 to
41d10fa
Compare
stefano81
enabled auto-merge (rebase)
November 3, 2026 09:24
stefano81
disabled auto-merge
November 3, 2026 09:25
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Note
This PR body was truncated due to platform limits.
This PR contains the following updates:
==5.19.0→==5.22.1Release Notes
huggingface/transformers (transformers)
v5.22.1: Patch release v5.22.1Compare Source
Patch release v5.22.1
We just had to fix the documentation!
** [Fix] Remove WIP doc for Ml4 (#50124) from @remi-or
v5.22.0: : Mistral Large 4 (Le Chonk)Compare Source
New Models
Mistral Large 4 (Le Chonk)
Mistral Large 4 is Mistral's largest and most capable model to date: a 1 trillion-parameter natively multimodal Mixture-of-Experts model with 49 billion active parameters per token (52 billion including embeddings and output layers), trained from scratch on 3,800 NVIDIA Grace Blackwell GPUs in Mistral's own datacenters in Europe.
Mistral Large 4 unifies instruction following, reasoning, coding, and agentic workflows in a single set of weights. While a generalist model, it shows especially good performance on workloads such as cybersecurity, finance, and law, and it surpasses frontier closed models on visual grounding. It is natively fluent in 160+ languages, including every official language of the European Union.
Key Features
Mistral Large 4 includes the following architectural choices:
Mistral Large 4 offers the following capabilities:
low,medium,high(default) — so the same model can answer a quick chat reply or work through a complex agentic run.The model is released under the Mistral Ecosystem Open-Weight License: a modified version of the MIT license, for both commercial and non-commercial use — operating a Model-as-a-Service business requires a separate commercial license with Mistral.
Recommended Settings
'low'/'medium'/'high'— default is'high'.reasoning_effort="high"for complex prompts and agentic usage,'low'for fast chat replies and'medium'for day to day usage.temperature=1.0top_p=0.95Breaking changes
Quantization
FP8 quantized models can now be used for adapter training (such as LoRA with PEFT). Backward passes were added to the dense and grouped FP8 matmuls, and
is_trainableis nowTrue. Backward also dequantizes experts in chunks and skips autograd at inference, which cuts peak memory in the MoE layer backward from about 28.9 GiB to 2.8 GiB at 8k tokens. Separately, the AMD CI image now installBugfixes and improvements
PermuteForRopeto act on single tensors while renaming (#50092) by @Cyrilvallez in [#50092]rope_scalingsetter for configs that assign it during init (#50080) by @hmellor in [#50080]git blameon a partial clone (#50007) by @nrenvo in [#50007]Significant community contributions
The following contributors have made significant changes to the library over the last release:
_init_weightsgaps in 7 audio models and enabletest_can_init_all_missing_weightsfor them (#49386)v5.21.1: Patch release v5.21.1Compare Source
Patch release v5.21.1
** Fix rope_scaling setter for configs that assign it during init (#50080 )
We broke remote code loading in vllm, this fixes it sorry everyone!
v5.21.0Compare Source
v5.20.2: Patch release v5.20.2Compare Source
Patch release: v5.20.2
This patch notably solves an issue where some inputs could loose their original type inside
prepare_inputs_for_generationingenerate, leading to problems with multimodal models. It also fixes some other issues listed on the below commits.It contains the following commits:
v5.20.1Compare Source
Release v5.20.1
Special surprise including the new model Beam from Reflection AI 🥳
Beam is a token-efficient, frontier-scale model focused on coding and agentic use cases. It is built for teams that want to run a powerful model on their own hardware, infrastructure, or cloud, and customize it to their specific needs.
Links: Documentation
Some real patch fixes
We broke loading models on older torch versions 😢 should be fixed now. But here is the full list of fixes
Full Changelog: huggingface/transformers@v5.20.0...v5.20.1
v5.20.0Compare Source
Release v5.20.0
New Model additions
Muse Spark Open
Muse Spark Open is a multimodal mixture-of-experts model from Meta that takes text, images, audio, and video as input and generates text. It has 816B total parameters (42B active per token) and a context window of up to 1,048,576 tokens shared across modalities. It is trained for agentic coding tasks such as multi-file editing, debugging, and tool use. Checkpoints are available in BF16 and NVFP4 formats.
Links: Documentation | Blog Post
Ling 3.0
Ling 3.0 Flash is a hybrid Mixture-of-Experts language model from InclusionAI. Its decoder groups five Kimi Delta Attention (KDA) layers with one Multi-head Latent Attention (MLA) layer. The MLA blocks use interleaved rotary embeddings and a head-wise output gate, and the feed-forward blocks combine routed and shared experts. It is natively supported in Transformers as
BailingHybridForCausalLM, so it loads withouttrust_remote_code=True.Links: Documentation
Kolibri1
Kolibri 1 is Aleph Alpha's mixture-of-experts reasoning model, focused on German and English. Each of its 50 layers routes every token to 6 of 384 experts using sigmoid routing, and adds one shared expert. Four sliding-window attention layers alternate with one full-attention layer, and RoPE is applied only in the sliding-window layers.
Links: Documentation
MiniCPM-V 4.7
MiniCPM-V is a series of efficient multimodal large language models developed by OpenBMB. Like MiniCPM-V 4.6, the MiniCPM-V 4.7 architecture pairs a SigLIP vision encoder that has a window-attention merger with a Qwen3.5 language model backbone, and supports both 4x and 16x visual downsampling modes.
The main addition over 4.6 is canvas M-RoPE: instead of numbering visual tokens along a single 1-D sequence, the model lays every image out on a 2-D canvas and assigns each visual token a
(temporal, height, width)position, so slices of the same image keep their spatial relationship and video frames keep their temporal order.UnlimitedOcr
UnlimitedOcr is a single 3B-parameter OCR model from Baidu with a standard 32,768-token context length. It builds on DeepSeek OCR and replaces all decoder attention layers with Reference Sliding Window Attention (R-SWA). This keeps the KV cache constant during decoding, so the model can transcribe dozens of pages in a single forward pass. It supports "gundam" mode (640x640 tiles with dynamic cropping) and "base" mode (a single 1024x1024 view), and it can output region detections with bounding boxes.
Links: Documentation | Paper
UltraBERT
UltraBERT is a family of bidirectional text encoders trained with a combination of masking ratios, pruning, and distillation for classification and retrieval tasks. The released encoders support sequences of up to 32,768 tokens. They use rotary position embeddings, RMSNorm, SwiGLU feed-forward layers, and alternating local and global attention (every third layer is global). Checkpoints include UltraBERT-150M and UltraBERT-380M for masked language modeling and fine-tuning, and UltraBERT-Embed-150M for text embeddings.
Links: Documentation | Paper
MOSS-Transcribe-Diarize
MOSS-Transcribe-Diarize 0.9B is an end-to-end audio understanding model for long-form multi-speaker transcription, speaker diarization, and timestamps. It combines a Whisper-style encoder, a 4× frame merge step, a multi-modal projector, and a Qwen3 language model. It produces joint transcription and diarization in a single pass, with segments formatted as
[start][S01]text[end]. It supports long-form audio through Whisper-window chunking in the processor, promptable transcription with hotword biasing, and 50+ languages.Links: Documentation
Breaking changes
🚨 General breaking notice that inference with EP has regressions in speed; we are working on fixing this ASAP! 🚨
Deprecated checks and methods have been removed from
cache_utils.py(affecting models such as minimax_m3_vl and pi0), so any code still calling those deprecated cache methods must switch to the current cache API.Accidental imports from other models were removed from several model files (e.g. embedding_gemma2, ijepa, lightglue), and the modular converter now avoids adding such cross-model imports. If you imported these names through those model modules, import them from their original modules instead.
The minimum
kernelsversion is now 0.17.2, which allowsuse_fallback=KernelizeFallback.ALLso that a failed kernel load falls back to the original implementation instead of raising an error. Upgradekernelsto 0.17.2 or later.A new
SubConfigSpeccentralizes the initialization of sub-configs, so default sub-configs are collected from dataclass fields instead of by instantiating the class, and this touches the configuration utilities and many model configs. Custom or existing nested configs may need to be updated to declare their sub-configs through this mechanism.The LUKE and mLUKE tokenizers now treat entity special tokens such as
<ent>as special tokens (not split, honored byskip_special_tokens, normalized) while keeping the same spacing as regular vocabulary tokens when decoding. Tokenization and decoding output involving these tokens may therefore differ from before, so check any code or saved outputs that depend on the old behavior.Exporters
Transformers exporters now support quantization directly during export across Dynamo, ONNX, OpenVINO, and ExecuTorch. The new unified quantization API integrates PT2E, ONNX Runtime, and NNCF quantizers, enabling static and dynamic quantization as well as weight-only compression (INT8/INT4), with configurable calibration workflows. This makes it easier to export optimized models for efficient inference across different backends.
For more information see the docs: https://huggingface.co/docs/transformers/exporters_quantization
Parallelization
This release improves distributed training and inference. FSDP × TP model and optimizer checkpoints can now be saved and loaded across different parallel configurations, and expert-parallel dispatch, tensor-parallel plans, static-cache
generate(), and the caching allocator's byte count were fixed. Tied LM heads in composite models now get their own TP plan entry,enable_expert_parallelapplies the EP plan alone again, TPU loading uses the `torGeneration
This release fixes static-cache
generate()bugs with host-side inputs and under tensor parallelism. It also speeds up Whisper generation on TPU by cutting device syncs and recompilation, and fixes Whisper batched timestamp decoding for tensor and NumPy inputs. Pipelines get a cleanup of generation-argument handling that removes spurious warnings.Vision
Vision processing was updated in several places. Image decoding now uses torchcodec, Molmo2's image and video kwargs follow the library's standard conventions, and Pixtral Flash Attention no longer detects false image boundaries after the axial RoPE refactor, which had caused CUDA illegal memory access errors and incorrect output. Several CI and Docker changes also shipped: GPU images moved to Python 3.11, a numpy/pyarrow conflict
Kernels
This release improves kernel support across hardware.
flash_attention_2now falls back to the AITER kernel on ROCm whenflash-attnisn't installed, and T5-style models get a contiguous SDPA position-bias mask so they use fused kernels again. The Hub rotary kernel is now limited to CUDA inference because its backward pass gave incorrect gradients. The Atlas Inference Gated DeltaNet kernels for Strix Halo andCache
This release fixes several cache-related bugs: static-cache
generate()now handles host-side nested inputs and tensor parallelism correctly, the offloaded cache prefetch no longer races with the offloading copy, and Phi3 LongRoPE cache rebuilds at the context boundary now receive the full prefix. It also counts the query token in MiMo-V2-Flash's sliding window and corrects the quantized cache (hqq/quanto) documentation examBugfixes and improvements
torch.mm(..., out_dtype=...)to ONNX (#49635) by @IlyasMoutawwakil in [#49635]MtpModelhelpers for frameworks that own the drafting loop (#49544) by @hmellor in [#49544]create_chunked_causal_mask()crashing withblock_sequence_ids(#49457) by @kaixuanliu in [#49457]Configuration
📅 Schedule: (UTC)
🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.
♻ Rebasing: Whenever PR is behind base branch, or you tick the rebase/retry checkbox.
🔕 Ignore: Close this PR and you won't be reminded about this update again.
This PR was generated by Mend Renovate. View the repository job log.