-
Notifications
You must be signed in to change notification settings - Fork 1.2k
Pull requests: huggingface/tokenizers
Author
Label
Projects
Milestones
Reviews
Assignee
Sort
Pull requests list
fix(ci): harden GitHub Actions workflows (#2119)
#2410
opened Sep 15, 2026 by
hf-security-analysis
Bot
Loading…
feat(python): add Arrow string batch tokenization
#2400
opened Sep 10, 2026 by
granthamtaylor
Loading…
bpe: do not slice merge tokens at a byte offset that may not be a boundary
#2398
opened Sep 10, 2026 by
apollo-2006
Loading…
Fix BERT normalization of CJK Extension E boundary characters
#2395
opened Sep 8, 2026 by
mottopanikeiku
Loading…
fix: Unigram prune_sentence_pieces uses alternatives[id].len() not alternatives.len()
#2394
opened Sep 8, 2026 by
VirajMishra1
Loading…
fix(python): forward unk_id in SentencePieceUnigramTokenizer
#2393
opened Sep 7, 2026 by
Con-Benksl
Loading…
fix(bpe): drop heap entries with non-positive pair_counts before u64 cast
#2389
opened Sep 3, 2026 by
okxint
Loading…
perf(decode): index the vocab slab by id, not through the MPHF slot
#2387
opened Sep 3, 2026 by
ArthurZucker
Collaborator
Loading…
fix: correctly decode numpy UCS-4 arrays instead of treating as UTF-8
#2386
opened Sep 3, 2026 by
VirajMishra1
Loading…
docs: rewrite the README for the crate split
#2385
opened Sep 2, 2026 by
ArthurZucker
Collaborator
Loading…
v1.0.0-rc0: the train/encode split, repacked into 53 commits
#2384
opened Sep 2, 2026 by
ArthurZucker
Collaborator
Loading…
perf(normalizers): fast-path BertNormalizer for non-CJK input
#2383
opened Sep 2, 2026 by
bsachart
Loading…
Fix UnicodeScripts pre-tokenizer dropping leading spaces
#2382
opened Sep 2, 2026 by
Kayvan-Zahiri
Loading…
perf(parallelism): default the pool to physical cores, not SMT siblings
#2380
opened Sep 2, 2026 by
ArthurZucker
Collaborator
Loading…
perf(train): select top-k in the trainers instead of ordering everything
#2379
opened Sep 2, 2026 by
ArthurZucker
Collaborator
Loading…
perf(word-cache): scan the tag window with shrn, not an emulated movemask
#2377
opened Sep 2, 2026 by
ArthurZucker
Collaborator
Loading…
perf(pre_tokenizers): eliminate per-token heap allocations in ByteLevel decoder
#2376
opened Sep 1, 2026 by
bsachart
Loading…
Fix BPE training across consecutive line endings
#2370
opened Aug 30, 2026 by
Alphaxiaoteng
Loading…
fix(python): preserve async encode character offsets
#2369
opened Aug 28, 2026 by
gry67673905
Loading…
Previous Next
ProTip!
Type g i on any issue or pull request to go back to the issue listing page.