Skip to content

Pull requests: huggingface/tokenizers

New pull request New
21 Open 694 Closed
21 Open 694 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Reviews
Filter by reviews
No reviews Review required Approved review Changes requested
Assignee
Filter by who’s assigned
Assigned to nobody Loading
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Pull requests list

[docs] Whitespace
#1785 opened May 27, 2025 by stevhliu Loading…
1
Add Truncate pre-tokenizer
#1783 opened May 27, 2025 by ArthurZucker • Draft
Add benchmark for deserializing large added vocab + optimizations
#1782 opened May 27, 2025 by ArthurZucker • Draft
1
Update decode stream api
#1780 opened May 27, 2025 by ArthurZucker Loading…
1
Update __init__.pyi: fix 525: SyntaxWarning: invalid escape sequence '\w'
#1764 opened Apr 18, 2025 by wyattscarpenter Loading…
1
Make unigram cache optional
#1763 opened Apr 18, 2025 by wangrunji0408 Loading…
2
Implement from_bytes and read_bytes Methods in WordPiece Tokenizer for WebAssembly Compatibility
#1758 opened Mar 31, 2025 by sondalex Loading…
3
Itertools upgrade
#1756 opened Mar 25, 2025 by sftse Loading…
4
Implement Append normalizer
#1755 opened Mar 24, 2025 by austinleedavis Loading…
1
Pre-tokenizers that support multi-word/non-whitespace BPE in single pass
#1753 opened Mar 22, 2025 by mjbommar Loading…
2
Switch to FXHash
#1752 opened Mar 19, 2025 by MeetThePatel Loading…
13
Add FxHash and ShortStringOptimization.
#1733 opened Feb 10, 2025 by MeetThePatel Loading…
3 of 4 tasks
5
Add with_sequence for decode stream
#1725 opened Jan 21, 2025 by ArthurZucker • Draft
3
Does windows aarch work ?
#1719 opened Jan 10, 2025 by Narsil Loading…
1
Fixed Length Pre-Tokenizer
#1713 opened Jan 5, 2025 by jonvet Loading…
1
5
Draft backtrack
#1712 opened Jan 3, 2025 by ArthurZucker • Draft
2
Bpe clones
#1707 opened Dec 28, 2024 by sftse Loading…
4
[WIP] free speed/mem optimizations with ahash, dary_heap, and compact_str
#1618 opened Aug 21, 2024 by mjbommar Loading…
5
🚨 breaking: Fix training with special tokens
#1617 opened Aug 19, 2024 by ArthurZucker Loading…
1
2
Fast regex
#1605 opened Aug 8, 2024 by ArthurZucker • Draft
3
[Feature] support Assign token to update the content of a token Stale
#1570 opened Jul 12, 2024 by ArthurZucker • Draft
1 task
1
7
ProTip! What’s not been updated in a month: updated:<2025-04-27.