Skip to content

Issues: huggingface/tokenizers

ByteLevelBPETokenizer output seems weird
#203 opened Mar 24, 2020 by seyyaw
Open 2
Training a model from in-memory data
#198 by loicbarrault was closed Nov 28, 2020
Closed 1
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

43 Open 956 Closed
43 Open 956 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

Reduce vocab size for BPE tokenizer Feature Request
#1668 opened Oct 29, 2024 by fzyzcjy
8
Incremental Detokenization Feature Request
#1666 opened Oct 26, 2024 by robertgshaw2-neuralmagic
9
docs-check.yml uses node12 which is deprecated
#1658 opened Oct 17, 2024 by hamirmahal
1
1
Allow users to select/write encoding strategies Feature Request
#1655 opened Oct 16, 2024 by pietrolesci
2
Serializing k-mer style pre-tokenizer
#1654 opened Oct 15, 2024 by millanp95
Inconsistent behaviour of PreTrainedTokenizerFasts on diacritics marked texts bug Something isn't working
#1663 opened Oct 11, 2024 by sven-nm
2 of 4 tasks
1
Disable pretty-print when saving tokenizer.json files Feature Request
#1656 opened Oct 7, 2024 by xenova
1
How to build a custom tokenizer on top of a exsiting Llama 3.2 tokenizer? training
#1644 opened Oct 5, 2024 by yakhyo
7
Adding tokens to a tokenizer with subword support?
#1637 opened Sep 27, 2024 by noamgat
1
NormalizedString.clear() broken? bug Something isn't working
#1636 opened Sep 25, 2024 by lkurlandski
1
Adding many AddedTokens makes loading a tokenizer extremely slow.
#1635 opened Sep 25, 2024 by stephantul
4
Cannot inject custom PreTokenizer into Tokenizer
#1634 opened Sep 23, 2024 by Old-Shatterhand
6
README.md contains non-functional code
#1633 opened Sep 19, 2024 by ahenkes1
2
Access utf-8 byte sequence for each token
#1628 opened Sep 9, 2024 by DanielHesslow
2
Rust: How to handle models with precompiled_charsmap = null Feature Request
#1627 opened Sep 4, 2024 by kallebysantos
5
Tokenizer Quickstart Tutorial: Broken Links
#1625 opened Sep 3, 2024 by SinaMostafanejad
Special token gets tokenized while training tokenizer from scratch
#1624 opened Sep 2, 2024 by LalchandPandia
1
ModuleNotFoundError: No module named 'tokenizers.tokenizers'
#1619 opened Aug 25, 2024 by jpferraro1
6
BPE trainer ignoring special tokens.
#1616 opened Aug 16, 2024 by henrycharlesworth
1
3
.NET bindings
#1615 opened Aug 16, 2024 by sappho192
Space after unnormalized token is added when use_fast=True for Llama tokenizer
#1613 opened Aug 14, 2024 by Butanium
10
RefMutContainer is unsound
#1612 opened Aug 13, 2024 by CheaterCodes
3
[test-infra] Enable Codecov for tokenizers
#1611 opened Aug 12, 2024 by hvaara
Support for Golang now or support a cli for other languages?
#1601 opened Aug 7, 2024 by xuxiaoxia96
2
ValueError: The following model_kwargs are not used by the model: ['num_beans'] (note: typos in the generate arguments will also show up in this list)[06/Aug/2024 14:34:35]
#1596 opened Aug 6, 2024 by navdeep8990
1
ProTip! Exclude everything labeled bug with -label:bug.