【Hackathon 7th No.43】完善 TokenizerFast 功能支持 part 1 #9407
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thanks for your contribution! |
yinfan98
changed the title
|
yinfan98
force-pushed
the
tokenizer_fast
branch
from
451d07d to
6d95920
Compare
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## develop #9407 +/- ##
===========================================
- Coverage 53.08% 52.96% -0.12%
===========================================
Files 687 689 +2
Lines 109472 109412 -60
===========================================
- Hits 58114 57952 -162
- Misses 51358 51460 +102 ☔ View full report in Codecov by Sentry. |
yinfan98
force-pushed
the
tokenizer_fast
branch
from
5f1403e to
7161b60
Compare
|
cc：@DrownFish19 麻烦再帮我看下pr吧，感谢 |
DrownFish19
reviewed
Nov 26, 2024
| @@ -0,0 +1,131 @@ | |||
| # Copyright (c) 2024 PaddlePaddle Authors. All Rights Reserved. | |||
| # | |||
There was a problem hiding this comment.
辛苦在这里增加一下HuggingFace的Copyright
| ("bloom", "BloomTokenizer"), | ||
| ( | ||
| "bloom", | ||
| ("BloomTokenizer", "BloomTokenizerFast" if is_tokenizers_available() else None), |
| "LlamaTokenizer": LlamaConverter, | ||
| "BertTokenizer": BertConverter, | ||
| } | ||
| SLOW_TO_FAST_CONVERTERS = {"LlamaTokenizer": LlamaConverter, "BertTokenizer": BertConverter} |
There was a problem hiding this comment.
这里的convert是可以通用吗？后续可以验证一下
There was a problem hiding this comment.
这里应该没有新加bloom的，因为我看在hf上bloom只有fast，没有convert的流程
# for free
to join this conversation on GitHub.
Already have an account?
# to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR types
Others
PR changes
Models
Description
为Bloom提供tokenizer fast支持，顺便想问一下。是对test里每个def我都要添加一个fast的测试吗～感谢🙏