Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

Add support for DeepSeek #73

Merged
merged 1 commit into from
Jul 24, 2024
Merged

Add support for DeepSeek #73

merged 1 commit into from
Jul 24, 2024

Conversation

Saibo-creator
Copy link
Collaborator

fix issue #72

  • Exclude special tokens in ByteTrie initialization.
  • Handle deepseek-ai/deepseek-coder as GPT2TokenizerMiddleMapping.
  • Add get_special_token_ids_to_excluded to TCFG_Tokenizer and extend in TCFG_LlamaTokenizer to support deepseek
  • Added unit tests for deepseek-ai/deepseek-coder-1.3b-base.

@Saibo-creator
Add support for DeepSeek
ccbffa4 
- Exclude special tokens in  initialization.
- Handle  as .
- Add  to  and extend in  to support deepseek
- Added unit tests for .
@Saibo-creator Saibo-creator merged commit de95d73 into epfl-dlab:main Jul 24, 2024
@Saibo-creator Saibo-creator mentioned this pull request Jul 24, 2024
Closed
# for free to join this conversation on GitHub. Already have an account? # to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@Saibo-creator