Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

Add Mamba PR #83

Merged
merged 60 commits into from
Mar 4, 2024
Merged

Add Mamba PR #83

merged 60 commits into from
Mar 4, 2024

Conversation

3outeille
Copy link
Member

No description provided.

@3outeille
fix: llama import
abafcb9
@3outeille
feat: at leading loading mamba properly
e8b8314
@3outeille
loss going down
d9513b0
@3outeille
refacto: init mamba weights
147fd25
@3outeille
fix: dtype for brrr compatibility
3e4dda1
@3outeille
fix: bring back strict union to fix NanotronConfigs + use torch.dtype…
bebbf6f 
… in config
@3outeille
fix: revert back to casting str to torch dtype
ad384b9
@3outeille
fix
02d21de
@3outeille
fix: mismatch params counting due to no tie embedding weight
1168bfe
@3outeille
fix: run_generate is now compatible with Brrr
7bfc56c
@3outeille
chore: single file mamba slow (for residual fix)
1fe1cd4
@3outeille
fix: residual for slow mamba
720df8e
@3outeille
fix(mamba-slow): transpose embedding
7902959
@3outeille
update(mamba-slow)
b2ca553
@3outeille
fix: unified random unit
3c4ef42
@3outeille
feat(optimizer): can now apply weight decay to specifcs parameters
afcbb10
@3outeille
Merge branch 'mamba' into mamba-update
7216b11
@3outeille
refacto: mamba1
a6a163f
@3outeille
feat: add selective scan interface
ad17887
@3outeille
feat: make fast path compatible with TP
6bfd6aa
@3outeille
refacto: no split weight
f0e219b
@3outeille
refacto: cleaner init weights
ae312fe
@3outeille
refacto: init weights
ff26ff5
@3outeille
feat: save/load _no_weight_decay attribute for Mamba model
0ab7c9b
@3outeille
clean import
3caf7fe
@3outeille
clean utils in mamba
ec6f954
@3outeille
cleaning initialization for mamba
1773420
@3outeille
no sync conv and rmsnorm for mamba
6fc0d9a
@3outeille
Merge branch 'mamba-update' into mamba-pr
6e8fd1f
@3outeille
clean run_train
fa6953d
xrsrke
xrsrke requested changes Mar 3, 2024
View reviewed changes
NouamaneTazi
NouamaneTazi reviewed Mar 4, 2024
View reviewed changes
@3outeille 3outeille marked this pull request as ready for review March 4, 2024 12:52
xrsrke
xrsrke requested changes Mar 4, 2024
View reviewed changes
Copy link
Member

@xrsrke xrsrke left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. some last changes before merging

NouamaneTazi
NouamaneTazi approved these changes Mar 4, 2024
View reviewed changes
Copy link
Member

@NouamaneTazi NouamaneTazi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Very cleaaaan 🐍🚀

@3outeille
Copy link
Member Author

thank you @NouamaneTazi and @xrsrke for taking the time to review the PR ❤️

@NouamaneTazi NouamaneTazi merged commit 5e128fc into huggingface:main Mar 4, 2024
3 checks passed
# for free to join this conversation on GitHub. Already have an account? # to comment
Reviewers

@xrsrke xrsrke xrsrke requested changes

@NouamaneTazi NouamaneTazi NouamaneTazi approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@3outeille @xrsrke @NouamaneTazi