Code

Add Mamba PR#83

Code

Merged

NouamaneTazi merged 60 commits intohuggingface:mainhuggingface/nanotron:mainfrom 3outeille:main3outeille/nanotron:mainCopy head branch name to clipboardMar 4, 2024

Commits

Commits on Jan 24, 2024

• fix: llama import

  3outeille

  committed
  abafcb9

  Copy full SHA for abafcb9

Commits on Feb 8, 2024

• feat: at leading loading mamba properly

  3outeille

  committed
  e8b8314

  Copy full SHA for e8b8314

• loss going down

  3outeille

  committed
  d9513b0

  Copy full SHA for d9513b0

• refacto: init mamba weights

  3outeille

  committed
  147fd25

  Copy full SHA for 147fd25

• fix: dtype for brrr compatibility

  3outeille

  committed
  3e4dda1

  Copy full SHA for 3e4dda1

• fix: bring back strict union to fix NanotronConfigs + use torch.dtype in config

  3outeille

  committed
  bebbf6f

  Copy full SHA for bebbf6f

• fix: revert back to casting str to torch dtype

  3outeille

  committed
  ad384b9

  Copy full SHA for ad384b9

• fix

  3outeille

  committed
  02d21de

  Copy full SHA for 02d21de

• fix: mismatch params counting due to no tie embedding weight

  3outeille

  committed
  1168bfe

  Copy full SHA for 1168bfe

• fix: run_generate is now compatible with Brrr

  3outeille

  committed
  7bfc56c

  Copy full SHA for 7bfc56c

• chore: single file mamba slow (for residual fix)

  3outeille

  committed
  1fe1cd4

  Copy full SHA for 1fe1cd4

• fix: residual for slow mamba

  3outeille

  committed
  720df8e

  Copy full SHA for 720df8e

• fix(mamba-slow): transpose embedding

  3outeille

  committed
  7902959

  Copy full SHA for 7902959

• update(mamba-slow)

  3outeille

  committed
  b2ca553

  Copy full SHA for b2ca553

• fix: unified random unit

  3outeille

  committed
  3c4ef42

  Copy full SHA for 3c4ef42

• feat(optimizer): can now apply weight decay to specifcs parameters

  3outeille

  committed
  afcbb10

  Copy full SHA for afcbb10

Commits on Feb 12, 2024

• Merge branch 'mamba' into mamba-update

  3outeille

  committed
  7216b11

  Copy full SHA for 7216b11

• refacto: mamba1

  3outeille

  committed
  a6a163f

  Copy full SHA for a6a163f

• feat: add selective scan interface

  3outeille

  committed
  ad17887

  Copy full SHA for ad17887

• feat: make fast path compatible with TP

  3outeille

  committed
  6bfd6aa

  Copy full SHA for 6bfd6aa

Commits on Feb 13, 2024

• refacto: no split weight

  3outeille

  committed
  f0e219b

  Copy full SHA for f0e219b

• refacto: cleaner init weights

  3outeille

  committed
  ae312fe

  Copy full SHA for ae312fe

Commits on Feb 14, 2024

• refacto: init weights

  3outeille

  committed
  ff26ff5

  Copy full SHA for ff26ff5

Commits on Feb 16, 2024

• feat: save/load _no_weight_decay attribute for Mamba model

  3outeille

  committed
  0ab7c9b

  Copy full SHA for 0ab7c9b

Commits on Feb 28, 2024

• clean import

  3outeille

  committed
  3caf7fe

  Copy full SHA for 3caf7fe

• clean utils in mamba

  3outeille

  committed
  ec6f954

  Copy full SHA for ec6f954

• cleaning initialization for mamba

  3outeille

  committed
  1773420

  Copy full SHA for 1773420

• no sync conv and rmsnorm for mamba

  3outeille

  committed
  6fc0d9a

  Copy full SHA for 6fc0d9a

• Merge branch 'mamba-update' into mamba-pr

  3outeille

  committed
  6e8fd1f

  Copy full SHA for 6e8fd1f

• clean run_train

  3outeille

  committed
  fa6953d

  Copy full SHA for fa6953d

• clean run_train

  3outeille

  committed
  20dc9ec

  Copy full SHA for 20dc9ec

Commits on Feb 29, 2024

• rename folder

  3outeille

  committed
  beb73ee

  Copy full SHA for beb73ee

• add assert only ALL_REDUCE mode

  3outeille

  committed
  d6ae604

  Copy full SHA for d6ae604

• add typing to mamba

  3outeille

  committed
  5d161b1

  Copy full SHA for 5d161b1

• remove init method in favor of new initialization method

  3outeille

  committed
  befb66d

  Copy full SHA for befb66d

Commits on Mar 1, 2024

• fix test serialize

  3outeille

  committed
  35370b7

  Copy full SHA for 35370b7

• small fix

  3outeille

  committed
  790eddc

  Copy full SHA for 790eddc

• add mamba example

  3outeille

  committed
  37867d1

  Copy full SHA for 37867d1

• fix logger

  3outeille

  committed
  54a9c63

  Copy full SHA for 54a9c63

• update README

  3outeille

  committed
  3953a5c

  Copy full SHA for 3953a5c

• update yaml

  3outeille

  committed
  3fa2194

  Copy full SHA for 3fa2194

• move to examples

  3outeille

  committed
  d4bba7c

  Copy full SHA for d4bba7c

• update readme

  3outeille

  committed
  a52c37f

  Copy full SHA for a52c37f

• discard run_generate for now

  3outeille

  committed
  1f54ae2

  Copy full SHA for 1f54ae2

Commits on Mar 4, 2024

• revert dynamic weight decay for now

  3outeille

  committed
  7616ce7

  Copy full SHA for 7616ce7

• unifying init_model_randomly for other models

  3outeille

  committed
  e944f63

  Copy full SHA for e944f63

• decouple mamba logic from core

  3outeille

  committed
  8224221

  Copy full SHA for 8224221

• fix logging + init_method

  3outeille

  committed
  d28c968

  Copy full SHA for d28c968

• delete yaml file

  3outeille

  committed
  576eb56

  Copy full SHA for 576eb56

• small fix

  3outeille

  committed
  ea2fa96

  Copy full SHA for ea2fa96

• fix tp assert

  3outeille

  committed
  a8448bb

  Copy full SHA for a8448bb

• separate config mamba

  3outeille

  committed
  106d59c

  Copy full SHA for 106d59c

• update requirements readme

  3outeille

  committed
  db889b9

  Copy full SHA for db889b9

• various fix

  3outeille

  committed
  f4816ee

  Copy full SHA for f4816ee

• Merge branch 'main' into main

  3outeille

  authored
  d243b70

  Copy full SHA for d243b70

• fix import

  3outeille

  committed
  1098e19

  Copy full SHA for 1098e19

• change directory level

  3outeille

  committed
  ae6af18

  Copy full SHA for ae6af18

• decouple Mamba logic from sync weight

  3outeille

  committed
  baba00e

  Copy full SHA for baba00e

• fix decorator

  3outeille

  committed
  7b513e8

  Copy full SHA for 7b513e8

• small fixes

  3outeille

  committed
  dbefcc7

  Copy full SHA for dbefcc7