Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

[Misc] Add uninitialized params tracking for AutoWeightsLoader #10327

Merged
merged 28 commits into from
Nov 18, 2024
Merged

[Misc] Add uninitialized params tracking for AutoWeightsLoader #10327

merged 28 commits into from
Nov 18, 2024

Conversation

Isotr0py
Copy link
Collaborator

@Isotr0py Isotr0py commented Nov 14, 2024 •
edited by github-actions bot
Loading

Currently, model have uninitialized parameters won't raise any error even a warning message. This makes some bugs like #10297 (comment) and #9422 (comment) caused by incorrect weights loading/model initialization difficult to be found.

  • Add feature to track uninitialized weights for AutoWeightsLoader.

@Isotr0py
add weights loading tracker for llama
b561511 
Signed-off-by: Isotr0py <2037008807@qq.com>
@github-actions GitHub Actions
Copy link

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

  • Add ready label to the PR
  • Enable auto-merge.

🚀

@Isotr0py
modify qwen2
5842622 
Signed-off-by: Isotr0py <2037008807@qq.com>
@Isotr0py
code format
3b044c3 
Signed-off-by: Isotr0py <2037008807@qq.com>
@Isotr0py Isotr0py marked this pull request as draft November 14, 2024 13:48
@DarkLight1337 DarkLight1337 self-assigned this Nov 14, 2024
@Isotr0py
modify blip
494f1d4 
Signed-off-by: Isotr0py <2037008807@qq.com>
@Isotr0py
modify more models
50458a0 
Signed-off-by: Isotr0py <2037008807@qq.com>
@mergify Mergify
Copy link

mergify bot commented Nov 15, 2024

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @Isotr0py.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify bot added the needs-rebase label Nov 15, 2024
@Isotr0py
modify models
3b9e4d1 
Signed-off-by: Isotr0py <2037008807@qq.com>
@Isotr0py
change up to mamba
e61b030 
Signed-off-by: Isotr0py <2037008807@qq.com>
@Isotr0py
modify remain models
a28d846 
Signed-off-by: Isotr0py <2037008807@qq.com>
@Isotr0py
code format
aabd999 
Signed-off-by: Isotr0py <2037008807@qq.com>
@Isotr0py
revert bert
201ffc1 
Signed-off-by: Isotr0py <2037008807@qq.com>
@Isotr0py
Merge branch 'main' into track-weight-load
ee407d4
@mergify mergify bot removed the needs-rebase label Nov 16, 2024
@Isotr0py
fix orion lints
2957bea 
Signed-off-by: Isotr0py <2037008807@qq.com>
@Isotr0py
fix load_weights lints
3c935d5 
Signed-off-by: Isotr0py <2037008807@qq.com>
@Isotr0py
fix incorrect implement
40baa36 
Signed-off-by: Isotr0py <2037008807@qq.com>
@Isotr0py Isotr0py marked this pull request as ready for review November 16, 2024 14:45
@Isotr0py
align autoweightsloader
025adfb 
Signed-off-by: Isotr0py <2037008807@qq.com>
@Isotr0py
fix phi-3v
7357c3a 
Signed-off-by: Isotr0py <2037008807@qq.com>
@Isotr0py
add striction for AutoWeightsLoader
f32dd0e 
Signed-off-by: Isotr0py <2037008807@qq.com>
DarkLight1337
DarkLight1337 approved these changes Nov 17, 2024
View reviewed changes
Copy link
Member

@DarkLight1337 DarkLight1337 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the QoL improvement!

@DarkLight1337 DarkLight1337 enabled auto-merge (squash) November 17, 2024 09:13
@github-actions github-actions bot added the ready ONLY add when PR is ready to merge/full CI is needed label Nov 17, 2024
@DarkLight1337
Copy link
Member

Hmm. not sure whether we're being too strict or is the weight loader buggy?

@Isotr0py
track compressed_tensors cache scale
016ea94 
Signed-off-by: Isotr0py <2037008807@qq.com>
@Isotr0py
Copy link
Collaborator Author

Isotr0py commented Nov 17, 2024 •
edited
Loading

Hmmm, seems that strict=True is too strict for sharded safetensors, because in that case the weights of a module will be sharded into multiple files. It will trigger strict check then raise error while the module is partially loaded from one of the shared files...

Perhaps we should add the loading check on top of DefaultModelLoader in loader.py instead of AutoWeightsLoader.

@DarkLight1337
Copy link
Member

Agreed, let's move the weight loading check to a higher level.

@Isotr0py
move strict check to high-level model loader
0ae6fe6 
Signed-off-by: Isotr0py <2037008807@qq.com>
@DarkLight1337
Copy link
Member

DarkLight1337 commented Nov 17, 2024 •
edited
Loading

Since we may skip loading some layers in vision models / PP case, we should update the check to error out when weights inside the initialized model aren't loaded, rather than when some weights from the checkpoint aren't loaded.

@Isotr0py
fix vit weight tracking
24da01e 
Signed-off-by: Isotr0py <2037008807@qq.com>
@Isotr0py
fix llama pp
7287182 
Signed-off-by: Isotr0py <2037008807@qq.com>
@Isotr0py
revert bert
2bb9cd8 
Signed-off-by: Isotr0py <2037008807@qq.com>
@Isotr0py
fix qwen2 cls
3581ee3 
Signed-off-by: Isotr0py <2037008807@qq.com>
@Isotr0py
revert embedding models
7c2507e 
Signed-off-by: Isotr0py <2037008807@qq.com>
@Isotr0py
remove strict in phi3v
e74ff67 
Signed-off-by: Isotr0py <2037008807@qq.com>
@DarkLight1337 DarkLight1337 merged commit c4e4643 into vllm-project:main Nov 18, 2024
50 checks passed
lk-chen pushed a commit to lk-chen/vllm that referenced this pull request Nov 18, 2024
@Isotr0py
[Misc] Add uninitialized params tracking for AutoWeightsLoader (vll…
ce3ae6f 
…m-project#10327)

Signed-off-by: Isotr0py <2037008807@qq.com>
Signed-off-by: Linkun Chen <lkchen+anyscale@github.com>
@Isotr0py Isotr0py deleted the track-weight-load branch November 18, 2024 06:22
@DarkLight1337
Copy link
Member

I found that this PR causes speculative tests to fail on main. PTAL @Isotr0py

@Isotr0py Isotr0py mentioned this pull request Nov 19, 2024
Merged
coolkp pushed a commit to coolkp/vllm that referenced this pull request Nov 20, 2024
@Isotr0py @coolkp
[Misc] Add uninitialized params tracking for AutoWeightsLoader (vll…
a42dd3e 
…m-project#10327)

Signed-off-by: Isotr0py <2037008807@qq.com>
KuntaiDu pushed a commit to KuntaiDu/vllm that referenced this pull request Nov 20, 2024
@Isotr0py
[Misc] Add uninitialized params tracking for AutoWeightsLoader (vll…
f40ee7c 
…m-project#10327)

Signed-off-by: Isotr0py <2037008807@qq.com>
mfournioux pushed a commit to mfournioux/vllm that referenced this pull request Nov 20, 2024
@Isotr0py @mfournioux
[Misc] Add uninitialized params tracking for AutoWeightsLoader (vll…
fcb2ebd 
…m-project#10327)

Signed-off-by: Isotr0py <2037008807@qq.com>
Signed-off-by: Maxime Fournioux <55544262+mfournioux@users.noreply.github.com>
rickyyx pushed a commit to rickyyx/vllm that referenced this pull request Nov 20, 2024
@Isotr0py @rickyyx
[Misc] Add uninitialized params tracking for AutoWeightsLoader (vll…
4e9c8f9 
…m-project#10327)

Signed-off-by: Isotr0py <2037008807@qq.com>
Signed-off-by: rickyx <rickyx@anyscale.com>
tlrmchlsmth pushed a commit to neuralmagic/vllm that referenced this pull request Nov 23, 2024
@Isotr0py @tlrmchlsmth
[Misc] Add uninitialized params tracking for AutoWeightsLoader (vll…
9c69d4f 
…m-project#10327)

Signed-off-by: Isotr0py <2037008807@qq.com>
Signed-off-by: Tyler Michael Smith <tyler@neuralmagic.com>
prashantgupta24 pushed a commit to opendatahub-io/vllm that referenced this pull request Dec 3, 2024
@Isotr0py @prashantgupta24
[Misc] Add uninitialized params tracking for AutoWeightsLoader (vll…
d3fe99b 
…m-project#10327)

Signed-off-by: Isotr0py <2037008807@qq.com>
sleepwalker2017 pushed a commit to sleepwalker2017/vllm that referenced this pull request Dec 13, 2024
@Isotr0py
[Misc] Add uninitialized params tracking for AutoWeightsLoader (vll…
8f5969f 
…m-project#10327)

Signed-off-by: Isotr0py <2037008807@qq.com>
# for free to join this conversation on GitHub. Already have an account? # to comment
Reviewers

@DarkLight1337 DarkLight1337 DarkLight1337 approved these changes

Assignees

@DarkLight1337 DarkLight1337

Labels
ready ONLY add when PR is ready to merge/full CI is needed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@Isotr0py @DarkLight1337