Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

misc: enhance allocator error info and add shape check for prefill begin forward functions #413

Merged
merged 7 commits into from
Jul 31, 2024

Conversation

yzh119
Copy link
Collaborator

@yzh119 yzh119 commented Jul 31, 2024

This PR makes the following changes to the codebase:

  1. make the allocators error information more informative, more specifically, we print the buffer name and requested buffer size in runtime errors for debugging.
  2. add checks in prefill wrappers begin_forward functions to make sure qo and kv indptr array size matches.

These efforts are designed for avoiding issues such as #362 , which needs to be fixed on vllm side, but we should have more friendly debugging information for locating the potential bugs.

@yzh119 yzh119 merged commit 5e36c52 into main Jul 31, 2024
@yzh119 yzh119 mentioned this pull request Jul 31, 2024
yzh119 added a commit that referenced this pull request Jul 31, 2024
##
[0.1.3](v0.1.2...v0.1.3)
(2024-07-31)

### Bugfix

* bugfix: Fix cudagraph mode of BatchPrefillWithRaggedKVCacheWrapper
([#412](#412))
([9907bc](9907bc1))
* fix cu118 cub usage for sampling kernels
([#410](#410))
([58d359](58d3593))

### Misc

* enhance allocator error info and add shape check for prefill begin
forward functions
([#413](#413))
([5e36c5](5e36c52))
@yzh119 yzh119 deleted the bugfix-362 branch August 3, 2024 00:20
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant