Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

Suppose Gemma model shape #130

Closed
yzh119 opened this issue Feb 21, 2024 · 1 comment
Closed

Suppose Gemma model shape #130

yzh119 opened this issue Feb 21, 2024 · 1 comment

Comments

@yzh119
Copy link
Collaborator

yzh119 commented Feb 21, 2024 •
edited
Loading

Gemma uses head_dim=256 which is enabled in pip wheels by default. We should compile kernels for head_dim=256 and change some kernel parameters for best performance in this case.
@yzh119 yzh119 mentioned this issue Feb 22, 2024
Merged
yzh119 added a commit that referenced this issue Feb 25, 2024
@yzh119
feat: enable head_dim=256 for attention kernels (#132)
0372acc 
As mentioned in #130 , the kernels for `head_dim=256` are not compiled
by default, this PR expose these attention kernels to pip wheels and
adds unittests/benchmarks for `head_dim=256`.
@yzh119
Copy link
Collaborator Author

yzh119 commented Feb 25, 2024

Fixed in #132 .

@yzh119 yzh119 closed this as completed Feb 25, 2024
@yzh119 yzh119 mentioned this issue Feb 27, 2024
Closed
3 tasks
# for free to join this conversation on GitHub. Already have an account? # to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@yzh119