New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

refactor: reduce the binary size of batch decode kernels #343

Merged

yzh119 merged 2 commits into main from reduce-decode-datatype

Jun 30, 2024

Collaborator

yzh119 commented Jun 30, 2024

This PR refactors the batch decode related kernels, and make the following breaking changes:

remove batch_decode_with_padded_kv_cache operator, we encourage user to use BatchDecodeWithPagedKVCacheWrapper.
Delete redundant DTypeQ * DTypeKV combinations, now we only support the following cases:
DTypeQ == DTypeKV
DTypeQ is a float16 and DTypeKV is a float8

The output data type follows the query data type.

yzh119 added 2 commits

June 30, 2024 06:53

upd

1915b8f


          add group size 2 back for decode kernels

2ebea3d

yzh119 merged commit 0d333ff into main

yzh119 deleted the reduce-decode-datatype branch

June 30, 2024 07:14

# for free to join this conversation on GitHub. Already have an account? # to comment

Labels

None yet