Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Improve the implementation of Manual Cast and IPEX support #14597

Merged
merged 7 commits into from
Jan 9, 2024

Conversation

KohakuBlueleaf
Copy link
Collaborator

Description

  • Improved the efficiency of manual cast with type checking.
  • Apply the behavior of precision=full into manual cast
    • Also ensure fp8 can be used with precision=full on autocastable devices.
      (With manual cast, which is as fast as autocast + output upcast. I just chose the simples one)
  • Use float32 for MHA layer in ipex since some computation in it is not good for IPEX devices.
  • Have checked this implementation can be used on NV Cuda/Apple M/Intel ARC. With latest ipex, Arc A770 can use manual cast with fp8 features.
  • Manual Cast on A770 is now faster then autocast('xpu'), even with ipex.optimize.

Checklist:

@AUTOMATIC1111 AUTOMATIC1111 merged commit 905b142 into dev Jan 9, 2024
6 checks passed
@AUTOMATIC1111 AUTOMATIC1111 deleted the improved-manual-cast branch January 9, 2024 16:33
@w-e-w w-e-w mentioned this pull request Feb 17, 2024
@pawel665j pawel665j mentioned this pull request Apr 16, 2024
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants