Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

doc: add Intel MKL compilation for building library with server example #99

Open
wants to merge 5 commits into
base: main
Choose a base branch
from

Conversation

xiangyang-95
Copy link

@xiangyang-95 xiangyang-95 commented Jun 17, 2024

This pull request adds steps to build library with server example with Intel MKL backend.
This will also add support for AVX_VNNI hardware instruction sets in Intel CPU.

@xiangyang-95 xiangyang-95 changed the title DOC: add Intel MKL compilation for building library with server example doc: add Intel MKL compilation for building library with server example Jun 17, 2024
@vansangpfiev
Copy link
Contributor

Thanks for your PR. We will review it soon.
Seems like the PR has some conflicts, could you please help to fix them?

@xiangyang-95
Copy link
Author

@vansangpfiev I have resolved the conflict. Feel free to let me know if need any info or modifications.

@vansangpfiev vansangpfiev requested a review from hiento09 June 20, 2024 01:41
@xiangyang-95
Copy link
Author

Hi @hiento09, may I know are there any changes needed? I am also planned to contribute with a version running on SYCL as well

@dan-menlo
Copy link

@xiangyang-95 FYI, we will be working on Intel in a couple of sprints - thank you for the patience as we get things in order 🙏

@xiangyang-95
Copy link
Author

Hi @dan-homebrew, I am actually from Intel team. Please let me know if anything I can help.

@dan-menlo
Copy link

Hi @dan-homebrew, I am actually from Intel team. Please let me know if anything I can help.

Hey yes I saw! We also spoke with a few of your colleagues.

On our part, we need to take a bit of time to refactor our implementation of cortex.llama.cpp in Jan:

Jan

  • We will allow users to download multiple versions of llama.cpp (e.g. compiled with sycl, avx2)
  • We will have a UI which allows user to select which version of llama.cpp they want to use
  • We will pre-bundle several versions of llama.cpp, and make additional ones (e.g. sycl available for pull)

Cortex.cpp

  • For cortex.cpp, most of the work is right now infrastructural as we bootstrap the CLI and basic functions
  • We are implementing cortex engines install, which right now relies on hardware auto-detection
  • cortex engines install llama.cpp will need to expand to support the different llama.cpp variants

# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants