Possible privateGPT integration ?

[darkstorm2150]

Wonder if it is possible to implement https://github.com/imartinez/privateGPT into llama.cpp, The reason I see this is because the inference speed with cublas is insanely fast with the current models 13b, having features of digesting any information would skyrocket llama.cpp for real-time applications.

[SlyEcho]

Follow the instructions for llama-cpp-python installation, which it uses: https://github.com/abetlen/llama-cpp-python#installation-with-openblas--cublas--clblast