Skip to content

CUDA acceleration doesn't seem to work #1445

New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom
Closed
megupta opened this issue May 14, 2023 · 1 comment
Closed

CUDA acceleration doesn't seem to work #1445

megupta opened this issue May 14, 2023 · 1 comment

Comments

@megupta
Copy link

megupta commented May 14, 2023 •
edited
Loading

I compiled the latest code in this repo with cuBLAS support as described in the README

It doesn't seem to be utilizing my 1070 although main is running in nvidia-smi

llama_model_load_internal: [cublas] offloading 0 layers to GPU
llama_model_load_internal: [cublas] total VRAM used: 0 MB

What am I missing here?
@FSSRepo
Copy link
Collaborator

FSSRepo commented May 14, 2023 •
edited
Loading

Add the option -ngl 10, for upload 10 layers to your gpu memory

@Bearsaerker Bearsaerker mentioned this issue Mar 12, 2025
Open
# for free to join this conversation on GitHub. Already have an account? # to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@Green-Sky @megupta @FSSRepo