convert the 7B model to ggml FP16 format fails on RPi 4B

[davidrutland]

Everything's OK until this step

python3 convert-pth-to-ggml.py models/7B/ 1
{'dim': 4096, 'multiple_of': 256, 'n_heads': 32, 'n_layers': 32, 'norm_eps': 1e-06, 'vocab_size': 32000}
n_parts = 1
Processing part 0
Killed

models/7B/ggml-model-f16.bin isn't created

[davidrutland]

Ah. Got it. I think I need a bigger SD card.

[JavSensei]

I have the same problem, the ram increases until it is completely filled and that's as far as it goes

[Ronsor]

I'm pretty sure you need memory for both the original fp16 model + the converted model. A RPi 4 (even with 8 GB of RAM) isn't going to have enough.

[BrianSemiglia]

Would it be okay to share a torrent of converted models? Would be a smaller download

[Ronsor]

There are indeed torrents in places like 4chan. I personally don't think model weights can even be copyrighted, but the repo maintainer probably doesn't want to risk it, otherwise there would likely already be a torrent link provided.

[xiliuya]

convert the 7B model need 16G RAM.

[apaz-cli]

If you're having memory trouble you should make a swapfile. I only have 8GB of memory, and I'm going back and forth between tasks, so I just wrote a script that makes it easy to create and destroy swapfiles.

https://github.com/apaz-cli/Scripts/blob/master/swapcreate
https://github.com/apaz-cli/Scripts/blob/master/swapdestroy

Or you can find instructions online.

[G2G2G2G]

Ah. Got it. I think I need a bigger SD card.

you can convert it on any computer then just copy it over.

[gjmulder]

This looks to be a RAM issue. Closing.