Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

[QUESTION] Why model is taking so long to load? #1665

Open
MariGaS opened this issue Aug 30, 2024 · 0 comments
Open

[QUESTION] Why model is taking so long to load? #1665

MariGaS opened this issue Aug 30, 2024 · 0 comments
Labels
bug Something isn't working

Comments

@MariGaS
Copy link

MariGaS commented Aug 30, 2024

I download the files of SOLAR 10.7B (https://huggingface.co/neuralmagic/SOLAR-10.7B-Instruct-v1.0-pruned50-quant-ds) in my MacBook Pro (Ventura 13.4) takes 4-5 minutes to load the model and works fine.

Expected behavior
I expected to work like that in another computer with linux, but takes a lot of time to load

Environment
Include all relevant environment information:

  1. OS Ubuntu 16.04.7 LTS
  2. Python version 3.9.7
  3. DeepSparse version or commit hash deepsparse-nightly==1.8.0.20240502
  4. ML framework version(s) torch==2.4.0, torchaudio==2.4.0, torchvision==0.19.0
  5. Other Python package versions onnx==1.14.1,numpy==1.26.3, transformers==4.43.4
  6. CPU info - output of deepsparse/src/deepsparse/arch.bin or output of cpu_architecture() as follows:
>>> import deepsparse.cpu
>>> print(deepsparse.cpu.cpu_architecture())

'L1_data_cache_size': 32768, 'L1_instruction_cache_size': 32768, 'L2_cache_size': 4194304, 'L3_cache_size': 16777216, 'architecture': 'x86_64', 'available_cores_per_socket': 12, 'available_num_cores': 12, 'available_num_hw_threads': 12, 'available_num_numa': 1, 'available_num_sockets': 1, 'available_sockets': 1, 'available_threads_per_core': 1, 'bf16': False, 'cores_per_socket': 12, 'dotprod': False, 'i8mm': False, 'isa': 'avx512', 'num_cores': 12, 'num_hw_threads': 12, 'num_numa': 1, 'num_sockets': 1, 'threads_per_core': 1, 'vbmi': False, 'vbmi2': False, 'vendor': 'GenuineIntel', 'vendor_id': 'Intel', 'vendor_model': 'Intel Xeon Processor (Skylake, IBRS, no TSX)', 'vnni': False, 'zen1': False
To Reproduce
from deepsparse import TextGeneration
model = TextGeneration(model_path="./SOLAR")

Errors
Captura de pantalla 2024-08-30 a la(s) 10 55 19

@MariGaS MariGaS added the bug Something isn't working label Aug 30, 2024
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant