[Bug]: KeyError: 'type'. when inferencing Llama 3.2 3B Instruct

[RonanKMcGovern]

Your current environment

The output of `python collect_env.py`

--2024-09-26 15:08:57--  https://raw.githubusercontent.com/vllm-project/vllm/main/collect_env.py
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.108.133, 185.199.109.133, 185.199.110.133, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 25389 (25K) [text/plain]
Saving to: ‘collect_env.py’

collect_env.py      100%[===================>]  24.79K  --.-KB/s    in 0s      

2024-09-26 15:08:58 (94.2 MB/s) - ‘collect_env.py’ saved [25389/25389]

Collecting environment information...
2024-09-26 15:09:06.936526: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2024-09-26 15:09:07.195978: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2024-09-26 15:09:07.266656: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2024-09-26 15:09:07.688307: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-09-26 15:09:09.928315: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
PyTorch version: 2.4.0+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.3 LTS (x86_64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: 14.0.0-1ubuntu1.1
CMake version: version 3.30.3
Libc version: glibc-2.35

Python version: 3.10.12 (main, Sep 11 2024, 15:47:36) [GCC 11.4.0] (64-bit runtime)
Python platform: Linux-6.1.85+-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: 12.2.140
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: Tesla T4
Nvidia driver version: 535.104.05
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.6
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.6
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.6
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.6
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.6
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.6
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.6
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                         x86_64
CPU op-mode(s):                       32-bit, 64-bit
Address sizes:                        46 bits physical, 48 bits virtual
Byte Order:                           Little Endian
CPU(s):                               2
On-line CPU(s) list:                  0,1
Vendor ID:                            GenuineIntel
Model name:                           Intel(R) Xeon(R) CPU @ 2.20GHz
CPU family:                           6
Model:                                79
Thread(s) per core:                   2
Core(s) per socket:                   1
Socket(s):                            1
Stepping:                             0
BogoMIPS:                             4399.99
Flags:                                fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single ssbd ibrs ibpb stibp fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm rdseed adx smap xsaveopt arat md_clear arch_capabilities
Hypervisor vendor:                    KVM
Virtualization type:                  full
L1d cache:                            32 KiB (1 instance)
L1i cache:                            32 KiB (1 instance)
L2 cache:                             256 KiB (1 instance)
L3 cache:                             55 MiB (1 instance)
NUMA node(s):                         1
NUMA node0 CPU(s):                    0,1
Vulnerability Gather data sampling:   Not affected
Vulnerability Itlb multihit:          Not affected
Vulnerability L1tf:                   Mitigation; PTE Inversion
Vulnerability Mds:                    Vulnerable; SMT Host state unknown
Vulnerability Meltdown:               Vulnerable
Vulnerability Mmio stale data:        Vulnerable
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed:               Vulnerable
Vulnerability Spec rstack overflow:   Not affected
Vulnerability Spec store bypass:      Vulnerable
Vulnerability Spectre v1:             Vulnerable: __user pointer sanitization and usercopy barriers only; no swapgs barriers
Vulnerability Spectre v2:             Vulnerable; IBPB: disabled; STIBP: disabled; PBRSB-eIBRS: Not affected; BHI: Vulnerable (Syscall hardening enabled)
Vulnerability Srbds:                  Not affected
Vulnerability Tsx async abort:        Vulnerable

Versions of relevant libraries:
[pip3] numpy==1.26.4
[pip3] nvidia-cublas-cu12==12.1.3.1
[pip3] nvidia-cuda-cupti-cu12==12.1.105
[pip3] nvidia-cuda-nvcc-cu12==12.6.68
[pip3] nvidia-cuda-nvrtc-cu12==12.1.105
[pip3] nvidia-cuda-runtime-cu12==12.1.105
[pip3] nvidia-cudnn-cu12==9.1.0.70
[pip3] nvidia-cufft-cu12==11.0.2.54
[pip3] nvidia-curand-cu12==10.3.2.106
[pip3] nvidia-cusolver-cu12==11.4.5.107
[pip3] nvidia-cusparse-cu12==12.1.0.106
[pip3] nvidia-ml-py==12.560.30
[pip3] nvidia-nccl-cu12==2.20.5
[pip3] nvidia-nvjitlink-cu12==12.6.68
[pip3] nvidia-nvtx-cu12==12.1.105
[pip3] optree==0.12.1
[pip3] pyzmq==24.0.1
[pip3] torch==2.4.0
[pip3] torchaudio==2.4.1+cu121
[pip3] torchsummary==1.5.1
[pip3] torchvision==0.19.0
[pip3] transformers==4.45.0
[pip3] triton==3.0.0
[conda] Could not collect
ROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: N/A
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
GPU0	CPU Affinity	NUMA Affinity	GPU NUMA ID
GPU0	 X 	0-1		N/A		N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

Model Input Dumps

No response

🐛 Describe the bug

Error:

KeyError                                  Traceback (most recent call last)
[<ipython-input-41-4c7e309514ca>](https://localhost:8080/#) in <cell line: 16>()
     14 
     15 # Create an LLM.
---> 16 llm = LLM(
     17     model="meta-llama/Llama-3.2-3B-Instruct",
     18     tokenizer="meta-llama/Llama-3.2-3B-Instruct",

4 frames
[/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/llm.py](https://localhost:8080/#) in __init__(self, model, tokenizer, tokenizer_mode, trust_remote_code, tensor_parallel_size, dtype, quantization, revision, tokenizer_revision, seed, gpu_memory_utilization, swap_space, **kwargs)
     91             use `float16` instead.
     92         quantization: The method used to quantize the model weights. Currently,
---> 93             we support "awq", "gptq", and "fp8" (experimental).
     94             If None, we first check the `quantization_config` attribute in the
     95             model config file. If that is None, we assume the model weights are

[/usr/local/lib/python3.10/dist-packages/vllm/engine/llm_engine.py](https://localhost:8080/#) in from_engine_args(cls, engine_args)
    238             "decoding_config=%r, observability_config=%r, "
    239             "seed=%d, served_model_name=%s, use_v2_block_manager=%s, "
--> 240             "num_scheduler_steps=%d, multi_step_stream_outputs=%s, "
    241             "enable_prefix_caching=%s, use_async_output_proc=%s, "
    242             "use_cached_outputs=%s, mm_processor_kwargs=%s)",

[/usr/local/lib/python3.10/dist-packages/vllm/engine/arg_utils.py](https://localhost:8080/#) in create_engine_configs(self)
    196             '--tokenizer',
    197             type=nullable_str,
--> 198             default=EngineArgs.tokenizer,
    199             help='Name or path of the huggingface tokenizer to use. '
    200             'If unspecified, model name or path will be used.')

[/usr/local/lib/python3.10/dist-packages/vllm/config.py](https://localhost:8080/#) in __init__(self, model, tokenizer, tokenizer_mode, trust_remote_code, download_dir, load_format, dtype, seed, revision, tokenizer_revision, max_model_len, quantization)
     91         quantization_param_path: Path to JSON file containing scaling factors.
     92             Used to load KV cache scaling factors into the model when KV cache
---> 93             type is FP8_E4M3 on ROCm (AMD GPU). In the future these will also
     94             be used to load activation and weight scaling factors when the
     95             model dtype is FP8_E4M3 on ROCm.

[/usr/local/lib/python3.10/dist-packages/vllm/config.py](https://localhost:8080/#) in _get_and_verify_max_len(hf_config, max_model_len)
    491             return self.hf_config.num_attention_heads
    492         if self.hf_config.model_type == "dbrx":
--> 493             return getattr(self.hf_config.attn_config, "kv_n_heads",
    494                            self.hf_config.num_attention_heads)
    495 

KeyError: 'type'

Reproduction (T4 on Colab):

!pip install vllm transformers -qU

import os
os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "1"
os.environ["HF_TOKEN"]=input()

from vllm import LLM, SamplingParams
import torch

# Sample prompts.
prompts = [
    "Hello, my name is",
    "The president of the United States is",
    "The capital of France is",
    "The future of AI is",
]

# Create a sampling params object.
sampling_params = SamplingParams(temperature=0)

# Create an LLM.
llm = LLM(
    model="meta-llama/Llama-3.2-3B-Instruct",
    # trust_remote_code=True,
    # dtype="float16",
    # rope_scaling={"type": "extended", "factor": 8.0},
    )

# Generate texts from the prompts. The output is a list of RequestOutput objects
# that contain the prompt, generated text, and other information.
outputs = llm.generate(prompts, sampling_params)

# Print the outputs.
for output in outputs:
    prompt = output.prompt
    answer = output.outputs[0].text
    print(f"Prompt: {prompt!r}, Generated text: {answer!r}")

There seem to be quite a few different errors with different models, like google/gemma-2-2b-it which gives:

ImportError                               Traceback (most recent call last)
[<ipython-input-43-78ec61c78513>](https://localhost:8080/#) in <cell line: 16>()
     14 
     15 # Create an LLM.
---> 16 llm = LLM(
     17     # model="meta-llama/Llama-3.2-3B-Instruct",
     18     model="google/gemma-2-2b-it",

4 frames
[/usr/local/lib/python3.10/dist-packages/vllm/worker/worker.py](https://localhost:8080/#) in <module>
      8 
      9 import vllm.envs as envs
---> 10 from vllm.config import (CacheConfig, DeviceConfig, LoadConfig, LoRAConfig,
     11                          ModelConfig, ObservabilityConfig, ParallelConfig,
     12                          PromptAdapterConfig, SchedulerConfig,

ImportError: cannot import name 'DeviceConfig' from 'vllm.config' (/usr/local/lib/python3.10/dist-packages/vllm/config.py)

---------------------------------------------------------------------------
NOTE: If your import is failing due to a missing package, you can
manually install dependencies using either !pip or !apt.

To view examples of installing some common dependencies, click the
"Open Examples" button below.

or microsoft/Phi-3-mini-128k-instruct gives:

AssertionError                            Traceback (most recent call last)
[<ipython-input-44-d902b9f3a8ff>](https://localhost:8080/#) in <cell line: 16>()
     14 
     15 # Create an LLM.
---> 16 llm = LLM(
     17     # model="meta-llama/Llama-3.2-3B-Instruct",
     18     model="microsoft/Phi-3-mini-128k-instruct",

4 frames
[/usr/local/lib/python3.10/dist-packages/vllm/entrypoints/llm.py](https://localhost:8080/#) in __init__(self, model, tokenizer, tokenizer_mode, trust_remote_code, tensor_parallel_size, dtype, quantization, revision, tokenizer_revision, seed, gpu_memory_utilization, swap_space, **kwargs)
     91             use `float16` instead.
     92         quantization: The method used to quantize the model weights. Currently,
---> 93             we support "awq", "gptq", and "fp8" (experimental).
     94             If None, we first check the `quantization_config` attribute in the
     95             model config file. If that is None, we assume the model weights are

[/usr/local/lib/python3.10/dist-packages/vllm/engine/llm_engine.py](https://localhost:8080/#) in from_engine_args(cls, engine_args)
    238             "decoding_config=%r, observability_config=%r, "
    239             "seed=%d, served_model_name=%s, use_v2_block_manager=%s, "
--> 240             "num_scheduler_steps=%d, multi_step_stream_outputs=%s, "
    241             "enable_prefix_caching=%s, use_async_output_proc=%s, "
    242             "use_cached_outputs=%s, mm_processor_kwargs=%s)",

[/usr/local/lib/python3.10/dist-packages/vllm/engine/arg_utils.py](https://localhost:8080/#) in create_engine_configs(self)
    196             '--tokenizer',
    197             type=nullable_str,
--> 198             default=EngineArgs.tokenizer,
    199             help='Name or path of the huggingface tokenizer to use. '
    200             'If unspecified, model name or path will be used.')

[/usr/local/lib/python3.10/dist-packages/vllm/config.py](https://localhost:8080/#) in __init__(self, model, tokenizer, tokenizer_mode, trust_remote_code, download_dir, load_format, dtype, seed, revision, tokenizer_revision, max_model_len, quantization)
     91         quantization_param_path: Path to JSON file containing scaling factors.
     92             Used to load KV cache scaling factors into the model when KV cache
---> 93             type is FP8_E4M3 on ROCm (AMD GPU). In the future these will also
     94             be used to load activation and weight scaling factors when the
     95             model dtype is FP8_E4M3 on ROCm.

[/usr/local/lib/python3.10/dist-packages/vllm/config.py](https://localhost:8080/#) in _get_and_verify_max_len(hf_config, max_model_len)
    489             if "kv_n_heads" in self.hf_config.attn_config:
    490                 return self.hf_config.attn_config["kv_n_heads"]
--> 491             return self.hf_config.num_attention_heads
    492         if self.hf_config.model_type == "dbrx":
    493             return getattr(self.hf_config.attn_config, "kv_n_heads",

AssertionError:

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[DarkLight1337]

ROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: N/A
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
GPU0	CPU Affinity	NUMA Affinity	GPU NUMA ID
GPU0	 X 	0-1		N/A		N/A

It looks like vLLM wasn't installed properly. How did you install it?

[DarkLight1337]

ROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: N/A
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
GPU0	CPU Affinity	NUMA Affinity	GPU NUMA ID
GPU0	 X 	0-1		N/A		N/A

It looks like vLLM wasn't installed properly. How did you install it?

Nvm, it's a bug on our end (fixed by #8894).

Is your collect_env.py output from your local machine or Colab? Tbh I'm not sure how well vLLM is supported on Colab...

[DarkLight1337]

@WoosukKwon I think Colab uses TPU? In that case you may have a better idea

[RonanKMcGovern]

Weird, I was sure I had responded here and closed the issue.

Anyway:

• Colab uses a T4 (or A100, but I'm using T4).

Also, I was able to get running with this:

!pip install datasets hf_transfer vllm transformers -qU

then:

from vllm import LLM, SamplingParams
import torch

# Sample prompts.
prompts = [
    "Hello, my name is",
    "The president of the United States is",
    "The capital of France is",
    "The future of AI is",
]

# Create a sampling params object.
sampling_params = SamplingParams(temperature=0)

# Create an LLM.
llm = LLM(
    model="unsloth/Llama-3.2-3B-Instruct",
    # model="microsoft/Phi-3-mini-128k-instruct",
    # model="google/gemma-2-2b-it",
    # trust_remote_code=True,
    dtype="half",
    max_model_len=4096,
    )

# Generate texts from the prompts. The output is a list of RequestOutput objects
# that contain the prompt, generated text, and other information.
outputs = llm.generate(prompts, sampling_params)

# Print the outputs.
for output in outputs:
    prompt = output.prompt
    answer = output.outputs[0].text
    print(f"Prompt: {prompt!r}, Generated text: {answer!r}")