Llama-3 support?

[gururise]

Very interested to use this library with Llama 3. Any idea when Llama 3 will be supported?

[yannikkellerde]

Llama 3 is very much supported. The examples are not up to date I guess, but usage is similar to Llama 2 (And I recently used transformer-heads + Llama 3 in another project and it worked great).

Here some code that may help you get started

from transformer_heads import HeadConfig, create_headed_qlora
from transformer_heads.util.helpers import get_model_params

base_model_name = "meta-llama/Meta-Llama-3-8B"
model_params = get_model_params(base_model_name)
model_class = model_params["model_class"]
hidden_size = model_params["hidden_size"]
vocab_size = model_params["vocab_size"]

head_config = HeadConfig(
    name=f"my_head_name",
    layer_hook=-1,
    in_size=hidden_size,
...

From there, check the notebooks to finish designing your HeadConfig and to create you model.