This repo contains evaluation results generated with LLM-KG-Bench version 3 for 38* open and proprietary LLMs. This includes the configuration, the model format preference table and the plots in the plots directory.
The raw result files are passwort protected to reduce the chance of test data leakage into LLM training data. Please do not spread them unencrypted. You can unzip them with Leave-Finest-Coats-Serena-Cheats-46
*Note: The LLM "Solar pro preview 22B" skipped several test cases due to context window limitations.
The following table shows an overview of Capability Compass plots for open LLMs. Each line contains a LLM model family, the columns sort the LLMs according to their parameter count. For more details see in plots/README.md
| Family | 0.5B | 1B | 1.5B | 3B/3.8B | 7B/8B | MoE Active 6-14B | 14B | 32B/33B | 70B/72B |
|---|---|---|---|---|---|---|---|---|---|
| Llama 3.0 |  |
 |
|||||||
| --> | Meta-Llama-3-8B-Instruct | Meta-Llama-3-70B-Instruct | |||||||
| Llama 3.1 |  |
 |
|||||||
| --> | Llama-3.1-8B-Instruct | Llama-3.1-70B-Instruct | |||||||
| Llama 3.2 |  |
 |
|||||||
| --> | Llama-3.2-1B-Instruct | Llama-3.2-3B-Instruct | |||||||
| Llama 3.3 |  |
||||||||
| --> | Llama-3.3-70B-Instruct | ||||||||
| Phi 3.0 |  |
 |
 |
||||||
| --> | Phi-3-mini-128k-instruct | Phi-3-small-128k-instruct | Phi-3-medium-128k-instruct | ||||||
| Phi 3.5 |  |
 |
|||||||
| --> | Phi-3.5-mini-instruct | Phi-3.5-MoE-instruct | |||||||
| Qwen2 |  |
 |
 |
 |
 |
||||
| --> | Qwen2-0.5B-Instruct | Qwen2-1.5B-Instruct | Qwen2-7B-Instruct | Qwen2-57B-A14B-Instruct | Qwen2-72B-Instruct | ||||
| Qwen2.5 |  |
 |
 |
 |
 |
 |
 |
||
| --> | Qwen2.5-0.5B-Instruct | Qwen2.5-1.5B-Instruct | Qwen2.5-3B-Instruct | Qwen2.5-7B-Instruct | Qwen2.5-14B-Instruct | Qwen2.5-32B-Instruct | Qwen2.5-72B-Instruct | ||
| Qwen2.5-Coder |  |
||||||||
| --> | Qwen2.5-Coder-32B-Instruct | ||||||||
| Infly-OpenCoder |  |
||||||||
| --> | OpenCoder-8B-Instruct | ||||||||
| Deepseek-coder |  |
||||||||
| --> | deepseek-coder-33b-instruct |
| RdfConnectionExplainStatic | RdfFriendCount-1 | RdfFriendCount-2 | RdfSyntaxFixList | Sparql2AnswerListOrga | Text2AnswerListOrga | |
|---|---|---|---|---|---|---|
| Claude 3.5 Haiku | JSON | TTL | TTL | - | - | - |
| Claude 3.5 Sonnet | - | - | - | - | - | - |
| Deepseek-Coder-33B | - | JSON | JSON | JSON | - | - |
| GPT3.5 2024/01 | - | JSON | JSON | - | - | TTL |
| GPT4o 2024/11 | - | - | - | JSON | - | - |
| GPT4o-mini 2024/07 | - | - | TTL | - | - | TTL |
| GPTo1-mini 2024/09 | - | - | - | - | - | - |
| GPTo1-pre 2024/09 | - | - | - | - | - | - |
| Gemini 1.5 Flash | - | TTL | - | - | - | - |
| Gemini 1.5 Pro | - | - | - | - | - | - |
| Gemini 2.0 Flash Exp | - | - | - | - | - | - |
| Llama-3.3-70B | - | JSON | JSON | JSON | - | - |
| Meta-Llama-3-70B | - | JSON | JSON | JSON | - | - |
| Meta-Llama-3-8B | TTL | - | - | - | - | - |
| Meta-Llama-3.1-70B | - | - | - | JSON | TTL | - |
| Meta-Llama-3.1-8B | - | JSON | JSON | JSON | - | - |
| Meta-Llama-3.2-1B | - | - | - | JSON | - | - |
| Meta-Llama-3.2-3B | JSON | TTL | TTL | JSON | - | - |
| OpenCoder-8B | - | - | - | - | TTL | - |
| Phi-3-medium-128k | TTL | - | JSON | - | - | TTL |
| Phi-3-mini-128k | - | - | JSON | JSON | - | - |
| Phi-3-small-128k | JSON | - | - | TTL | - | - |
| Phi-3.5-mini | TTL | JSON | - | JSON | TTL | - |
| Qwen2-0.5B | - | - | JSON | - | - | - |
| Qwen2-1.5B | TTL | JSON | JSON | JSON | - | - |
| Qwen2-57B-A14B | - | - | - | JSON | TTL | TTL |
| Qwen2-72B | - | - | - | JSON | - | - |
| Qwen2-7B | JSON | - | - | - | - | TTL |
| Qwen2.5-0.5B | - | - | - | - | - | - |
| Qwen2.5-1.5B | - | - | - | JSON | - | - |
| Qwen2.5-14B | TTL | - | - | - | - | TTL |
| Qwen2.5-32B | - | TTL | TTL | JSON | JSON | - |
| Qwen2.5-3B | JSON | - | - | JSON | - | - |
| Qwen2.5-72B | JSON | JSON | JSON | - | - | TTL |
| Qwen2.5-7B | JSON | - | - | - | TTL | TTL |
| Qwen2.5-Coder-32B | - | TTL | TTL | - | - | - |
| Solar-pro-preview-22B* | - | - | - | JSON | - | - |
| Phi_3.5_MoE_Instruct | - | JSON | JSON | JSON | TTL | - |