From 05f5333f56717d9ee6c8f1086da6dc6384a2ba58 Mon Sep 17 00:00:00 2001
From: zwcolin <ziruiw2000@gmail.com>
Date: Mon, 19 Aug 2024 15:47:36 -0400
Subject: [PATCH] update readme.md to include latest leaderboard results and
 some add-ons for evaluation tips

---
 README.md | 72 ++++++++++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 71 insertions(+), 1 deletion(-)

diff --git a/README.md b/README.md
index b0a2b37..b4a922f 100644
--- a/README.md
+++ b/README.md
@@ -107,6 +107,17 @@ python src/generate.py \
 `gen-<model_name>-<mode>_<split>.json`. This file stores your model's responses.
 
 Note: if you are evaluating a model that is **hosted on cloud and can only be accessed via an API**, the `--model_path` argument will correspond to the name of the model e.g., `gpt-4o-2024-05-13`. Also, in creating the custom file in the `src/generate_lib` directory, you need to implement an additional function i.e., `get_client_model` that takes in the `model_path` argument and the `api_key` argument. In addition, you need to add another `elif` statement in `get_client_fn` inside `src/generate_lib/utils.py` with instructions similar to the above. Specific instructions to implement `get_client_model` function differ by API providers, and examples are provided in `gpt.py`, `gemini.py`, `reka.py`, `claude.py`, and `qwen.py`.
+Note: you may see a header like:
+```py
+### HEADER START ###
+import os
+vlm_codebase = os.environ['VLM_CODEBASE_DIR']
+
+import sys
+sys.path.append(vlm_codebase + '/<codebase>')
+### HEADER END ###
+```
+in existing implementations in `generate_lib`. This becomes convenient when the model needs to be run with some author-provided code, and you want to load/call the model with their code. In this way, you have a local directory which contains their code (i.e., if you are developing a model or cloning a codebase from github), and you append the directory to the system PATH.
 
 ### Evaluation
 
@@ -138,6 +149,9 @@ This python script will automatically match the `scores-<model_name>-<mode>_<spl
 We release full results on the validation set (i.e., generated responses, grading done by LLMs and the aggregated stats) for all models we tested in our [HuggingFace Repo](https://huggingface.co/datasets/princeton-nlp/CharXiv/tree/main/existing_evaluations). If you are interested in doing some fine-grained analysis on these results or calculate some customized metrics, feel free to use them.
 
 ## 🏆 Leaderboard
+
+Please refer to our official website by clicking the model name for detailed model information (e.g., huggingface model version or API version).
+
 | Reasoning              | Descriptive                   |
 |------------------------|-------------------------------|
 | TC = Text-in-Chart     | INEX = Information Extraction |
@@ -147,7 +161,63 @@ We release full results on the validation set (i.e., generated responses, gradin
 |                        | COMP = Compositionality       |
 
 
-<table><tbody><tr><th>Metadata</th><th></th><th></th><th>Reasoning</th><th></th><th></th><th></th><th></th><th>Descriptive</th><th></th><th></th><th></th><th></th><th></th></tr><tr><td>Model</td><td>Weight</td><td>Size [V/L] (B)</td><td>Overall</td><td>TC</td><td>TG</td><td>NC</td><td>NG</td><td>Overall</td><td>INEX</td><td>ENUM</td><td>PATT</td><td>CNTG</td><td>COMP</td></tr><tr><td>Human</td><td>N/A</td><td>Unknown</td><td>80.50</td><td>77.27</td><td>77.78</td><td>84.91</td><td>83.41</td><td>92.10</td><td>91.40</td><td>91.20</td><td>95.63</td><td>93.38</td><td>92.86</td></tr><tr><td>Claude 3.5 Sonnet</td><td>Proprietary</td><td>Unknown</td><td>60.20</td><td>61.14</td><td>78.79</td><td>63.79</td><td>46.72</td><td>84.30</td><td>82.62</td><td>88.86</td><td>90.61</td><td>90.08</td><td>48.66</td></tr><tr><td>GPT-4o</td><td>Proprietary</td><td>Unknown</td><td>47.10</td><td>50.00</td><td>61.62</td><td>47.84</td><td>34.50</td><td>84.45</td><td>82.44</td><td>89.18</td><td>90.17</td><td>85.50</td><td>59.82</td></tr><tr><td>Gemini 1.5 Pro</td><td>Proprietary</td><td>Unknown</td><td>43.30</td><td>45.68</td><td>56.57</td><td>45.69</td><td>30.57</td><td>71.97</td><td>81.79</td><td>64.73</td><td>79.48</td><td>76.34</td><td>15.18</td></tr><tr><td>InternVL Chat V2.0 Pro</td><td>Proprietary</td><td>Unknown</td><td>39.80</td><td>40.00</td><td>60.61</td><td>44.40</td><td>25.76</td><td>76.83</td><td>77.11</td><td>84.67</td><td>77.07</td><td>78.88</td><td>27.23</td></tr><tr><td>InternVL Chat V2.0 76B</td><td>Open</td><td>5.9 / 70</td><td>38.90</td><td>40.00</td><td>59.60</td><td>42.67</td><td>24.02</td><td>75.17</td><td>77.11</td><td>78.69</td><td>76.20</td><td>79.13</td><td>32.14</td></tr><tr><td>GPT-4V</td><td>Proprietary</td><td>Unknown</td><td>37.10</td><td>38.18</td><td>57.58</td><td>37.93</td><td>25.33</td><td>79.92</td><td>78.29</td><td>85.79</td><td>88.21</td><td>80.92</td><td>41.07</td></tr><tr><td>GPT-4o Mini</td><td>Proprietary</td><td>Unknown</td><td>34.10</td><td>35.23</td><td>47.47</td><td>32.33</td><td>27.95</td><td>74.92</td><td>74.91</td><td>82.81</td><td>69.21</td><td>79.13</td><td>35.71</td></tr><tr><td>Gemini 1.5 Flash</td><td>Proprietary</td><td>Unknown</td><td>33.90</td><td>36.36</td><td>54.55</td><td>30.60</td><td>23.58</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>InternVL Chat V2.0 26B</td><td>Open</td><td>5.9 / 20</td><td>33.40</td><td>33.18</td><td>51.52</td><td>41.81</td><td>17.47</td><td>62.40</td><td>71.35</td><td>61.02</td><td>55.90</td><td>67.94</td><td>6.25</td></tr><tr><td>Claude 3 Sonnet</td><td>Proprietary</td><td>Unknown</td><td>32.20</td><td>31.59</td><td>50.51</td><td>31.47</td><td>26.20</td><td>73.65</td><td>75.74</td><td>81.92</td><td>76.64</td><td>72.26</td><td>8.48</td></tr><tr><td>Claude 3 Haiku</td><td>Proprietary</td><td>Unknown</td><td>31.80</td><td>29.77</td><td>45.45</td><td>34.48</td><td>27.07</td><td>65.08</td><td>69.87</td><td>69.98</td><td>64.85</td><td>61.83</td><td>8.04</td></tr><tr><td>Phi-3 Vision</td><td>Open</td><td>0.3 / 4</td><td>31.60</td><td>31.36</td><td>46.46</td><td>35.78</td><td>21.40</td><td>60.48</td><td>67.62</td><td>61.18</td><td>54.59</td><td>65.39</td><td>6.25</td></tr><tr><td>Claude 3 Opus</td><td>Proprietary</td><td>Unknown</td><td>30.20</td><td>26.36</td><td>50.51</td><td>33.62</td><td>25.33</td><td>71.55</td><td>75.62</td><td>73.69</td><td>73.58</td><td>70.48</td><td>26.79</td></tr><tr><td>InternVL Chat V1.5</td><td>Open</td><td>5.9 / 20</td><td>29.20</td><td>30.00</td><td>45.45</td><td>32.33</td><td>17.47</td><td>58.50</td><td>69.63</td><td>52.95</td><td>53.06</td><td>64.63</td><td>5.80</td></tr><tr><td>Reka Core</td><td>Proprietary</td><td>Unknown</td><td>28.90</td><td>27.50</td><td>41.41</td><td>28.45</td><td>26.64</td><td>55.60</td><td>58.90</td><td>50.52</td><td>65.72</td><td>71.25</td><td>10.71</td></tr><tr><td>Ovis 1.5 Gemma2 9B</td><td>Open</td><td>0.4 / 9</td><td>28.40</td><td>26.14</td><td>44.44</td><td>33.19</td><td>20.96</td><td>62.60</td><td>64.29</td><td>71.75</td><td>56.33</td><td>66.16</td><td>5.80</td></tr><tr><td>Ovis 1.5 Llama3 8B</td><td>Open</td><td>0.4 / 8</td><td>28.20</td><td>27.27</td><td>49.49</td><td>31.03</td><td>17.90</td><td>60.15</td><td>61.39</td><td>68.93</td><td>56.33</td><td>61.83</td><td>7.14</td></tr><tr><td>Cambrian 34B</td><td>Open</td><td>1.9 / 34</td><td>27.30</td><td>24.55</td><td>44.44</td><td>27.59</td><td>24.89</td><td>59.73</td><td>59.31</td><td>70.94</td><td>53.28</td><td>64.63</td><td>5.36</td></tr><tr><td>Reka Flash</td><td>Proprietary</td><td>Unknown</td><td>26.60</td><td>26.59</td><td>39.39</td><td>30.60</td><td>17.03</td><td>56.45</td><td>61.39</td><td>48.59</td><td>69.87</td><td>72.52</td><td>7.14</td></tr><tr><td>Mini Gemini HD Yi 34B</td><td>Open</td><td>0.5 / 34</td><td>25.00</td><td>26.59</td><td>43.43</td><td>27.16</td><td>11.79</td><td>52.68</td><td>53.86</td><td>55.04</td><td>65.50</td><td>53.94</td><td>2.23</td></tr><tr><td>InternLM XComposer2 4KHD</td><td>Open</td><td>0.3 / 7</td><td>25.00</td><td>23.86</td><td>43.43</td><td>29.31</td><td>14.85</td><td>54.65</td><td>61.09</td><td>54.08</td><td>51.53</td><td>59.80</td><td>6.70</td></tr><tr><td>MiniCPM-V2.5</td><td>Open</td><td>0.4 / 8</td><td>24.90</td><td>25.23</td><td>43.43</td><td>25.43</td><td>15.72</td><td>59.27</td><td>62.28</td><td>61.90</td><td>56.77</td><td>68.96</td><td>10.27</td></tr><tr><td>Qwen VL Max</td><td>Proprietary</td><td>Unknown</td><td>24.70</td><td>26.14</td><td>41.41</td><td>24.57</td><td>14.85</td><td>41.48</td><td>50.42</td><td>28.41</td><td>53.71</td><td>51.15</td><td>4.46</td></tr><tr><td>VILA 1.5 40B</td><td>Open</td><td>5.9 / 34</td><td>24.00</td><td>21.59</td><td>41.41</td><td>25.00</td><td>20.09</td><td>38.67</td><td>42.88</td><td>29.62</td><td>51.31</td><td>50.89</td><td>9.82</td></tr><tr><td>Reka Edge</td><td>Proprietary</td><td>Unknown</td><td>23.50</td><td>20.23</td><td>32.32</td><td>30.60</td><td>18.78</td><td>33.65</td><td>36.65</td><td>28.49</td><td>34.72</td><td>52.16</td><td>4.91</td></tr><tr><td>Gemini 1.0 Pro</td><td>Proprietary</td><td>Unknown</td><td>22.80</td><td>20.91</td><td>48.48</td><td>18.10</td><td>20.09</td><td>54.37</td><td>67.97</td><td>39.23</td><td>60.48</td><td>62.60</td><td>8.93</td></tr><tr><td>LLaVA 1.6 Yi 34B</td><td>Open</td><td>0.3 / 34</td><td>22.50</td><td>20.45</td><td>37.37</td><td>23.71</td><td>18.78</td><td>51.05</td><td>46.38</td><td>63.44</td><td>56.11</td><td>51.91</td><td>5.80</td></tr><tr><td>Mini Gemini HD Llama3 8B</td><td>Open</td><td>0.5 / 8</td><td>19.00</td><td>19.77</td><td>36.36</td><td>21.12</td><td>7.86</td><td>44.42</td><td>49.41</td><td>39.23</td><td>51.09</td><td>55.98</td><td>1.79</td></tr><tr><td>InternLM XComposer2</td><td>Open</td><td>0.3 / 7</td><td>18.70</td><td>16.14</td><td>38.38</td><td>21.98</td><td>11.79</td><td>38.75</td><td>34.10</td><td>43.58</td><td>46.72</td><td>52.93</td><td>5.80</td></tr><tr><td>MiniCPM-V2</td><td>Open</td><td>0.4 / 2.4</td><td>18.50</td><td>17.95</td><td>33.33</td><td>19.40</td><td>12.23</td><td>35.77</td><td>39.74</td><td>36.56</td><td>26.42</td><td>44.53</td><td>5.36</td></tr><tr><td>IDEFICS 2</td><td>Open</td><td>0.4 / 7</td><td>18.20</td><td>15.45</td><td>35.35</td><td>17.24</td><td>17.03</td><td>32.77</td><td>36.12</td><td>27.28</td><td>40.83</td><td>43.26</td><td>3.12</td></tr><tr><td>IDEFICS 2 Chatty</td><td>Open</td><td>0.4 / 7</td><td>17.80</td><td>15.45</td><td>34.34</td><td>19.83</td><td>13.10</td><td>41.55</td><td>34.88</td><td>54.56</td><td>45.63</td><td>44.27</td><td>6.70</td></tr><tr><td>MoAI</td><td>Open</td><td>0.3 / 7</td><td>17.50</td><td>9.32</td><td>36.36</td><td>21.12</td><td>21.40</td><td>28.70</td><td>31.20</td><td>21.23</td><td>39.96</td><td>40.46</td><td>7.59</td></tr><tr><td>DeepSeek VL</td><td>Open</td><td>0.5 / 7</td><td>17.10</td><td>16.36</td><td>32.32</td><td>19.83</td><td>9.17</td><td>45.80</td><td>49.11</td><td>45.20</td><td>42.79</td><td>60.31</td><td>4.91</td></tr><tr><td>SPHINX V2</td><td>Open</td><td>1.9 / 13</td><td>16.10</td><td>13.86</td><td>28.28</td><td>17.67</td><td>13.54</td><td>30.25</td><td>35.59</td><td>24.37</td><td>41.05</td><td>29.52</td><td>1.79</td></tr><tr><td>Qwen VL Plus</td><td>Proprietary</td><td>Unknown</td><td>16.00</td><td>15.45</td><td>45.45</td><td>12.07</td><td>8.30</td><td>28.93</td><td>33.33</td><td>17.92</td><td>32.10</td><td>56.23</td><td>2.23</td></tr><tr><td>LLaVA 1.6 Mistral 7B</td><td>Open</td><td>0.3 / 7</td><td>13.90</td><td>11.36</td><td>32.32</td><td>16.81</td><td>7.86</td><td>35.40</td><td>34.70</td><td>33.98</td><td>48.91</td><td>42.49</td><td>8.48</td></tr><tr><td>ChartGemma</td><td>Open</td><td>0.4 / 2</td><td>12.50</td><td>11.59</td><td>24.24</td><td>16.81</td><td>4.80</td><td>21.30</td><td>27.58</td><td>18.97</td><td>14.19</td><td>19.59</td><td>4.46</td></tr><tr><td>Random (GPT-4o)</td><td>N/A</td><td>Unknown</td><td>10.80</td><td>4.32</td><td>39.39</td><td>5.60</td><td>16.16</td><td>19.85</td><td>21.65</td><td>16.71</td><td>23.80</td><td>25.70</td><td>5.36</td></tr></tbody></table>
+| Metadata                  |                 |                | Reasoning |       |       |       |       | Descriptive |       |       |       |       |       |
+| ------------------------- | --------------- | -------------- | --------- | ----- | ----- | ----- | ----- | ----------- | ----- | ----- | ----- | ----- | ----- |
+| Model                     | Weight          | Size [V/L] (B) | Overall   | TC    | TG    | NC    | NG    | Overall     | INEX  | ENUM  | PATT  | CNTG  | COMP  |
+|🎖️Human                     | N/A             | Unknown        | 80.50     | 77.27 | 77.78 | 84.91 | 83.41 | 92.10       | 91.40 | 91.20 | 95.63 | 93.38 | 92.86 |
+|🥇Claude 3.5 Sonnet         | Proprietary     | Unknown        | 60.20     | 61.14 | 78.79 | 63.79 | 46.72 | 84.30       | 82.62 | 88.86 | 90.61 | 90.08 | 48.66 |
+|🥈GPT-4o                    | Proprietary     | Unknown        | 47.10     | 50.00 | 61.62 | 47.84 | 34.50 | 84.45       | 82.44 | 89.18 | 90.17 | 85.50 | 59.82 |
+|🥉Gemini 1.5 Pro            | Proprietary     | Unknown        | 43.30     | 45.68 | 56.57 | 45.69 | 30.57 | 71.97       | 81.79 | 64.73 | 79.48 | 76.34 | 15.18 |
+| InternVL Chat V2.0 Pro    | Proprietary     | Unknown        | 39.80     | 40.00 | 60.61 | 44.40 | 25.76 | 76.83       | 77.11 | 84.67 | 77.07 | 78.88 | 27.23 |
+| InternVL Chat V2.0 76B    | Open            | 5.9 / 70       | 38.90     | 40.00 | 59.60 | 42.67 | 24.02 | 75.17       | 77.11 | 78.69 | 76.20 | 79.13 | 32.14 |
+| GPT-4V                    | Proprietary     | Unknown        | 37.10     | 38.18 | 57.58 | 37.93 | 25.33 | 79.92       | 78.29 | 85.79 | 88.21 | 80.92 | 41.07 |
+| GPT-4o Mini               | Proprietary     | Unknown        | 34.10     | 35.23 | 47.47 | 32.33 | 27.95 | 74.92       | 74.91 | 82.81 | 69.21 | 79.13 | 35.71 |
+| Gemini 1.5 Flash          | Proprietary     | Unknown        | 33.90     | 36.36 | 54.55 | 30.60 | 23.58 | \-          | \-    | \-    | \-    | \-    | \-    |
+| InternVL Chat V2.0 26B    | Open            | 5.9 / 20       | 33.40     | 33.18 | 51.52 | 41.81 | 17.47 | 62.40       | 71.35 | 61.02 | 55.90 | 67.94 | 6.25  |
+| Claude 3 Sonnet           | Proprietary     | Unknown        | 32.20     | 31.59 | 50.51 | 31.47 | 26.20 | 73.65       | 75.74 | 81.92 | 76.64 | 72.26 | 8.48  |
+| Claude 3 Haiku            | Proprietary     | Unknown        | 31.80     | 29.77 | 45.45 | 34.48 | 27.07 | 65.08       | 69.87 | 69.98 | 64.85 | 61.83 | 8.04  |
+| Phi-3 Vision              | Open            | 0.3 / 4        | 31.60     | 31.36 | 46.46 | 35.78 | 21.40 | 60.48       | 67.62 | 61.18 | 54.59 | 65.39 | 6.25  |
+| MiniCPM-V2.6 (Upsize+CoT) | Open            | 0.4 / 8        | 31.00     | 30.00 | 41.41 | 37.93 | 21.40 | 57.05       | 67.85 | 49.56 | 53.49 | 62.85 | 14.29 |
+| Claude 3 Opus             | Proprietary     | Unknown        | 30.20     | 26.36 | 50.51 | 33.62 | 25.33 | 71.55       | 75.62 | 73.69 | 73.58 | 70.48 | 26.79 |
+| InternVL Chat V1.5        | Open            | 5.9 / 20       | 29.20     | 30.00 | 45.45 | 32.33 | 17.47 | 58.50       | 69.63 | 52.95 | 53.06 | 64.63 | 5.80  |
+| GLM 4V 9B                 | Open            | 4.4 / 9        | 29.10     | 30.68 | 42.42 | 33.19 | 16.16 | 57.62       | 67.97 | 61.66 | 43.45 | 45.04 | 8.48  |
+| Reka Core                 | Proprietary     | Unknown        | 28.90     | 27.50 | 41.41 | 28.45 | 26.64 | 55.60       | 58.90 | 50.52 | 65.72 | 71.25 | 10.71 |
+| Ovis 1.5 Gemma2 9B        | Open            | 0.4 / 9        | 28.40     | 26.14 | 44.44 | 33.19 | 20.96 | 62.60       | 64.29 | 71.75 | 56.33 | 66.16 | 5.80  |
+| Ovis 1.5 Llama3 8B        | Open            | 0.4 / 8        | 28.20     | 27.27 | 49.49 | 31.03 | 17.90 | 60.15       | 61.39 | 68.93 | 56.33 | 61.83 | 7.14  |
+| Cambrian 34B              | Open            | 1.9 / 34       | 27.30     | 24.55 | 44.44 | 27.59 | 24.89 | 59.73       | 59.31 | 70.94 | 53.28 | 64.63 | 5.36  |
+| MiniCPM-V2.6 (Upsize)     | Open            | 0.4 / 8        | 27.10     | 21.59 | 45.45 | 35.34 | 21.40 | 61.62       | 69.28 | 55.93 | 60.48 | 72.01 | 19.64 |
+| Reka Flash                | Proprietary     | Unknown        | 26.60     | 26.59 | 39.39 | 30.60 | 17.03 | 56.45       | 61.39 | 48.59 | 69.87 | 72.52 | 7.14  |
+| Mini Gemini HD Yi 34B     | Open            | 0.5 / 34       | 25.00     | 26.59 | 43.43 | 27.16 | 11.79 | 52.68       | 53.86 | 55.04 | 65.50 | 53.94 | 2.23  |
+| InternLM XComposer2 4KHD  | Open            | 0.3 / 7        | 25.00     | 23.86 | 43.43 | 29.31 | 14.85 | 54.65       | 61.09 | 54.08 | 51.53 | 59.80 | 6.70  |
+| MiniCPM-V2.5              | Open            | 0.4 / 8        | 24.90     | 25.23 | 43.43 | 25.43 | 15.72 | 59.27       | 62.28 | 61.90 | 56.77 | 68.96 | 10.27 |
+| Qwen VL Max               | Proprietary     | Unknown        | 24.70     | 26.14 | 41.41 | 24.57 | 14.85 | 41.48       | 50.42 | 28.41 | 53.71 | 51.15 | 4.46  |
+| VILA 1.5 40B              | Open            | 5.9 / 34       | 24.00     | 21.59 | 41.41 | 25.00 | 20.09 | 38.67       | 42.88 | 29.62 | 51.31 | 50.89 | 9.82  |
+| Reka Edge                 | Proprietary     | Unknown        | 23.50     | 20.23 | 32.32 | 30.60 | 18.78 | 33.65       | 36.65 | 28.49 | 34.72 | 52.16 | 4.91  |
+| Gemini 1.0 Pro            | Proprietary     | Unknown        | 22.80     | 20.91 | 48.48 | 18.10 | 20.09 | 54.37       | 67.97 | 39.23 | 60.48 | 62.60 | 8.93  |
+| LLaVA 1.6 Yi 34B          | Open            | 0.3 / 34       | 22.50     | 20.45 | 37.37 | 23.71 | 18.78 | 51.05       | 46.38 | 63.44 | 56.11 | 51.91 | 5.80  |
+| Mini Gemini HD Llama3 8B  | Open            | 0.5 / 8        | 19.00     | 19.77 | 36.36 | 21.12 | 7.86  | 44.42       | 49.41 | 39.23 | 51.09 | 55.98 | 1.79  |
+| CogAgent                  | Open            | 4.4 / 7        | 18.80     | 16.82 | 32.32 | 20.69 | 14.85 | 36.30       | 45.14 | 26.80 | 43.23 | 37.15 | 6.70  |
+| InternLM XComposer2       | Open            | 0.3 / 7        | 18.70     | 16.14 | 38.38 | 21.98 | 11.79 | 38.75       | 34.10 | 43.58 | 46.72 | 52.93 | 5.80  |
+| MiniCPM-V2                | Open            | 0.4 / 2.4      | 18.50     | 17.95 | 33.33 | 19.40 | 12.23 | 35.77       | 39.74 | 36.56 | 26.42 | 44.53 | 5.36  |
+| IDEFICS 2                 | Open            | 0.4 / 7        | 18.20     | 15.45 | 35.35 | 17.24 | 17.03 | 32.77       | 36.12 | 27.28 | 40.83 | 43.26 | 3.12  |
+| IDEFICS 2 Chatty          | Open            | 0.4 / 7        | 17.80     | 15.45 | 34.34 | 19.83 | 13.10 | 41.55       | 34.88 | 54.56 | 45.63 | 44.27 | 6.70  |
+| MoAI                      | Open            | 0.3 / 7        | 17.50     | 9.32  | 36.36 | 21.12 | 21.40 | 28.70       | 31.20 | 21.23 | 39.96 | 40.46 | 7.59  |
+| DeepSeek VL               | Open            | 0.5 / 7        | 17.10     | 16.36 | 32.32 | 19.83 | 9.17  | 45.80       | 49.11 | 45.20 | 42.79 | 60.31 | 4.91  |
+| DocOwl 1.5 Chat           | Domain-specific | 0.3 / 7        | 17.00     | 14.32 | 34.34 | 15.09 | 16.59 | 37.40       | 36.83 | 49.23 | 36.68 | 22.90 | 3.12  |
+| SPHINX V2                 | Open            | 1.9 / 13       | 16.10     | 13.86 | 28.28 | 17.67 | 13.54 | 30.25       | 35.59 | 24.37 | 41.05 | 29.52 | 1.79  |
+| Qwen VL Plus              | Proprietary     | Unknown        | 16.00     | 15.45 | 45.45 | 12.07 | 8.30  | 28.93       | 33.33 | 17.92 | 32.10 | 56.23 | 2.23  |
+| UReader                   | Domain-specific | 0.3 / 7        | 14.30     | 11.36 | 18.18 | 15.52 | 17.03 | 18.98       | 10.20 | 27.60 | 33.41 | 20.36 | 5.36  |
+| ChartLlama                | Domain-specific | 0.3 / 13       | 14.20     | 8.18  | 34.34 | 9.91  | 21.40 | 19.23       | 17.14 | 12.19 | 43.89 | 28.75 | 6.70  |
+| LLaVA 1.6 Mistral 7B      | Open            | 0.3 / 7        | 13.90     | 11.36 | 32.32 | 16.81 | 7.86  | 35.40       | 34.70 | 33.98 | 48.91 | 42.49 | 8.48  |
+| ChartGemma                | Domain-specific | 0.4 / 2        | 12.50     | 11.59 | 24.24 | 16.81 | 4.80  | 21.30       | 27.58 | 18.97 | 14.19 | 19.59 | 4.46  |
+| ChartAssistant            | Domain-specific | 1.9 / 13       | 11.70     | 9.09  | 27.27 | 10.34 | 11.35 | 16.93       | 16.43 | 16.87 | 16.57 | 27.74 | 2.68  |
+| ChartInstruct-FlanT5      | Domain-specific | 0.3 / 3        | 11.70     | 7.95  | 32.32 | 9.48  | 12.23 | 15.47       | 11.68 | 17.59 | 15.94 | 29.52 | 6.70  |
+| Random (GPT-4o)           | N/A             | Unknown        | 10.80     | 4.32  | 39.39 | 5.60  | 16.16 | 19.85       | 21.65 | 16.71 | 23.80 | 25.70 | 5.36  |
+| DocOwl 1.5 Omni           | Domain-specific | 0.3 / 7        | 9.10      | 5.45  | 14.14 | 9.48  | 13.54 | 25.70       | 34.46 | 17.92 | 31.88 | 17.56 | 4.46  |
+| ChartInstruct-Llama2      | Domain-specific | 0.3 / 7        | 8.80      | 4.09  | 23.23 | 7.76  | 12.66 | 21.40       | 23.31 | 15.50 | 33.19 | 27.48 | 4.91  |
+| TinyChart                 | Domain-specific | 0.4 / 3        | 8.30      | 5.00  | 13.13 | 6.47  | 14.41 | 16.15       | 13.82 | 14.61 | 24.67 | 28.50 | 3.12  |
+| UniChart-ChartQA          | Domain-specific | 1.9 / 8        | 5.70      | 3.41  | 6.06  | 3.45  | 12.23 | 19.32       | 9.91  | 38.26 | 12.23 | 19.08 | 0.45  |
+| TextMonkey                | Domain-specific | 8-Feb          | 3.90      | 2.50  | 4.04  | 3.02  | 7.42  | 12.45       | 12.16 | 17.92 | 8.73  | 6.36  | 2.68  |
 
 ## 📜 License
 Our original data contributions (all data except the charts) are distributed under the [CC BY-SA 4.0](data/LICENSE) license. Our code is licensed under [Apache 2.0](LICENSE) license. The copyright of the charts belong to the original authors, where you can find the source in `image_metadata_val.json` and `image_metadata_test.json` under the data folder.