Update ClickBench benchmarks with DataFusion 34

[alamb]

Is your feature request related to a problem or challenge?

DataFusion 34 has been released. It woudl be great to update ClickBench https://benchmark.clickhouse.com/ with runs from the latest version.

It appears we still are at 33: https://github.com/ClickHouse/ClickBench/blob/main/datafusion/results/partitioned.json

Describe the solution you'd like

Perhaps we can follow the model of ClickHouse/ClickBench#145 (thanks @kmitchener )

Describe alternatives you've considered

No response

Additional context

No response

[kmitchener]

I re-ran the benchmark and v34 is actually a little bit slower than v33. I filter the benchmark for just type: "stateless" on machine: "c6a.4xlarge, 500gb gp2" to compare against similar systems. You can compare the index.html from the PR to the one on the site: filtered benchmark.

[alamb]

I re-ran the benchmark and v34 is actually a little bit slower than v33. I filter the benchmark for just type: "stateless" on machine: "c6a.4xlarge, 500gb gp2" to compare against similar systems. You can compare the index.html from the PR to the one on the site: filtered benchmark.

Thanks @kmitchener !

Do you have a sense for how reproduceable the numbers are? Many of the reported performance differences are less than 100ms, and at that range I have found significant variation.

I wonder if you ran it more than once and are they reproducable?

[kmitchener]

Do you have a sense for how reproduceable the numbers are? Many of the reported performance differences are less than 100ms, and at that range I have found significant variation.

I wonder if you ran it more than once and are they reproducible?

I ran this on the same instance that I ran the v33 benchmarks on (it's been powered down since the v33 run). The numbers are consistently slightly slower compared to v33. I ran it twice, since it was unexpected. I also updated to latest Rust and ran again -- that's what I published.

I'll run the v33 benchmark again on the same instance and see what it produces.

[kmitchener]

I've re-run the v33 benchmarks on the same instance and modified the benchmark so it will display both 33 and 34 at the same time so you can compare the runs:

You can grab that from -> https://github.com/kmitchener/ClickBench/blob/new-run-of-datafusion-33/index.html

[alamb]

🤔 I don't really have any good reason / understanding of what is different

I looked at q13 briefly which has the largest reported change and I don't really have any good explanation about why it is reported 8% slower.

And I profiled running it locally,

It doesn't seem to show anything obvious -- the biggest offenders are managing the group keys which I don't think changed much

[alamb]

Thanks @kmitchener -- I filed #8836 to look into the results

[alamb]

The results for 34 were merged, and we just release 36 (including #8827) . Let's see track the work for Updating ClickBench for DataFusion 36 in #9404

Thanks again @kmitchener