-
Notifications
You must be signed in to change notification settings - Fork 74
xsv 0.13.0 `stats` compared with qsv 1.0.0 `stats`
Joel Natividad edited this page Dec 4, 2024
·
6 revisions
As referenced in Reddit qsv 1.0.0 announcement thread:
% hyperfine 'xsv stats --everything ~/Downloads/NYC_311_SR_2010-2020-sample-1M.csv -o /tmp/xsvstats-everything.csv'
Benchmark 1: xsv stats --everything ~/Downloads/NYC_311_SR_2010-2020-sample-1M.csv -o /tmp/xsvstats-everything.csv
Time (mean ± σ): 3.246 s ± 0.251 s [User: 15.675 s, System: 1.192 s]
Range (min … max): 3.024 s … 3.887 s 10 runs
% hyperfine 'qsv stats --everything ~/Downloads/NYC_311_SR_2010-2020-sample-1M.csv -o /tmp/qsvstats-everything.csv'
Benchmark 1: qsv stats --everything ~/Downloads/NYC_311_SR_2010-2020-sample-1M.csv -o /tmp/qsvstats-everything.csv
Time (mean ± σ): 2.158 s ± 0.181 s [User: 10.857 s, System: 0.663 s]
Range (min … max): 1.947 s … 2.469 s 10 runs
% hyperfine 'qsv stats --everything --infer-dates ~/Downloads/NYC_311_SR_2010-2020-sample-1M.csv -o /tmp/qsvstats-everything-inferdates.csv'
Benchmark 1: qsv stats --everything --infer-dates ~/Downloads/NYC_311_SR_2010-2020-sample-1M.csv -o /tmp/qsvstats-everything-inferdates.csv
Time (mean ± σ): 2.801 s ± 0.172 s [User: 16.112 s, System: 0.753 s]
Range (min … max): 2.578 s … 3.029 s 10 runs
- These benchmarks were compiled on an Apple Mac Mini 2023 model with an M2 Pro chip with 12 CPU cores & 32GB of RAM with a 1TB SSD running macOS Sequoia 15.2 Developer Beta, using a million row, 41 column, 520 MB sample of NYC's 311 data.
xsv
was installed by runningcargo install xsv
.
12 columns
3.246 secs
https://github.com/dathere/qsv/blob/master/docs/misc/xsvstats-everything.csv
37 columns and four dataset level "stats" (rowcount, colcount, filesize, fingerprint hash)
See https://github.com/dathere/qsv/wiki/Supplemental#stats-command-output-explanation for more info.
2.158 secs
https://github.com/dathere/qsv/blob/master/docs/misc/qsvstats-everything.csv
same as above, but with date-inferencing
2.801 seconds
https://github.com/dathere/qsv/blob/master/docs/misc/qsvstats-everything-inferdates.csv