Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

feat(ch): use writev when possible #413

Merged
merged 11 commits into from
Sep 25, 2024
Merged

Conversation

tdakkota
Copy link
Contributor

@tdakkota tdakkota commented Aug 2, 2024

Reduce memory footprint by avoiding unnecessary data copying during inserts.

goos: linux
goarch: amd64
pkg: github.com/ClickHouse/ch-go
cpu: AMD Ryzen 9 5950X 16-Core Processor            
                        │   old.txt    │               new.txt               │
                        │    sec/op    │   sec/op     vs base                │
Insert/Rows10000-32       396.3µ ±  6%   421.2µ ± 1%   +6.27% (p=0.000 n=15)
Insert/Rows100000-32      521.8µ ±  6%   687.5µ ± 1%  +31.76% (p=0.000 n=15)
Insert/Rows1000000-32     3.562m ± 12%   3.485m ± 1%        ~ (p=0.305 n=15)
Insert/Rows10000000-32    32.29m ±  3%   31.77m ± 2%   -1.61% (p=0.023 n=15)
Insert/Rows100000000-32   421.7m ±  6%   230.5m ± 9%  -45.35% (p=0.000 n=15)
geomean                   6.314m         5.939m        -5.93%

                        │    old.txt    │               new.txt                │
                        │      B/s      │     B/s       vs base                │
Insert/Rows10000-32       192.5Mi ±  7%   181.1Mi ± 1%   -5.90% (p=0.000 n=15)
Insert/Rows100000-32      1.428Gi ±  6%   1.084Gi ± 1%  -24.10% (p=0.000 n=15)
Insert/Rows1000000-32     2.092Gi ± 14%   2.138Gi ± 1%        ~ (p=0.305 n=15)
Insert/Rows10000000-32    2.307Gi ±  2%   2.345Gi ± 2%   +1.64% (p=0.023 n=15)
Insert/Rows100000000-32   1.767Gi ±  6%   3.233Gi ± 8%  +82.97% (p=0.000 n=15)
geomean                   1.180Gi         1.255Gi        +6.31%

                        │      old.txt      │                new.txt                │
                        │       B/op        │     B/op      vs base                 │
Insert/Rows10000-32           10.31Ki ±  0%   10.45Ki ± 0%    +1.42% (p=0.000 n=15)
Insert/Rows100000-32          10.64Ki ±  0%   10.66Ki ± 0%    +0.19% (p=0.000 n=15)
Insert/Rows1000000-32         32.03Ki ±  9%   10.94Ki ± 0%   -65.85% (p=0.000 n=15)
Insert/Rows10000000-32      2309.09Ki ±  8%   11.59Ki ± 0%   -99.50% (p=0.000 n=15)
Insert/Rows100000000-32   260431.62Ki ± 25%   12.43Ki ± 1%  -100.00% (p=0.000 n=15)
geomean                       291.7Ki         11.19Ki        -96.16%

                        │  old.txt   │              new.txt              │
                        │ allocs/op  │ allocs/op   vs base               │
Insert/Rows10000-32       325.0 ± 0%   328.0 ± 0%  +0.92% (p=0.000 n=15)
Insert/Rows100000-32      325.0 ± 0%   328.0 ± 0%  +0.92% (p=0.000 n=15)
Insert/Rows1000000-32     326.0 ± 0%   329.0 ± 0%  +0.92% (p=0.000 n=15)
Insert/Rows10000000-32    326.0 ± 0%   329.0 ± 0%  +0.92% (p=0.000 n=15)
Insert/Rows100000000-32   331.0 ± 0%   333.0 ± 0%  +0.60% (p=0.000 n=15)
geomean                   326.6        329.4       +0.86%

@tdakkota tdakkota force-pushed the perf/use-writev branch 2 times, most recently from 333a54b to 949f3cb Compare September 25, 2024 02:07
`proto.Writer` API still reduces total memory consumption and prevents unnecessary data copying
@tdakkota
Copy link
Contributor Author

At c420f0b

goos: linux
goarch: amd64
pkg: github.com/ClickHouse/ch-go
cpu: AMD Ryzen 9 5950X 16-Core Processor            
                        │   old.txt    │               new.txt                │
                        │    sec/op    │    sec/op     vs base                │
Insert/Rows10000-32       454.3µ ±  1%    471.0µ ± 2%   +3.66% (p=0.000 n=15)
Insert/Rows100000-32      851.8µ ±  3%   1003.2µ ± 2%  +17.77% (p=0.000 n=15)
Insert/Rows1000000-32     7.611m ±  1%    6.609m ± 1%  -13.16% (p=0.000 n=15)
Insert/Rows10000000-32    62.03m ± 17%    58.12m ± 3%   -6.30% (p=0.000 n=15)
Insert/Rows100000000-32   758.8m ± 13%    515.4m ± 5%  -32.08% (p=0.000 n=15)
geomean                   10.68m          9.867m        -7.57%

                        │    old.txt     │                new.txt                │
                        │      B/s       │      B/s       vs base                │
Insert/Rows10000-32        167.9Mi ±  1%    162.0Mi ± 2%   -3.53% (p=0.000 n=15)
Insert/Rows100000-32       895.7Mi ±  4%    760.5Mi ± 2%  -15.09% (p=0.000 n=15)
Insert/Rows1000000-32     1002.4Mi ±  1%   1154.4Mi ± 1%  +15.16% (p=0.000 n=15)
Insert/Rows10000000-32     1.201Gi ± 14%    1.282Gi ± 3%   +6.72% (p=0.000 n=15)
Insert/Rows100000000-32   1005.4Mi ± 11%   1480.4Mi ± 5%  +47.24% (p=0.000 n=15)
geomean                    714.7Mi          773.2Mi        +8.19%

                        │      old.txt      │                new.txt                │
                        │       B/op        │     B/op      vs base                 │
Insert/Rows10000-32           10.38Ki ±  0%   10.89Ki ± 0%    +4.95% (p=0.000 n=15)
Insert/Rows100000-32          10.93Ki ±  0%   10.92Ki ± 0%    -0.16% (p=0.002 n=15)
Insert/Rows1000000-32         60.91Ki ±  2%   11.33Ki ± 0%   -81.41% (p=0.000 n=15)
Insert/Rows10000000-32      4123.41Ki ± 36%   11.92Ki ± 0%   -99.71% (p=0.000 n=15)
Insert/Rows100000000-32   390642.05Ki ±  0%   13.67Ki ± 6%  -100.00% (p=0.000 n=15)
geomean                       406.8Ki         11.70Ki        -97.12%

                        │  old.txt   │              new.txt              │
                        │ allocs/op  │ allocs/op   vs base               │
Insert/Rows10000-32       325.0 ± 0%   329.0 ± 0%  +1.23% (p=0.000 n=15)
Insert/Rows100000-32      325.0 ± 0%   329.0 ± 0%  +1.23% (p=0.000 n=15)
Insert/Rows1000000-32     326.0 ± 0%   330.0 ± 0%  +1.23% (p=0.000 n=15)
Insert/Rows10000000-32    326.0 ± 0%   330.0 ± 0%  +1.23% (p=0.000 n=15)
Insert/Rows100000000-32   333.0 ± 1%   334.0 ± 0%       ~ (p=0.685 n=15)
geomean                   327.0        330.4       +1.04%

@tdakkota tdakkota marked this pull request as ready for review September 25, 2024 02:45
@ernado ernado merged commit e4a2d07 into ClickHouse:main Sep 25, 2024
25 checks passed
@tdakkota tdakkota deleted the perf/use-writev branch September 25, 2024 22:56
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants