Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

rawx: Performance improvements #1746

Merged
merged 17 commits into from
Apr 24, 2019
Merged

rawx: Performance improvements #1746

merged 17 commits into from
Apr 24, 2019

Conversation

jfsmig
Copy link
Contributor

@jfsmig jfsmig commented Apr 21, 2019

SUMMARY

TL;DR: the golang rawx is now much faster but requires a new dependency on golang.org/x/sys/unix

  • GET now manages to use the sendfile() function
  • Larger buffers at upload
  • file-descriptor oriented xattr operations
  • syscalls parsimony
  • post-reply asynchronous notifications with a buffered channel

Known TODO: make the golang rawx multi-tube.

ISSUE TYPE

enhancement

COMPONENT NAME

rawx

SDS VERSION

openio 4.4.3.dev6

@fvennetier
Copy link
Member

This should be rebased on the master branch since it is a rather big change and requires a new dependency.

@fvennetier fvennetier changed the base branch from 4.4.x to master April 23, 2019 16:48
@codecov-io
Copy link

Codecov Report

Merging #1746 into master will increase coverage by 0.39%.
The diff coverage is 81.46%.

Impacted file tree graph

@@            Coverage Diff            @@
##           master   #1746      +/-   ##
=========================================
+ Coverage   79.12%   79.5%   +0.39%     
=========================================
  Files         361     377      +16     
  Lines       68034   69815    +1781     
  Branches     6885    6920      +35     
=========================================
+ Hits        53822   55501    +1679     
  Misses      14212   14212              
- Partials        0     102     +102
Impacted Files Coverage Δ
rawx/rawx.go 85.19% <100%> (ø)
rawx/chunk_info.go 80.82% <100%> (ø)
rawx/logger.go 73.12% <100%> (ø)
rawx/chunkrepo.go 70.84% <100%> (ø)
rawx/handler_chunk.go 77.63% <67.57%> (ø)
rawx/filerepo.go 75% <76.11%> (ø)
rawx/notifier.go 70.59% <98.04%> (ø)
meta0v2/meta0_remote.c 74.29% <0%> (-8.57%) ⬇️
oio/account/client.py 81.61% <0%> (-5.74%) ⬇️
tests/functional/cli/admin/test_service_decache.py 81.25% <0%> (-4.16%) ⬇️
... and 72 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a34b824...2d2d058. Read the comment docs.

@fvennetier fvennetier merged commit fd88c5e into open-io:master Apr 24, 2019
@jfsmig jfsmig deleted the 4.4.x-rawx-golang-speedup branch April 25, 2019 08:54
# for free to join this conversation on GitHub. Already have an account? # to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants