Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

feat: piecereader: Allow parallel access #10895

Closed
wants to merge 4 commits into from

Conversation

magik6k
Copy link
Contributor

@magik6k magik6k commented May 19, 2023

Related Issues

Closes #10532

Proposed Changes

  • Allow pieceReader reader-getters to define exact read size
  • Rewrite pieceReader.ReadAt to be called in parallel, while still caching small amounts ahead for parallel sequential reads
    • This is necessary, because carv1 reads byte-by-byte for the header/cid varints, saves ~9 requests per block read
  • Kept the old more aggressive prefetching mechanism for ReadSeeker, which is only used by the indexing scan, and makes it wayy faster

Additional Info

  • Make sure tests pass
  • Test on a real setup, best in booster-bitswap
    • With a race detector

Checklist

Before you mark the PR ready for review, please make sure that:

  • Commits have a clear commit message.
  • PR title is in the form of of <PR type>: <area>: <change being made>
    • example: fix: mempool: Introduce a cache for valid signatures
    • PR type: fix, feat, build, chore, ci, docs, perf, refactor, revert, style, test
    • area, e.g. api, chain, state, market, mempool, multisig, networking, paych, proving, sealing, wallet, deps
  • New features have usage guidelines and / or documentation updates in
  • Tests exist for new functionality or change in behavior
  • CI is green

@magik6k magik6k requested a review from a team as a code owner May 19, 2023 16:52
@magik6k magik6k changed the title Feat/palallel piecereader feat: pierereader: Allow parallel access May 19, 2023
@magik6k magik6k changed the title feat: pierereader: Allow parallel access feat: piecereader: Allow parallel access May 23, 2023
@magik6k magik6k closed this May 23, 2023
@rjan90 rjan90 deleted the feat/palallel-piecereader branch January 31, 2025 08:24
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Re-tune piece-reader to allow for random parallel access of data
1 participant