Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Cache the result of forerunner to a tmp file #177

Merged
merged 8 commits into from
Dec 23, 2019

Conversation

liuchengxu
Copy link
Owner

@liuchengxu liuchengxu commented Dec 22, 2019

Close #98

When the result set is too large for the built-in sync filter, and the furerunner job is completed, save the results to a tmp file instead of always running the heavy command again, which could be 2x faster.

$ cd ~ && fd --type f > home.txt

# Reading from a file
$ time /Users/xlc/.vim/plugged/vim-clap/target/release/maple "srli" --input /Users/xlc/home.txt --number 96
real	0m1.674s
user	0m1.640s
sys	0m0.093s

# Runing the command and pip into maple
$ time fd --type f | /Users/xlc/.vim/plugged/vim-clap/target/release/maple "srli" --number 96
real	0m3.173s
user	0m6.310s
sys	0m13.075s

This commit 3557163 using rayon's par_lines gains 3x faster again than reading file using std lib.

$ time /Users/xlc/.vim/plugged/vim-clap/target/release/maple "srli" --input /Users/xlc/home.txt --number 96
real	0m0.491s
user	0m2.838s
sys	0m0.158s

In conclusion, reading from a cached file can be 6x faster then using the pipe command.

@liuchengxu liuchengxu merged commit 4df7d68 into master Dec 23, 2019
@delete-merged-branch delete-merged-branch bot deleted the cache-forerunner-to-tmpfile branch December 23, 2019 10:50
liuchengxu pushed a commit that referenced this pull request Aug 17, 2021
* Create Starry_Network_milestone_1.md

* Starry.Network milestone 1 delivery

* update notes

* update notes

* Fix the wrong sequence

* update Article and Tutorial
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Spawn a forerunner job for the pure async provider
1 participant