-
Notifications
You must be signed in to change notification settings - Fork 330
New issue
Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? # to your account
Performance is misleading #9
Comments
I'll get you started: class EasyProcessor
include Sneakers::Worker
from_queue :work
def work(msg)
puts msg
sleep 1
end
end class EasyWorker
include Sidekiq::Worker
def perform(msg)
puts msg
sleep 1
end
end |
I really do wish you luck with Sneakers but remember that your users trust is valuable. Unrealistic benchmarks might get you some immediate PR but hurt that trust long-term. Saying you are 10x faster than Sidekiq is simply not true. |
Hi Mike. I'll start by saying I'm a fan of you as a developer and Sidekiq as a tool. Been using Sidekiq in production for years now. I'll take your advice seriously - and I hope the following discussion makes sense. I'll continue by saying that I hate to reinvent the wheel. It's been stated here with an in-depth explanation https://github.com/jondot/sneakers/wiki/Why-i-built-it
Eventually I did what every engineer would/should do - measure per my workload, use case, and budget. I benchmarked against Sidekiq because Sidekiq is my reference benchmark. It is what I start every project with and I love it. Microbenchmarks are silly. And it's stated at the opening paragraph - but - I will make sure to state it in a more explicit way and explain why it is silly. However, when dealing with infrastructure like I do on a daily basis, I use microbenchmarks to measure my upper bound, happy path of a tool which gives me an initial idea of the plumbing. What you're saying is completely true - doing long, I/O bound jobs would completely obliterate any performance differences when set up in a similar manner (same size threadpool, same process allocation). But If you ever need to do very fast jobs (such as event processing, which is one of my use cases), it starts to matter. Finally, if I included an I/O bound use case for a performance comparison - this would be more misleading because there's no limit to how slow or how complex or how strange your use case is. In some arbitrary cases Sidekiq, or Resque, or delayed_job or even a simple cron job would win over your KPIs. I will state it explicitly - the best way to benchmark is by oneself - with one's own use case. As a conclusion I will make the following updates to the Wiki page, which admittedly, was initially written in haste.
Sneakers was closed-source for a while, and I've used it for my own benefit for a while. My thinking was to open-source it and share it with others so they could be happy as I am with it. In an effort to promote a healthy discussion - I hope to get feedback from you about my suggested alterations if this is acceptable. Thanks Edit: I've now completed the updates. |
You are marketing Sneakers as a general purpose framework for background jobs and the common case is jobs with I/O, sending email, talking to a database, etc. MRI is not a good VM for CPU intensive tasks so people don't use it for things like real-time stream processing. Your use case might be perfectly reasonable but isn't common in the Ruby world. A benchmark which fakes some I/O via sleep is more general purpose IME. I have no doubt that Sneakers and RabbitMQ can dispatch more messages per second. Sidekiq has to make 3 roundtrips to Redis for each job, suboptimal but that's the price of using a general purpose tool like Redis. Thanks for updating the wiki page. The other thing I'd ask is that you publish the code for the benchmark so others can reproduce your results. |
Thanks Mike. I wasn't intending to market it as general purpose since Sidekiq is already great at that (actually I'm not sure I am even marketing per-se). In a retrospect, analyzing my wording, I guess I hoped attaching "performance" to the tag line would hit the cases where performance-oriented work loads is more important than the general work or all-purpose jobs, for example - job profiles with low-latency and high req/s such as event and log processing based scenarios. i.e. perhaps the difference is better illustrated here: all purpose background-processing framework for Ruby (Sidekiq) Seemed to me, at the time, that picking the second one focused my library at those niche cases where you need high-performance job profiles to work. Obviously, I use both Sidekiq and Sneakers to cover both those cases in production myself. Regarding stream processing on MRI, might be beneficial to check out Anecdotally, Sneakers is based on |
Wouldn't a benchmark that includes sleeps just add noise? "Oh look everything is one second slower now that we added a one second sleep". I care about the overhead on the framework the IO time isn't going to suddenly change because my job is in sneakers now...
|
@Donavan Do your jobs take no time to process? If Sneakers takes 50 usec of overhead and Sidekiq takes 100 usec of overhead but your job takes one second to actually process, you are talking about a miniscule difference in real world performance. If your jobs take 1 ms to process, the overhead becomes relevant but most people don't have jobs that take 1 ms to process. |
Your Performance wiki page is misleading. Benchmarking noops isn't anything close to realistic or useful to potential users. Please provide a real benchmark where each message requires some real processing (hits a database, performs some calculation, etc) and I think you'll find that Sneakers doesn't process jobs any faster than Sidekiq.
The text was updated successfully, but these errors were encountered: