Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Improve performance of Redis queue block_for when a worker has multiple queues to service #52826

Open
wants to merge 4 commits into
base: 11.x
Choose a base branch
from

Conversation

michael-scinocca
Copy link

This is a change to attempt to fix/improve #51612

I'm aware this was closed as not a bug, but it is still impacting some people and I believe there is still room for performance improvement here.

The crux of the issue is if you use the redis queue driver with block_for with a worker servicing multiple queues, it will always block for the block_for time before it can get to a job on a secondary (and further) queue. For example - if you configure block_for as 5 and run queue:work with --queue=default,low and are in a situation where you have 0 jobs in default a 100 jobs in low, the current implementation will block on default for 5 seconds each time before it runs a job in low, so we're getting 5 * 100 = 500 seconds of delay while getting through those 100 jobs.

What this change does is track if we had a job to process in our last round of checking the queues and not block in our next check if so. With this, if we got a job from the low queue in our last job round, we will not block on the default queue our next check to remove the delay before getting back to the low queue. If we eventually have no jobs on any queues we will go back to blocking each check. This maintains the efficiency of blocking in most cases without adding a delay if we have jobs to process. This also maintains priority of the default queue - it will still check default first, it just won't block under this condition.

I had to expose the $block out to the Queue\Worker, which I understand is specific to the RedisQueue implementation and not applicable for other connections, however, blocking as a concept is not specific to Redis, it could be applicable for other implementations, so I feel it doesn't muddy the waters.

# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant