You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We use Sidekiq Pro's super_fetch so that when a worker is forceably shut down (usually due to a deploy or runaway RAM usage), the job is then added back to the enqueued list when sidekiq starts back up. This works well 99% of the time.
But occasionally what we are seeing is that the digest locks for jobs running when the worker is shut down remain in place, and so Sidekiq fails to reenqueue the job when a replacement worker is spun up, and new jobs of the same type won't queue because the lock remains in place from before the shutdown.
This results in a stuck state. We have to manually go in and delete the locks in order for the jobs to be able to requeue.
Expected behavior
When a Sidekiq worker restarts, before sidekiqs super_fetch kicks in, sidekiq-unique-jobs should remove any locks that were linked to jobs on workers that no longer exist.
e.g. pseudo ruby code: SidekiqUniqueLocks.where.not(worker_id: Sidekiq.workers.map(&:id)).delete_all
Current behavior
Seems it removes the locks most of the time but it has a glitch where it doesn't, and it happens often enough to stop our background jobs from running smoothly.
Describe the bug
We use Sidekiq Pro's super_fetch so that when a worker is forceably shut down (usually due to a deploy or runaway RAM usage), the job is then added back to the enqueued list when sidekiq starts back up. This works well 99% of the time.
But occasionally what we are seeing is that the digest locks for jobs running when the worker is shut down remain in place, and so Sidekiq fails to reenqueue the job when a replacement worker is spun up, and new jobs of the same type won't queue because the lock remains in place from before the shutdown.
This results in a stuck state. We have to manually go in and delete the locks in order for the jobs to be able to requeue.
Expected behavior
When a Sidekiq worker restarts, before sidekiqs super_fetch kicks in, sidekiq-unique-jobs should remove any locks that were linked to jobs on workers that no longer exist.
e.g. pseudo ruby code:
SidekiqUniqueLocks.where.not(worker_id: Sidekiq.workers.map(&:id)).delete_all
Current behavior
Seems it removes the locks most of the time but it has a glitch where it doesn't, and it happens often enough to stop our background jobs from running smoothly.
Worker class
Additional context
Add any other context about the problem here.
The text was updated successfully, but these errors were encountered: