Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Lock digests not always removed after forced sidekiq worker shutdown #861

Open
KieranP opened this issue Dec 16, 2024 · 0 comments
Open

Comments

@KieranP
Copy link

KieranP commented Dec 16, 2024

Describe the bug

We use Sidekiq Pro's super_fetch so that when a worker is forceably shut down (usually due to a deploy or runaway RAM usage), the job is then added back to the enqueued list when sidekiq starts back up. This works well 99% of the time.

But occasionally what we are seeing is that the digest locks for jobs running when the worker is shut down remain in place, and so Sidekiq fails to reenqueue the job when a replacement worker is spun up, and new jobs of the same type won't queue because the lock remains in place from before the shutdown.

This results in a stuck state. We have to manually go in and delete the locks in order for the jobs to be able to requeue.

Expected behavior

When a Sidekiq worker restarts, before sidekiqs super_fetch kicks in, sidekiq-unique-jobs should remove any locks that were linked to jobs on workers that no longer exist.

e.g. pseudo ruby code: SidekiqUniqueLocks.where.not(worker_id: Sidekiq.workers.map(&:id)).delete_all

Current behavior

Seems it removes the locks most of the time but it has a glitch where it doesn't, and it happens often enough to stop our background jobs from running smoothly.

Worker class

class MyWorker
  include Sidekiq::Worker
  sidekiq_options(
    queue: :default,
    lock: :until_executed,
  )
end

Additional context
Add any other context about the problem here.

# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant