Multiple jobs running at the same time

[macmartine]

Describe the bug
Not sure if this is an issue or user error, but definitely feels like it should be working.

I'm trying to not be able to queue the same job multiple times (while it's already in the queue or running).

Rails 2.5.1 Sidekiq 6.0.6

Expected behavior
I'd expect to not be able to queue/run the same job twiece

Current behavior
Runs the same job multiple times

Worker class

class RefreshLoginWorker

  include Sidekiq::Worker
  include Sidekiq::Status::Worker

  sidekiq_options unique_across_workers: true, queue: 'credentials', :retry => false, lock: :until_executed

  def perform item_id
    sleep 5
  end

end

Additional context

Here we kick off some job one after another:

irb(main):016:0> RefreshLoginWorker.perform_async 20
=> "4e83f7a419b9aa047860af77"
irb(main):017:0> RefreshLoginWorker.perform_async 20
=> "682776c2fb45866f35cbf87f"
irb(main):018:0> RefreshLoginWorker.perform_async 20
=> "962dc0bae18cfc4ede6aea18"

Sidekiq log:

2020-07-18T17:30:48.468Z pid=561643 tid=gpbejyaar class=RefreshLoginWorker jid=4e83f7a419b9aa047860af77 INFO: start
2020-07-18T17:30:49.060Z pid=561643 tid=gpbejy9vf class=RefreshLoginWorker jid=682776c2fb45866f35cbf87f INFO: start
2020-07-18T17:30:53.469Z pid=561643 tid=gpbejyaar class=RefreshLoginWorker jid=4e83f7a419b9aa047860af77 elapsed=5.001 INFO: done
2020-07-18T17:30:53.470Z pid=561643 tid=gpbejyaar class=RefreshLoginWorker jid=962dc0bae18cfc4ede6aea18 INFO: start
2020-07-18T17:30:54.061Z pid=561643 tid=gpbejy9vf class=RefreshLoginWorker jid=682776c2fb45866f35cbf87f elapsed=5.001 INFO: done
2020-07-18T17:30:58.471Z pid=561643 tid=gpbejyaar class=RefreshLoginWorker jid=962dc0bae18cfc4ede6aea18 elapsed=5.001 INFO: done

[mhenrixon]

Any particular reason you are using unique_across_workers? Could you remove that and try without?

Also, which gem version are you on?

[macmartine]

@mhenrixon

Honestly, 'unique_across_workers' was just leftover from trying different options to get anything to work.

Same result without it.

Using version 5.0.2, as that's the version that installed by default when I included the gem.

Any ideas?

[loneaggie]

We are experiencing something similar, our options look like this:

sidekiq_options queue: :user, lock: :until_executed, backtrace: true, retry: 3, on_conflict: :log

1. We enqueue the job for work
2. Let the worker start the job (full job takes 20 minutes or so)
3. We verify in the UI there is a single unique digest
4. We kill the worker to simulate an infrastructure failure etc.
5. We verify the unique digest is still there
6. We spin up a new worker, and its starts working on the job, and it enqueues a new job (the job recovery is from SK Pro Super Fetch code)

I could understand the first worker picking the job out of super fetch and starting it, but the second is really blowing my mind.

[mhenrixon]

@loneaggie super fetch gets the job. It isn't a second one, it is the first one being restarted since you have retry turned on. The lock should stay until the worker is done performing the work.

Highly recommend giving SidekiqUniqueJobs v7.0.1 a try instead (you too @macmartine)