ER_LOCK_DEADLOCK

[leobarcellos]

I'm receiving constantly deadlock issues and I don't know what else I could do.
I was working with 5 workers, maybe they were competing. However I just modified to just 1 worker and it's still happening.

I even tried putting some locks (using acquireLock and releaseLock) before doing these jobs, but even this doesn't worked (the ER_LOCK_DEADLOCK error is from AWS RDS Aurora MySQL and I'm not sure how can I solve this)

Deadlock issues generally are solved on the application, but I don't know if this is the case, I even tried to use this repo:
https://www.npmjs.com/package/@tanjaae/knex-mysql2-deadlock

But could not manage it to work, I think they conflict with current knex/mysql2 versions.

@pushchris don't you suffer with these deadlock on worker instances?

[leobarcellos]

Is this expected behaviour? Or something is wrong because I have 8k+ jobs on queue?

[leobarcellos]

Update: I increased REDIS_CONCURRENCY of workers and increased memory a little bit, now sometimes I get 0 in queue, and on busy hours it's average 3k in queue

However, I sill see deadlock issues.

[pushchris]

If you have perpetual jobs in the queue you normally should spin up an additional worker to help handle the load. Would generally suggest going that approach vs increasing the concurrency too high. There will most likely always be jobs in the queue though if you have data always coming in, unsure of what your setup is.

As for the deadlocks, looks like they are getting retried (jobs all retry) but still failing which is interesting and leads me to believe whatever the lock is may be fairly long running. Could you see if you can get any more information on the deadlocks? https://www.percona.com/blog/how-to-deal-with-mysql-deadlocks/ would be curious to see what table is preventing the inserts. Typically inserts don't have problems unless 1) the record already exists (which is most likely not the case here since on retry it still fails) or 2) an index behind the record is being blocked by other inserts of operations.