When database is down, worker will down

[vitalyzhakov]

What steps will reproduce the problem?

1. Install yii2-queue, using DB Mysql driver
2. Start process via supervisord https://github.com/yiisoft/yii2-queue/blob/master/docs/guide/worker.md
3. Kill database or drop table queue
4. In php error.log see errors, processes will died.
5. When achieving max number of retries, worker will down forever.
6. Up database. Worker died.

What's expected?

Worker live when database is live.

What do you get instead?

2019-01-16 12:19:23,808 INFO gave up: yii-queue-worker_01 entered FATAL state, too many start retries too quickly
2019-01-16 12:19:23,814 INFO exited: yii-queue-worker_00 (exit status 1; not expected)
2019-01-16 12:19:24,815 INFO gave up: yii-queue-worker_00 entered FATAL state, too many start retries too quickly

Workers is down.

Additional info

Q	A
Yii version	2.0.15.1
PHP version	7.1.14
Operating system	Debian 8

[samdark]

That should be solved with supervisor.

[samdark]

https://github.com/yiisoft/yii2-queue/blob/master/docs/guide/worker.md

[vitalyzhakov]

@samdark ,
I did not find the solution about this issue on docs page
https://github.com/yiisoft/yii2-queue/blob/master/docs/guide/worker.md

Can you tell solution?

[samdark]

Do you really want it trying to connect to dead database forever writing to logs endlessly?

[samdark]

autorestart=true this will ensure supervisord will try to start worker again if it terminated abnormally.

[vitalyzhakov]

Do you really want it trying to connect to dead database forever writing to logs endlessly?

@samdark , I think, automatic disaster recovery is the best way.

But if it is difficult now to develop, description about this library behavior and example solution may be situate in docs.

If I understand, supervisord config file must include startretries=99999. Is it right?

[samdark]

Yes. The time between retries will increment exponentially though.

[vitalyzhakov]

@samdark , hello!
The time between retries will increment exponentially though.
Supervisord does not have this option now Supervisor/supervisor#659

[samdark]

It is exponential by default.

[YanAlex]

There is a problem with Windows, there are no supervisot support

[samdark]

https://stackoverflow.com/questions/7629813/is-there-windows-analog-to-supervisord

[samdark]

Closing since actually it's mentioned in current docs.