Weblate Backup stuck

[zisuu]

Hello

We are using Weblate on a Kubernetes platform, using the official Weblate helm chart. We are running website version 5.5

Suddenly, we noticed no new backups in the WebUI Backup overview (https://translate.tools.pnet.ch/manage/backup/). The Weblate instance has been running for over a year, and we have never had issues with the Backup.

We also got two Alerts in the performance report:
weblate.E009 | The Celery tasks queue is too long, either the worker is not running, or is too slow.
weblate.E019 | The Celery does not process tasks, or is too slow in processing them.

We could fix this for a short period by just killing the pods. After that, the Alerts were gone, and the backup continued to work for up to three days. Now we have the same issue again.

I guess it could be that we are hitting some resource constraints. However, my main issue is that I can't see/find any celery logs. The pod does not have a /var/log/celery.log . I can't find anything celery related in the pod logs of Weblate, nor in the Redis pod logs.

I have seen that it is recommended to set up collecting error reports and monitoring performance. But, because we are running in a strictly regulated environment, we are not able to use such a SaaS service to forward our logs. That's why we don't have such a setup.

Thanks, anyone, for a hint on how to troubleshoot this further.

[nijel]

Try upgrading to 5.6, that considerably reduced memory usage and that might be causing issues for you.