Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

DockerCompose: worker_1 error #6

Open
AppliNH opened this issue Jul 22, 2020 · 2 comments
Open

DockerCompose: worker_1 error #6

AppliNH opened this issue Jul 22, 2020 · 2 comments

Comments

@AppliNH
Copy link

AppliNH commented Jul 22, 2020

Hello,

When I run docker-compose up using the latest images, I get the following error :

worker_1  | [2020-07-22 14:24:38,075: ERROR/MainProcess] Control command error: OperationalError("\nCannot route message for exchange 'reply.celery.pidbox': Table empty or key no longer exists.\nProbably the key ('_kombu.binding.reply.celery.pidbox') has been removed from the Redis database.\n",)
worker_1  | Traceback (most recent call last):
worker_1  |   File "/usr/lib/python3.6/site-packages/kombu/connection.py", line 439, in _reraise_as_library_errors
worker_1  |     yield
worker_1  |   File "/usr/lib/python3.6/site-packages/kombu/connection.py", line 518, in _ensured
worker_1  |     return fun(*args, **kwargs)
worker_1  |   File "/usr/lib/python3.6/site-packages/kombu/messaging.py", line 203, in _publish
worker_1  |     mandatory=mandatory, immediate=immediate,
worker_1  |   File "/usr/lib/python3.6/site-packages/kombu/transport/virtual/base.py", line 605, in basic_publish
worker_1  |     message, exchange, routing_key, **kwargs
worker_1  |   File "/usr/lib/python3.6/site-packages/kombu/transport/virtual/exchange.py", line 70, in deliver
worker_1  |     for queue in _lookup(exchange, routing_key):
worker_1  |   File "/usr/lib/python3.6/site-packages/kombu/transport/redis.py", line 877, in _lookup
worker_1  |     exchange, redis_key))
worker_1  | kombu.exceptions.InconsistencyError:
worker_1  | Cannot route message for exchange 'reply.celery.pidbox': Table empty or key no longer exists.
worker_1  | Probably the key ('_kombu.binding.reply.celery.pidbox') has been removed from the Redis database.
worker_1  |
worker_1  |
worker_1  | During handling of the above exception, another exception occurred:
worker_1  |
worker_1  | Traceback (most recent call last):
worker_1  |   File "/usr/lib/python3.6/site-packages/celery/worker/pidbox.py", line 46, in on_message
worker_1  |     self.node.handle_message(body, message)
worker_1  |   File "/usr/lib/python3.6/site-packages/kombu/pidbox.py", line 145, in handle_message
worker_1  |     return self.dispatch(**body)
worker_1  |   File "/usr/lib/python3.6/site-packages/kombu/pidbox.py", line 115, in dispatch
worker_1  |     ticket=ticket)
worker_1  |   File "/usr/lib/python3.6/site-packages/kombu/pidbox.py", line 151, in reply
worker_1  |     serializer=self.mailbox.serializer)
worker_1  |   File "/usr/lib/python3.6/site-packages/kombu/pidbox.py", line 285, in _publish_reply
worker_1  |     **opts
worker_1  |   File "/usr/lib/python3.6/site-packages/kombu/messaging.py", line 181, in publish
worker_1  |     exchange_name, declare,
worker_1  |   File "/usr/lib/python3.6/site-packages/kombu/connection.py", line 551, in _ensured
worker_1  |     errback and errback(exc, 0)
worker_1  |   File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__
worker_1  |     self.gen.throw(type, value, traceback)
worker_1  |   File "/usr/lib/python3.6/site-packages/kombu/connection.py", line 444, in _reraise_as_library_errors
worker_1  |     sys.exc_info()[2])
worker_1  |   File "/usr/lib/python3.6/site-packages/vine/five.py", line 194, in reraise
worker_1  |     raise value.with_traceback(tb)
worker_1  |   File "/usr/lib/python3.6/site-packages/kombu/connection.py", line 439, in _reraise_as_library_errors
worker_1  |     yield
worker_1  |   File "/usr/lib/python3.6/site-packages/kombu/connection.py", line 518, in _ensured
worker_1  |     return fun(*args, **kwargs)
worker_1  |   File "/usr/lib/python3.6/site-packages/kombu/messaging.py", line 203, in _publish
worker_1  |     mandatory=mandatory, immediate=immediate,
worker_1  |   File "/usr/lib/python3.6/site-packages/kombu/transport/virtual/base.py", line 605, in basic_publish
worker_1  |     message, exchange, routing_key, **kwargs
worker_1  |   File "/usr/lib/python3.6/site-packages/kombu/transport/virtual/exchange.py", line 70, in deliver
worker_1  |     for queue in _lookup(exchange, routing_key):
worker_1  |   File "/usr/lib/python3.6/site-packages/kombu/transport/redis.py", line 877, in _lookup
worker_1  |     exchange, redis_key))
worker_1  | kombu.exceptions.OperationalError:
worker_1  | Cannot route message for exchange 'reply.celery.pidbox': Table empty or key no longer exists.
worker_1  | Probably the key ('_kombu.binding.reply.celery.pidbox') has been removed from the Redis database.

And the web interface can't be reached on port 5000.
Any ideas ?

Thanks a lot :)
@AppliNH
Copy link
Author

AppliNH commented Jul 22, 2020

I ended up modifying a bit the docker-compose file, and now the web interface is accessible.

However, whenever I upload a file, I get the following error from the webApp :

web_1     | [I 200722 15:01:34 web:2064] 200 GET / (192.168.160.1) 2.49ms
plik_1    | [07/22/2020 15:01:39][INFO    ][[192.168.160.4]] POST /upload
plik_1    | [07/22/2020 15:01:39][INFO    ][[192.168.160.4]] POST /file/Tk3Z5RH5Vnu0uPGV/xwCStjzbXQlnRf4c/STDIN
web_1     | [E 200722 15:01:39 web:1591] Uncaught exception POST /v1/tasks (192.168.160.1)
web_1     |     HTTPServerRequest(protocol='http', host='localhost:5000', method='POST', uri='/v1/tasks', version='HTTP/1.1', remote_ip='192.168.160.1', headers={'Host': 'localhost:5000', 'Connection': 'keep-alive', 'Content-Length': '172550', 'Accept': '*/*', 'X-Requested-With': 'XMLHttpRequest', 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36', 'Content-Type': 'multipart/form-data; boundary=----WebKitFormBoundaryAEDdkpBolFa0ScZX', 'Origin': 'http://localhost:5000', 'Sec-Fetch-Site': 'same-origin', 'Sec-Fetch-Mode': 'cors', 'Sec-Fetch-Dest': 'empty', 'Referer': 'http://localhost:5000/', 'Accept-Encoding': 'gzip, deflate, br', 'Accept-Language': 'fr-FR,fr;q=0.9,en-GB;q=0.8,en;q=0.7,en-US;q=0.6'})
web_1     |     Traceback (most recent call last):
web_1     |       File "/usr/local/lib/python3.7/site-packages/kombu/utils/objects.py", line 42, in __get__
web_1     |         return obj.__dict__[self.__name__]
web_1     |     KeyError: 'backend'
web_1     |
web_1     |     During handling of the above exception, another exception occurred:
web_1     |
web_1     |     Traceback (most recent call last):
web_1     |       File "/usr/local/lib/python3.7/site-packages/tornado/web.py", line 1512, in _execute
web_1     |         result = yield result
web_1     |       File "/usr/local/lib/python3.7/site-packages/tornado/gen.py", line 1055, in run
web_1     |         value = future.result()
web_1     |       File "/usr/local/lib/python3.7/site-packages/tornado/concurrent.py", line 238, in result
web_1     |         raise_exc_info(self._exc_info)
web_1     |       File "<string>", line 4, in raise_exc_info
web_1     |       File "/usr/local/lib/python3.7/site-packages/tornado/gen.py", line 1063, in run
web_1     |         yielded = self.gen.throw(*exc_info)
web_1     |       File "/app/docbleach/controllers/upload.py", line 25, in post
web_1     |         async_res = yield self.add_task(link, filename)
web_1     |       File "/usr/local/lib/python3.7/site-packages/tornado/gen.py", line 1055, in run
web_1     |         value = future.result()
web_1     |       File "/usr/local/lib/python3.7/site-packages/tornado/concurrent.py", line 238, in result
web_1     |         raise_exc_info(self._exc_info)
web_1     |       File "<string>", line 4, in raise_exc_info
web_1     |       File "/usr/local/lib/python3.7/site-packages/tornado/gen.py", line 292, in wrapper
web_1     |         result = func(*args, **kwargs)
web_1     |       File "/usr/local/lib/python3.7/types.py", line 277, in wrapped
web_1     |         coro = func(*args, **kwargs)
web_1     |       File "/app/docbleach/controllers/upload.py", line 53, in add_task
web_1     |         kwargs={})
web_1     |       File "/usr/local/lib/python3.7/site-packages/celery/app/base.py", line 736, in send_task
web_1     |         self.backend.on_task_call(P, task_id)
web_1     |       File "/usr/local/lib/python3.7/site-packages/kombu/utils/objects.py", line 44, in __get__
web_1     |         value = obj.__dict__[self.__name__] = self.__get(obj)
web_1     |       File "/usr/local/lib/python3.7/site-packages/celery/app/base.py", line 1182, in backend
web_1     |         return self._get_backend()
web_1     |       File "/usr/local/lib/python3.7/site-packages/celery/app/base.py", line 900, in _get_backend
web_1     |         self.loader)
web_1     |       File "/usr/local/lib/python3.7/site-packages/celery/app/backends.py", line 65, in by_url
web_1     |         return by_name(backend, loader), url
web_1     |       File "/usr/local/lib/python3.7/site-packages/celery/app/backends.py", line 45, in by_name
web_1     |         cls = symbol_by_name(backend, aliases)
web_1     |       File "/usr/local/lib/python3.7/site-packages/kombu/utils/imports.py", line 57, in symbol_by_name
web_1     |         module = imp(module_name, package=package, **kwargs)
web_1     |       File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module
web_1     |         return _bootstrap._gcd_import(name[level:], package, level)
web_1     |       File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
web_1     |       File "<frozen importlib._bootstrap>", line 983, in _find_and_load
web_1     |       File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
web_1     |       File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
web_1     |       File "<frozen importlib._bootstrap_external>", line 724, in exec_module
web_1     |       File "<frozen importlib._bootstrap_external>", line 860, in get_code
web_1     |       File "<frozen importlib._bootstrap_external>", line 791, in source_to_code
web_1     |       File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
web_1     |       File "/usr/local/lib/python3.7/site-packages/celery/backends/redis.py", line 21
web_1     |         from . import async
web_1     |                           ^
web_1     |     SyntaxError: invalid syntax
web_1     | [E 200722 15:01:39 web:2064] 500 POST /v1/tasks (192.168.160.1) 170.28ms

@atanumaj
Copy link

I get below error -
root@ubuntu:/docker-containers/DocBleach-Web# docker-compose up -d
Building web
Sending build context to Docker daemon 1.854MB
[WARNING]: Empty continuation line found in:
RUN apk add -t build --no-cache --update-cache ca-certificates openssl curl && update-ca-certificates && wget -q -O /etc/apk/keys/sgerrand.rsa.pub https://alpine-pkgs.sgerrand.com/sgerrand.rsa.pub && wget https://github.com/sgerrand/alpine-pkg-glibc/releases/download/2.23-r3/glibc-2.23-r3.apk && apk add glibc-2.23-r3.apk && apk del build wget unzip && adduser -S -u 1000 worker && chmod o+x /usr/bin/plik && echo 'URL = "https://plik.root.gg"' > /home/worker/.plikrc
[WARNING]: Empty continuation lines will become errors in a future release.
Step 1/13 : FROM python:3.7-alpine
---> accf77502477
Step 2/13 : EXPOSE 5000
---> Using cache
---> 52879e55cfd8
Step 3/13 : HEALTHCHECK CMD curl --fail http://localhost:5000/ping
version: '2.1'

services:

Redis

redis:
image: redis

Our rest API

web:
build:
context: api
dockerfile: Dockerfile
image: docbleach/api:latest
environment:
- CELERY_BROKER=redis://redis/
- CELERY_RESULT_BACKEND=redis://redis/
- INTERNAL_PLIK_SERVER=${FINAL_PLIK_SERVER:-http://plik:8080}
ports:
- "${EXTERNAL_PORT:-9000}:5000"
tmpfs: /tmp

Celery worker, the one you should be scaling using, ie to have 10 instances:

docker-compose scale worker=10

worker:
build:
context: worker
dockerfile: Dockerfile
image: docbleach/worker:latest
environment:
- CELERY_BROKER=redis://redis/
- CELERY_RESULT_BACKEND=redis://redis/
- FINAL_PLIK_SERVER=${FINAL_PLIK_SERVER:-https://plik.root.gg}
tmpfs: /tmp

plik:
image: rootgg/plik
expose:
- 8080
~
~
~
~
"docker-compose.yml" [noeol] 38L, 876B 38,1 All
---> Using cache
---> ea2c7f795115
Step 4/13 : ENV INTERNAL_PLIK_SERVER https://plik.root.gg
---> Using cache
---> ab399ca49a78
Step 5/13 : ADD https://plik.root.gg/clients/linux-amd64/plik /usr/bin/plik
FROM openjdk:8-jre-alpine

ENTRYPOINT ["/app/entrypoint.sh"]

HEALTHCHECK CMD /usr/bin/celery inspect ping -A docbleach.tasks -d celery@$HOSTNAME

ENV FINAL_PLIK_SERVER https://plik.root.gg

Stored on Plik for now, as we don't yet have the GitHub Link

ENV DOCBLEACH_JAR https://github.com/docbleach/DocBleach/releases/download/v0.0.9/docbleach.jar

We add the Plik binary file

ADD https://plik.root.gg/clients/linux-amd64/plik /usr/bin/plik

RUN apk add --no-cache --update-cache
# Having up to date SSL certificates is always a good thing. :)
ca-certificates
openssl
python3
&&
update-ca-certificates && \

# Add glibc, wanted by plik (compiled with Go)
wget -q -O /etc/apk/keys/sgerrand.rsa.pub https://alpine-pkgs.sgerrand.com/sgerrand.rsa.pub && \
wget https://github.com/sgerrand/alpine-pkg-glibc/releases/download/2.23-r3/glibc-2.23-r3.apk && \
apk add glibc-2.23-r3.apk && \

# Setup Python 3
python3 -m ensurepip && \
rm -r /usr/lib/python*/ensurepip && \
pip3 install --upgrade pip setuptools && \
rm -r /root/.cache && \
# Remove openssl, only needed for wget
apk del openssl wget unzip && \

# We add an user
adduser -S -u 1000 worker && \

# Add default config for Plik, required
chmod o+x /usr/bin/plik && \
echo 'URL = "https://plik.root.gg"' > /home/worker/.plikrc

"Dockerfile" 52L, 1604B 1,1 Top
FROM openjdk:8-jre-alpine

ENTRYPOINT ["/app/entrypoint.sh"]

HEALTHCHECK CMD /usr/bin/celery inspect ping -A docbleach.tasks -d celery@$HOSTNAME

ENV FINAL_PLIK_SERVER https://plik.root.gg

Stored on Plik for now, as we don't yet have the GitHub Link

ENV DOCBLEACH_JAR https://github.com/docbleach/DocBleach/releases/download/v0.0.9/docbleach.jar
Downloading [==================================================>] 7.643MB/7.643MB

---> Using cache
---> ec80eb0aa85a
Step 6/13 : RUN apk add -t build --no-cache --update-cache ca-certificates openssl curl && update-ca-certificates && wget -q -O /etc/apk/keys/sgerrand.rsa.pub https://alpine-pkgs.sgerrand.com/sgerrand.rsa.pub && wget https://github.com/sgerrand/alpine-pkg-glibc/releases/download/2.23-r3/glibc-2.23-r3.apk && apk add glibc-2.23-r3.apk && apk del build wget unzip && adduser -S -u 1000 worker && chmod o+x /usr/bin/plik && echo 'URL = "https://plik.root.gg"' > /home/worker/.plikrc
---> Running in 59b6b2567c9a
fetch https://dl-cdn.alpinelinux.org/alpine/v3.16/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.16/community/x86_64/APKINDEX.tar.gz
(1/6) Installing openssl (1.1.1s-r0)
(2/6) Installing brotli-libs (1.0.9-r6)
(3/6) Installing nghttp2-libs (1.47.0-r0)
(4/6) Installing libcurl (7.83.1-r4)
(5/6) Installing curl (7.83.1-r4)
(6/6) Installing build (20221118.094241)
Executing busybox-1.35.0-r17.trigger
OK: 16 MiB in 42 packages
WARNING: ca-certificates.crt does not contain exactly one certificate or CRL: skipping
Connecting to github.com (20.207.73.82:443)
FROM python:3.7-alpine

EXPOSE 5000
HEALTHCHECK CMD curl --fail http://localhost:5000/ping

ENV INTERNAL_PLIK_SERVER https://plik.root.gg

We add the Plik binary file

ADD https://plik.root.gg/clients/linux-amd64/plik /usr/bin/plik

RUN apk add -t build --no-cache --update-cache
# Having up to date SSL certificates is always a good thing. :)
ca-certificates
openssl
# For the healthcheck
curl
&&
update-ca-certificates && \

# Add glibc, wanted by plik (compiled with Go)
wget -q -O /etc/apk/keys/sgerrand.rsa.pub https://alpine-pkgs.sgerrand.com/sgerrand.rsa.pub && \
wget https://github.com/sgerrand/alpine-pkg-glibc/releases/download/2.23-r3/glibc-2.23-r3.apk && \
apk add glibc-2.23-r3.apk && \

# Remove openssl, only needed for wget
apk del build wget unzip && \

# We add an user
adduser -S -u 1000 worker && \

# Add default config for Plik, required
chmod o+x /usr/bin/plik && \
echo 'URL = "https://plik.root.gg"' > /home/worker/.plikrc

Install the python dependencies

ADD requirements.txt /app/
WORKDIR /app/

RUN pip3 install --no-cache-dir -r requirements.txt

COPY . /app/
"Dockerfile" 46L, 1279B 1,1 Top
FROM python:3.7-alpine

EXPOSE 5000
HEALTHCHECK CMD curl --fail http://localhost:5000/ping
Connecting to objects.githubusercontent.com (185.199.111.133:443)
saving to 'glibc-2.23-r3.apk'
glibc-2.23-r3.apk 100% |********************************| 2874k 0:00:00 ETA
'glibc-2.23-r3.apk' saved
fetch https://dl-cdn.alpinelinux.org/alpine/v3.16/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.16/community/x86_64/APKINDEX.tar.gz
(1/1) Installing glibc (2.23-r3)
ERROR: glibc-2.23-r3: trying to overwrite etc/nsswitch.conf owned by alpine-baselayout-data-3.2.0-r23.
1 error; 20 MiB in 43 packages
The command '/bin/sh -c apk add -t build --no-cache --update-cache ca-certificates openssl curl && update-ca-certificates && wget -q -O /etc/apk/keys/sgerrand.rsa.pub https://alpine-pkgs.sgerrand.com/sgerrand.rsa.pub && wget https://github.com/sgerrand/alpine-pkg-glibc/releases/download/2.23-r3/glibc-2.23-r3.apk && apk add glibc-2.23-r3.apk && apk del build wget unzip && adduser -S -u 1000 worker && chmod o+x /usr/bin/plik && echo 'URL = "https://plik.root.gg"' > /home/worker/.plikrc' returned a non-zero code: 1
ERROR: Service 'web' failed to build : Build failed

# for free to join this conversation on GitHub. Already have an account? # to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@atanumaj @AppliNH