Production troubleshooting

Slow indexing

Check date last_update in table guidebook.es_sync_status.

Restart indexation

ssh compose0.exoscale.infra.camptocamp.org
source /opt/c2corg-docker/server-list.env
cd /opt/c2corg-docker/composition/production/workers/

# check what's going on
docker-compose logs -t syncer

# restart syncer
docker-compose restart syncer

Run a docker command

api instances are not located on composition machine. On each api folder (api1, api2 and api3), a .env file sets DOCKER_HOST to the real docker machine. But docker command does not read this file. So to run a command, use this :

source .env
docker --host=$DOCKER_HOST ...

No space left on device when pulling new image for releases

We do not clean old images on docker hosts (yes, we should...). So, when you pull new images, if you see the message no space left on device, do some clean up:

source .env
docker --host=$DOCKER_HOST images | grep api
docker --host=$DOCKER_HOST rmi <some_old_image_id>

Certificate issues

A cron script runs avery ??? months to renew SSL certificates. Sometimes it fails. The symptom are obvious on browsers, as users are prevented to access to our site. On your terminal:

# Get ips of our production servers, they are in the ANSWER SECTION
dig camptocamp.org

# check each of these ip with HTTPS
curl https://....

# if there is an issue, one (or both) curl request will fail.

To fix this, connect to production server, and execute:

/opt/c2corg-docker/composition/production/workers/update-cert.sh