Caddy service can't start

[ashkadov86]

I'm have ubuntu 20.04. After last upgrade system service caddy can't start with state Exit 2.
i tested on multiple hosts with ubuntu 20.04 with latest updates, same problem everywhere.
On ububntu 18.04 there are no such problems.
What could be the reason?

[AjdinDev]

I'm having the same issue.

[mihagrgicjelen]

+1

[charliebaer]

I am facing the same issue

[rcpehls]

log

goroutine 1 [running]:
github.com/mholt/caddy/vendor/github.com/miekg/dns.ClientConfigFromFile(0xbb4739, 0x10, 0x0, 0x0, 0x0)
src/github.com/mholt/caddy/vendor/github.com/miekg/dns/clientconfig.go:86 +0xad6
github.com/mholt/caddy/vendor/github.com/xenolf/lego/acme.getNameservers(0xbb4739, 0x10, 0xfeaf20, 0x2, 0x2, 0xf77460, 0xc4200402c0, 0xc420037f50)
src/github.com/mholt/caddy/vendor/github.com/xenolf/lego/acme/dns_challenge.go:40 +0x4d
github.com/mholt/caddy/vendor/github.com/xenolf/lego/acme.init()
src/github.com/mholt/caddy/vendor/github.com/xenolf/lego/acme/dns_challenge.go:33 +0x12d
github.com/mholt/caddy/caddy/caddymain.init()
:1 +0x75
main.init()
:1 +0x44
panic: runtime error: slice bounds out of range

[rcpehls]

I changed the network dockprom_monitor-net ownership from administrator to public and re-deploy caddy container

works for me

[nightah]

Can users with the issue please report the output of:

• docker version
• docker info

With the information you have provided I would suspect it's one of the packages you upgraded that's causing the issue.

[Slavenin]

docker version

Client: Docker Engine - Community
 Version:           20.10.3
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        48d30b5
 Built:             Fri Jan 29 14:33:21 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.3
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       46229ca
  Built:            Fri Jan 29 14:31:32 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.3
  GitCommit:        269548fa27e0089a8b8278fc4fc781d7f65a939b
 runc:
  Version:          1.0.0-rc92
  GitCommit:        ff819c7e9184c13b7c2607fe6c30ae19403a7aff
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)

Server:
 Containers: 7
  Running: 0
  Paused: 0
  Stopped: 7
 Images: 13
 Server Version: 20.10.3
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 269548fa27e0089a8b8278fc4fc781d7f65a939b
 runc version: ff819c7e9184c13b7c2607fe6c30ae19403a7aff
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 5.4.0-26-generic
 Operating System: Ubuntu 20.04.2 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 3.843GiB
 Name: prod-big-break-monitoring
 ID: 26GE:SEC5:MACD:SID6:DUF3:C5SD:3RNH:OZND:EDNA:XAMI:7DK6:NF5P
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No swap limit support
WARNING: No blkio weight support
WARNING: No blkio weight_device support

[Slavenin]

On centos work fine

[vasilyevmn]

Same.
Fresh Ubuntu 20.04 LTS

This is some kind of global problem:
https://stackoverflow.com/questions/66074495/i-am-trying-to-run-caddy-server-with-docker-on-ubuntu-20-04-2-lts-but-caddy-not
zuohuadong/docker-caddy#5

[ashkadov86]

I'm have ubuntu 20.04. After last upgrade system service caddy can't start with state Exit 2.
i tested on multiple hosts with ubuntu 20.04 with latest updates, same problem everywhere.
On ububntu 18.04 there are no such problems.
What could be the reason?

My solution was to use the original caddy image

[DigiDr]

OK, i've figured this out @nightah

The problem is that as of ubuntu 20.04 'trust-ad' is added to the end of resolv.conf. Could others confirm that /etc/resolv.conf ends with?

options edns0 trust-ad

Mine is:

nameserver 127.0.0.53
options edns0 trust-ad

When docker runs it copies the resolv.conf from the host. However, the trust-ad throws an error in caddy because of this bug in the main go dns module they import. This was fixed here:

miekg/dns@906238e

But caddy uses a version prior:

https://github.com/caddyserver/caddy/search?q=github.com%2Fmiekg%2Fdns

A quick fix it to somehow overwrite the resolv.conf that docker uses. Docker makes this slightly complicated, but the easiest way i found is to create a copy of /etc/resolv.conf somewhere, and edit it to remove the last trust-ad and change the nameserver ip to 127.0.0.11 (docker dns server default) - see below. Then change your docker-compose.yml and import it as a volume with read-only permissions where resolv.conf normally lives. You may also have to add ndots:0 - something docker appends after it copies the file for some complicated dns reason.

My edited resolv.conf looks like:

nameserver 127.0.0.11
options edns0 ndots:0

docker-compose.yml under services: caddy edit volumes to:

    volumes:
      - ./caddy:/etc/caddy
      - /path/to/local/resolv.conf:/etc/resolv.conf:ro

Result: container starts as expected.

An alternative fix would be to manually edit your resolv.conf in place on the system, so the copy that is brought across works. I haven't tried to do that because of the way ubuntu manages this file that i don't completely understand.

This is fixed in version 2 of caddy

[nightah]

Thanks for the write up @DigiDr, this issue has been resolved with v3.30.0.