Containers become unresponsive due to "too many connections"

[dancannon]

Expected behaviour

Applications remain accessible even when running many containers

Actual behaviour

After starting many services which create connections to the containers the containers become unresponsive and docker starts logging "too many connections" errors.

Information

We use docker for mac to run parts of our development environment (this includes various databases and message queues, processes outside of docker connect to these containers). Recently as we have created more services we noticed that if we run all of our services the docker containers become unresponsive, after some investigation we noticed that docker is reporting "too many connections" however we could not find anyway to increase the connection limit.

We noticed these errors after we start around 145 services while running 6 containers.

Diagnostic ID: FA8036A9-1DA0-4241-BA04-A87CB52C794B

Docker for Mac: version: 1.13.0-rc2-beta31 (de6f6c2)
OS X: version 10.11.5 (build: 15F34)
logs: /tmp/FA8036A9-1DA0-4241-BA04-A87CB52C794B/20161205-141057.tar.gz
[OK]     vmnetd
[OK]     dns
[OK]     driver.amd64-linux
[OK]     virtualization VT-X
[OK]     app
[OK]     moby
[OK]     system
[OK]     moby-syslog
[OK]     db
[OK]     env
[OK]     virtualization kern.hv_support
[OK]     slirp
[OK]     osxfs
[OK]     moby-console
[OK]     logs
[OK]     docker-cli
[OK]     menubar
[OK]     disk

Steps to reproduce the behaviour

1. Start a docker container which listens on a port
2. Expose the port to the host
3. Start many processes to the container
4. Eventually connections start to fail and viewing the docker logs using the diagnostics menu shows "too many connections"

[djs55]

By default Docker for Mac limits the number of file descriptors it opens to avoid hitting macOS system-wide limits. The current default maximum number of connections is set to 900 (which is perhaps a bit low). It can be updated like this:

$ cd ~/Library/Containers/com.docker.docker/Data/database/
$ git reset --hard
HEAD is now at 9410b78 last-start-time changed at 1480947038
$ cat com.docker.driver.amd64-linux/slirp/max-connections 
900
$ echo 1200 > com.docker.driver.amd64-linux/slirp/max-connections 
$ git add com.docker.driver.amd64-linux/slirp/max-connections 
$ git commit -s -m 'Update the maximum number of connections'
[master 227a248] Update the maximum number of connections
 1 file changed, 1 insertion(+), 1 deletion(-)

At this point you should see a notice in the logs like this:

$ syslog -k Sender Docker
<Notice>: updating connection limit to 1200

I recommend checking the values of these sysctls:

$ sysctl kern.maxfiles
kern.maxfiles: 12288
$ sysctl kern.maxfilesperproc
kern.maxfilesperproc: 10240

I believe this means I could bump my max-connections to 10240 for Docker in theory, but this would leave only 12288 - 10240 = 2048 descriptors left for everything else on the system. This might be ok, depending on the apps you need to run. I believe when you hit the global limit you start to see "Too many open files in system" errors

I guess that in your case, if you're running half of your system inside docker and half natively on the host you could give half of the descriptors to docker and keep the other half (plus a bit more, for email, web etc) for the native processes.

If you do play with the max-connections setting, let me know how it goes!

[dancannon]

Thanks @djs55, that appears to work!

[rogaha]

@dancannon thanks for the update. I'm closing this for now.

[vlad0337187]

@djs55 , I have such error:

username $: cat com.docker.driver.amd64-linux/slirp/max-connections
cat: com.docker.driver.amd64-linux/slirp/max-connections: No such file or directory

Mac: 10.13.1 (17B1003)
Docker: 17.09.0-ce-mac35 (19611)

[docker-robott]

Closed issues are locked after 30 days of inactivity.
This helps our team focus on active issues.

If you have found a problem that seems similar to this, please open a new issue.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows.
/lifecycle locked