TCP errors and flakiness when using many simultaneous connections

[avaskys]

When connecting to a service in a docker-for-mac container using many parallel TCP connections, I've been experiencing lots of unexpected TCP connection issues. The issue is quite easy to reproduce (see steps below).

Information

Diagnostic ID: 7277E55A-2918-4FD1-B691-EEB0D44E3E2F

Docker for Mac: version: 1.12.2-beta28 (71c4a00)
OS X: version 10.11.6 (build: 15G1004)
logs: /tmp/7277E55A-2918-4FD1-B691-EEB0D44E3E2F/20161017-153142.tar.gz
[OK]     docker-cli
[OK]     virtualization kern.hv_support
[OK]     menubar
[OK]     moby-syslog
[OK]     dns
[OK]     disk
[OK]     system
[OK]     app
[OK]     osxfs
[OK]     virtualization VT-X
[OK]     db
[OK]     slirp
[OK]     logs
[OK]     env
[OK]     vmnetd
[OK]     moby-console
[OK]     moby
[OK]     driver.amd64-linux

Steps to reproduce the behavior

Summary: start an apache container serving up a file. Run multiple curl instances in parallel. Note all of the networking errors.

1. dd if=/dev/zero of=zero.txt bs=1000 count=1
2. docker run -dit -p 8888:80 --name apache -v "$PWD":/usr/local/apache2/htdocs/ httpd:2.4
3. for i in {1..100}; do curl -s -S -0 --no-keepalive "http://127.0.0.1:8888/zero.txt?[1-100]" > /dev/null & done; wait
4. Note all of the (possibly-interleaved) errors printed out by curl.

Expected behavior

No TCP errors. This is what happens when using docker-machine or Docker on Linux.

Actual behavior

curl prints out lots of errors:

curl: (55) Send failure: Broken pipe
curl: (56) Recv failure: Connection reset by peer
curl: (56) Recv failure: Connection reset by peer
curl: (56) Recv failure: Connection reset by peer
curl: (56) Recv failure: Connection reset by peer
curl: (56) Recv failure: Connection reset by peer
curl: (56) Recv failure: Connection reset by peer
curl: (56) Recv failure: Connection reset by peer
curl: (56) Recv failure: Connection reset by peer
curl: (56) Recv failure: Connection reset by peer
curl: (56) Recv failure: Connection reset by peer
curl: (56) Recv failure: Connection reset by peer
curl: (56) Recv failure: Connection reset by peer
curl: (56) Recv failure: Connection reset by peer
curl: (56) Recv failure: Ccounrnle:c t(i5o6n)  rReescevt  fbayi lpuereer:
 Connection reset by peer
curl: (56) Recv failure: Connection reset by peer
curl: (56) Recv failure: Connection reset by peer
curl: (56) Recv failure: Connection reset by peer

[ijc]

@avaskys thanks for your report and sorry for the slow response. This looks to be the same issue as #883 which was fixed in stable in 1.12.3 and in one of the betas after beta28, certainly it should be fixed in beta30. If you can still reproduce on latest stable of beta then please reopen.

[avaskys]

Unfortunately I'm still able to reliably reproduce this on 1.12.3. Let me know if you need any additional information to help pinpoint the issue.

[ijc]

@avaskys thanks for the update. Please could you upload a new diagnostic with the newer version.

[avaskys]

New diagnostic uploaded.

Docker for Mac: version: 1.12.3 (583d1b8)
OS X: version 10.11.6 (build: 15G1108)
logs: /tmp/244EB2D7-09B0-4132-B235-7472A900F90A/20161116-111055.tar.gz
[OK]     vmnetd
[OK]     dns
[OK]     driver.amd64-linux
[OK]     virtualization VT-X
[OK]     app
[OK]     moby
[OK]     system
[OK]     moby-syslog
[OK]     db
[OK]     env
[OK]     virtualization kern.hv_support
[OK]     slirp
[OK]     osxfs
[OK]     moby-console
[OK]     logs
[OK]     docker-cli
[OK]     menubar
[OK]     disk

[ijc]

Thanks for the updated diagnostic. It contains:

Nov 15 23:36:10 andys-MacBook-Pro Docker[com.docker.slirp][88722] <Error>: Lwt.async failure (Failure nth): Raised at file "pervasives.ml", line 32, characters 22-33
    Called from file "lib/dns_forward.ml", line 38, characters 41-77
    Called from file "lib/dns_forward.ml", line 102, characters 16-57
    Called from file "src/core/lwt.ml", line 795, characters 20-24

Which is moby/vpnkit#131 and #578. The upstream issue has been closed so please subscribe to #578 for updates when it reaches a Docker for Mac beta.

[rreinurm]

This issue seems to be still present and can be reproduced with instructions provided in the first comment.
Diagnostic ID: 12CB4B63-7AE2-4F86-B681-92F8B6B77436

macOS: version 10.12.3 (build: 16D32)
logs: /tmp/12CB4B63-7AE2-4F86-B681-92F8B6B77436/20170309-215625.tar.gz
[OK]     vmnetd
[OK]     dns
[OK]     driver.amd64-linux
[OK]     virtualization VT-X
[OK]     app
[OK]     moby
[OK]     system
[OK]     moby-syslog
[OK]     db
[OK]     env
[OK]     virtualization kern.hv_support
[OK]     slirp
[OK]     osxfs
[OK]     moby-console
[OK]     logs
[OK]     docker-cli
[OK]     menubar
[OK]     disk

[MagnusS]

@rreinurm it looks like you hit the connection limit which is 900 by default. Your logs contain:

2017-03-08 14:21:20+02 Renno-Reinurm-MB-Pro Docker[com.docker.slirp][16222] <Error>: Socket.Stream: caught Hostnet.Host_uwt.Sockets.Too_many_connections
2017-03-08 14:21:51+02 Renno-Reinurm-MB-Pro Docker[com.docker.slirp][16222] <Error>: exceeded maximum number of forwarded connections (900)
2017-03-08 14:21:51+02 Renno-Reinurm-MB-Pro Docker[com.docker.slirp][16222] <Error>: PPP.listen callback caught Hostnet.Host_uwt.Sockets.Too_many_connections

You can increase the limit by following the instructions in #1009 (comment)

[rreinurm]

@MagnusS thanx for the quick response
I tried to increase the value and also increase kernel parameters, and restarted Docker4Mac but didn't see any change. I uploaded new diagnostics file with same diagnostic id.

[bigicoin]

I am also still experiencing this issue on the latest community edition for mac osx.
Version: 17.03.1-ce

[OccultWarlock]

I'm having this same issue on the latest Edge version - has none come up with a fix?

[bigicoin]

Issue is closed so I am guessing nobody is looking into it. I stopped using docker for my redis and just installed redis natively and haven't had a problem since.

[sheerun]

I experience this issue as well..

[horan-geeker]

I use redis docker also meet this problem when many concurrent get/set operate

[docker-robott]

Closed issues are locked after 30 days of inactivity.
This helps our team focus on active issues.

If you have found a problem that seems similar to this, please open a new issue.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows.
/lifecycle locked