Fix TCP accept handling when max incoming connection limit is reached #3315
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
We do not currently need a system/permanent error concept in server_socket and we may never need it. We can introduce it if we ever need it.
I don't see how connection aborted is a temporary error. Connection aborted is a permanent state.
We never want to stop listening for incming sockets
…500ms) At 500ms, it means that if connections arrive at a frequency higher than 2 per second, they will start backing up and we will start seeing CLOSE_WAIT socket poling up because the client will close the socket but the the node cannot close a socket it has not accepted yet. At 1ms the incoming connection request frequency would have to be higher than 1000 per second so that's probably OK. If we get 1000 connection requests per second per server then we will other problems.
This is to avoid busy loops in case something goes wrong and we get a persistent error.
Closed
clemahieu
approved these changes
Jun 2, 2021
clemahieu
pushed a commit
that referenced
this pull request
Jun 2, 2021
…#3315) This fixes the problem where we stop accepting TCP sockets permanently due to reaching the max incoming sockets limit. The symptom of this problem is a large number of CLOSE_WAIT TCP sockets. The approach of this solution is to never refuse to accept. We accept all connections but if the limit is reached, the connection is dropped immediately after being accepted.
# for free
to join this conversation on GitHub.
Already have an account?
# to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This fixes the problem where we stop accepting TCP sockets permanently due to reaching the max incoming sockets limit.
The symptom of this problem is a large number of CLOSE_WAIT TCP sockets.
The approach of this solution is to never refuse to accept.
We accept all connections but if the limit is reached, the connection is dropped immediately after being accepted.