Possible SYN flooding

[derwassi]

Hi,
I had an websocketpp server running for about a year now (only occasionally on a local linux machine for integration test of another system, so only one concurrent connection), but suddenly it stopped working: Clients cannot connect anymore (they timeout).

in dmesg i get the following error:
request_sock_TCP: Possible SYN flooding on port 9002. Dropping request. Check SNMP counters.
But there's only one connection, so it definetely cannot be "real" SYN flooding.

I tried it also with the echo_server and echo_client example => same problem.

I tried fiddling around with the "syn" options in /etc/sysctl.conf (enabling, disabling, changing memory), => no effect.

I also used debug_asio_no_tls => does not give any output on connection attempts.

However, it can't be my system all alone, when trying to set up a node js based websocket server, it works flawlessly.

Any ideas what might be the problem (as it is not a productive environment, workarounds are also highly appreciated)?

[na1pir]

Hi, I accounted same issues while upgrading Linux kernel in embedded system from 4.0 to 4.4. Still looking for solution or any possible workaround.

[zwelab]

Hi, I have same problem after upgrading kernel from 4.0 to 4.4. Is there any workaround for that?
Did may be someone try to replace boost asio with standart libraries?

[derwassi]

I switched over to https://github.com/eidheim/Simple-WebSocket-Server, which is also boost-based => apparently not a boost related issue...

[zaphoyd]

what is the best way to reproduce this?

[derwassi]

hard to say... As zweleb mentioned, it happened after a kernel upgrade.
Perhaps create a linux vm with a recent 4.5 kernel and try to run the example programs.
here's my uname -a: Linux wassi 4.5.0-sabayon #1 SMP Tue Mar 29 11:22:56 UTC 2016 x86_64 Intel(R) Core(TM) i7-2620M CPU @ 2.70GHz GenuineIntel GNU/Linux

[zaphoyd]

okay. is it something that happens consistently or will I need to throw a diverse/fuzzed workload at it?

[derwassi]

In my case, it happened consistently, on every connection.

[zwelab]

I have Linux 4.4.0-xilinx #1 SMP PREEMPT Wed May 18 15:46:50 CEST 2016 armv7l GNU/Linux
And this piece of code
m_endpoint.get_alog().write(websocketpp::log::alevel::app,"run");
std::stringstream ss;
ss << "Running telemetry server on port "<< port <<" using docroot=" << docroot;
m_endpoint.get_alog().write(websocketpp::log::alevel::app,ss.str());
m_docroot = docroot;
m_endpoint.set_reuse_addr(true);
// listen on specified port
m_endpoint.listen(boost::asio::ip::tcp::v4(), port);
m_endpoint.start_accept();

and each time I start this I see in dmesg

[ 46.573989] TCP: request_sock_TCP: Possible SYN flooding on port 9002. Dropping request. Check SNMP counters.
[ 48.204501] TCP: request_sock_TCP: Possible SYN flooding on port 9002. Dropping request. Check SNMP counters.

Btw:
typedef websocketpp::server<websocketpp::config::asio> server;
server m_endpoint;

[zaphoyd]

Hi all, thanks to some hardware provided by @na1pir I was able to figure out what the likely issue was here. Specifically, WebSocket++ previously used a value of 0 as the default listen queue backlog depth. The behavior of this value on most systems was to use the OS default queue depth. Certain OSes post Linux kernel 4.4 appear to treat this value differently, rejecting all connections instead.

I've updated the library (develop branch) to use the Asio default of socket_base::max_connections instead. I've confirmed that this fixes the issue, at least on @na1pir's hardware that definitely displays the issue without it.

For those using previous versions of the library a workaround is to use the endpoint::set_listen_backlog method to change the value to something other than 0. *::asio::socket_base::max_connections is probably a good choice.

If anyone else can try this and confirm the fix that would be helpful.

[na1pir]

Thanks @zaphoyd. I would tag this issue as resolved as soon as someone test this patch on x86 and/or x86_64 and confirms this it is universal for more then one architecture.

[a-krebs]

Experiencing the same symptoms on Ubuntu 16.04.1 LTS, kernel 4.4.0-36, x86_64.

Client connections time out, and kern.log shows SYN flooding warnings like this:

TCP: request_sock_TCP: Possible SYN flooding on port 42240. Dropping request.  Check SNMP counters.

@zaphoyd's changes from 0bb33e4 resolve the issue for me 👍

[mikken]

So I had this problem with Librevault and websocketpp 0.7.0.
Once updated to develop branch, the issue is gone.
Testing successful, please merge :)

[kwonoh]

I have experienced same issue on a machine (Ubuntu 16.04, Kernel 4.4.0-38-generic, ufw enabled) using websocketpp 0.7.0.
/var/log/syslog shows

TCP: request_sock_TCP: Possible SYN flooding on port 9002. Dropping request.  Check SNMP counters.

Even if I disabled ufw, the client still cannot connect to the server.
While there is no clear log like above in /var/log/syslog, I get same output from echo_client example.

[2016-09-26 23:13:53] [info] Error getting remote endpoint: system:107 (Transport endpoint is not connected)
[2016-09-26 23:13:53] [fail] WebSocket Connection Unknown - "" / 0 websocketpp.transport:9 Timer Expired
[2016-09-26 23:13:53] [error] handle_connect error: Timer Expired

The issue has been resolved when I use develop branch or server.set_listen_backlog(boost::asio::socket_base::max_connections); on 0.7.0.

However, there was no issue on other similar machine (Ubuntu 16.04, Kernel 4.4.0-38-generic) which never used ufw before.

[zaphoyd]

Sounds like this is fixed, will include in 0.8.0

[kohenkatz]

@kwonoh As far as I can tell, the reason that ufw breaks it is that ufw turns off "SYN Cookies" (which are on by default in stock Ubuntu). For some reason, setting the backlog to 0 works when /proc/sys/net/ipv4/tcp_syncookies is set to 1 but not when it is set to 0.

If you are running someone else's prebuilt binary (my use-case) and can't change the code to make a call to set_listen_backlog (or you can't change the code for any other reason), turning tcp_syncookies back on is a workaround until you can update to 0.8.0 (when it is released).