Opening a secure websocket to a server that is using SNI results in set_fail_handler: 8: TLS handshake failed

[moozzyk]

I am using websockets from cpprestsdk which internally uses websocketpp to handle websocket communication. I have two servers (Azure websites really) I am connecting to using secure websockets. Both of them are using the same certificate. I am able to connect to one of servers successfully while connecting to the other one results in a "TLS handshake failed" error. Note that I am able to connect successfully to both servers using javascript or C# secure websockets. The "TLS handshake failed" is quite general - how can I get more details about the cause of the failure and fix the issue?

[moozzyk]

Seems that the issue here is that the server is using SNI SSL and as a result the host name on the certificate does not match the name of the host from the request which causes TLS handshake failure. Open SSL apparently requires to specify servername to set "the TLS SNI (Server Name Indication) extension in the ClientHello message." Is there any way to set servername when using websocketpp?

(for the sake of completeness - the original error returned by OpenSSL was 336134278)

[zaphoyd]

Generally the way to get the detailed errors is to turn on higher levels of logging and look at the logs. It is on the todo-list to allow retrieving detailed errors from the transport & socket layers directly for helping in cases like this.

With respect to SNI: That is a good question. I'll take a look in more detail. In the meantime, as a place to start, in general you can set any TLS related settings you want directly via the same interface that Asio provides (which at worst, lets you directly call OpenSSL methods) via the tls_init_handler.

[moozzyk]

@zaphoyd - Re: SNI it appears that by default OpenSSL does not include the domain name in the TLS request but there is an option you could use to enforce it. This stackoverflow question indicates that calling SSL_set_tlsext_host_name(ssl, servername) before starting the SSL request should result in sending the host name in the TLS request making it possible for the server to return the correct certificate. Without the host name in the TLS if there are multiple certs/domains on the server the server does not know which one to returns and returns a default (?) one (for instance in our case because we are using Azure websites we are getting a certificate for *.azurewebsites.net and not the one that is bound to our domain)

[moozzyk]

A workaround to this issue provided by a colleague is along the lines:

client.set_socket_init_handler([host](websocketpp::connection_hdl, boost::asio::ssl::stream<boost::asio::ip::tcp::socket> &ssl_stream)
{
    SSL_set_tlsext_host_name(ssl_stream.native_handle(), host);
});

[zaphoyd]

This looks like a reasonable solution for now. This is something the library should be doing. I'll work on a fix.

[moozzyk]

Thanks for looking into it!

[zaphoyd]

An update: This has been fixed on the develop branch for a bit. The library will now set the appropriate SNI hostname automatically.

[moozzyk]

👍

[zaphoyd]

The 0.6.0 release included this fix.