Connection issue via Websocket to a node behind Nginx proxy

[niczy]

Description

When I try to connect my libp2p node to another libp2p node behind nginx proxy via websocket, I got following error:

Custom { kind: Other, error: Transport(Handshake("server rejected handshake; status code = 404"))

Seems the issue is that nginx depends on information of HTTP headers like host, path do decide which backed to route the traffic to. Current websocket transport impl doesn't send those information

See https://github.com/libp2p/rust-libp2p/pull/4180/files for the DNS/WS transport configuration.

Motivation

Allow us to run multiple libp2p node behind a nginx proxy over Websocket.

Current Implementation

Currently we won't be able to connect to libp2p node behind a nginx proxy with virtual host.

Are you planning to do it yourself in a pull request?

Maybe. Tried this, need to research more what we should send after a socket connection is established.
https://github.com/libp2p/rust-libp2p/pull/4180/files

[mxinden]

Seems the issue is that nginx depends on information of HTTP headers like host, path do decide which backed to route the traffic to. Current websocket transport impl doesn't send those information

Correct. The current websocket implementation does not support custom paths or HTTP headers. Can you load balance / multiplex based on different URLs, i.e. subdomains in your use-case?

[niczy]

Correct. The current websocket implementation does not support custom paths or HTTP headers. Can you load balance / multiplex based on different URLs, i.e. subdomains in your use-case?

That could work. But not very scalable for us if we want to run some IPFS node in production. Normally in a mid-size to large-size company, the LB is maintained by a dedicated team and LB relies on the virtual host to dispatch request to different services.

What we did is to have nginx to route traffic from example.com to a gateway service. The gateway service then dispatches the request further to the corresponding IPFS node based on the path "/p2p/XXXXX". When we start a new IPFS node, the node would register its peer id to the gateway service. This allows us to scale up/down the IPFS cluster automatically. Scaling would be more difficult with subdomains, because we would need to programmatically update DNS.

This approach works with js-libp2p since the connection was established with "wss://example.com/p2p/XXXXX".

[mxinden]

This approach works with js-libp2p since the connection was established with "wss://example.com/p2p/XXXXX".

Indeed rust-libp2p's websocket implementation falls short here. See also #2708. I am happy to accept patches to libp2p-websocket in case you are interested in contributing @niczy.

[niczy]

yeah, I can try

[niczy]

Seems we just need to pass the original "host_name" and "resource" param to the sokettio Client here:

rust-libp2p/transports/websocket/src/framed.rs

Line 351 in 52cf26f

let mut client = handshake::Client::new(stream, &addr.host_port, addr.path.as_ref());

Passing the correct "resource" param should fix the other issue #2708.

However websocket transport is wrapped within the dns transport which removes the information about the hostname.

I think we would need to refactor the code to stop treating DnsConfig as a Transport trait and use it as a helper function. Websocket crate can call dns library to resolve the IP address and pass the correct hostname and path to sokettio Client later.

[thomaseizinger]

I think we would need to refactor the code to stop treating DnsConfig as a Transport trait and use it as a helper function. Websocket crate can call dns library to resolve the IP address and pass the correct hostname and path to sokettio Client later.

I don't think this will be an option, DNS resolution should be composable with any transport.

[niczy]

Any suggestion how we can solve this? ( I am both very new to rust and rust-libp2p codebase :-) ).
Today websocket transport is wrapped inside DNSConfig, and doesn't have any information about the DNS name since the multiaddr is re-written by DNSConfig.

Other implementations like golang has this function to parse the original multiaddr to a URL that can be passed into net.Dial
https://github.com/libp2p/go-libp2p/blob/master/p2p/transport/websocket/addrs.go#L107

[thomaseizinger]

Any suggestion how we can solve this?

Unfortunately, the interface is constrained to just passing a multiaddr.

@mxinden Do you know if we have the same problem in other transports? It seems worthwhile solving that each inner transport of DNS would want to know that we originally received a domain name as an argument.

Can we (ab)-use the /sni multiaddr protocol here? The DNS transport could append that to a resolved address to indicate to the inner transport which original domain name we received.

[mxinden]

Today websocket transport is wrapped inside DNSConfig, and doesn't have any information about the DNS name since the multiaddr is re-written by DNSConfig.

It should be the other way around. I.e. websocket should wrap DNSConfig. See below as an example @niczy:

https://github.com/mxinden/rust-libp2p-server/blob/6b4c36a322127b0e5e4be8a09c8771b3cb92432b/src/main.rs#L109-L118

[github-actions]

Oops, seems like we needed more information for this issue, please comment with more details or this issue will be closed in 7 days.

[github-actions]

This issue was closed because it is missing author input.