GRPC connecting over TOR #992
Labels
Comments
|
Can you be more specific about how your attempts to use gRPC with Tor have failed so far? |
|
Hi @murgatroid99 . I've been running some tests on this: I have a grpc service running on port If I try connecting over the I tried switching out |
|
If the addresses can't be resolved by our existing name resolution then it's not going to work with the grpc libraries as they currently exist. We don't have an API for resolving other kinds of names |
|
According to https://2019.www.torproject.org/docs/tor-manual.html.en Tor now supports
I have successfully been able to use use gRPC over Tor by setting up a local Tor server that serves as an HTTP proxy using the This works with the native However, it does not work with the |
|
The latest version of grpc-js now supports using HTTP CONNECT configured by the |
|
Thats good progress @murgatroid99 . However, with |
|
Your previous trace log shows that the native |
|
The earlier example was when trying to connect directly to a .onion address which doesn't work. However it can be made to work by connecting via a local tor http tunnel. Here is a full trace of a successful connection to .onion address via local tor tunnel using grpc native https://gist.github.com/mrfelton/89fa4e9c0cff167ff949cba4c84081a5 |
|
FYI, you can reproduce this with LN-Zap/node-lnd-grpc#166 You can run this single test case to see it run. If you then swap out I should also point out though that my test cases will fail when using grpc-js regarless due to #1354 - however I patched my local copy of grpc-js to fix the http_proxy env var support but it still fails at the name resolution phase. |
|
In any case, |
|
Does it really need to do a dns lookup on the destination address when using a proxy? |
|
You're right. I published some new proxy handling code in |
|
I'm getting a different result with 0.8.1 - definitely looks like an improvement, though it still appears to have an issue. See logs here https://gist.github.com/mrfelton/50a517b223af0e4846e87aa24b8ba6b3 The logs indicate that the proxy was indeed used (so #1354 no longer seems to be an issue), and that a connection was established. However, the actual grpc call is hanging indefinitely at the last line. I'm expecting to receive an |
|
Can you get logs from the proxy server, to verify that data is getting sent through the HTTP CONNECT tunnel? That client definitely thinks it's sending the request to the server. |
|
I will try to get that. I can say though that the proxy server correctly forwards data to and from the destination if I switch out grpc-js for grpc |
|
I don't expect that the proxy server is misbehaving, I just hope that this can give us more information about what grpc-js is doing differently. |
|
Here are the full combined debug logs (lnd-grpc + grpc-js + tor) |
|
For comparison, here is the same thing but with native grpc in which the calls succeed.. |
|
OK, that's too many log lines. That file has 1500 lines of TOR logs between when gRPC makes the HTTP CONNECT request and when it gets the response, and another 1700 between when gRPC sends the gRPC request and the next time it logs anything. I can't tell what's relevant. |
|
I know. Unfortunately tor only seems to provide 2 log levels, |
|
lol, ok - here are logs at grpc-js: grpc: |
|
Those are definitely easier to read, but unfortunately that seems to swing too far the other way. On the native side, it looks like there are no TOR logs between when it sends the gRPC request and when it receives the response, so I can't tell what if anything is happening differently with grpc-js there. I dug into the debug logs again, and it looks to me like the TOR part is seeing grpc-js send the request, and it is trying to send the response back, but grpc-js never gets it for some reason. Unfortunately, grpc-js doesn't have the same level of introspection into low-level network events that the native library has, so I can't tell what exactly isn't happening correctly. |
|
OK, I have another idea: can you use |
|
Here are 2 dumps from wireshark for grpc vs grpc-js I'm not very familiar with this stuff so this doesn't mean a whole lot to me but it does appear to me that grpc-js never gets as far as doing the tls key exchange. I can see them both do grpc-native grpc-js |
|
In the grpc-js case, it looks like the server responded very differently to the TLS |
|
Here is another version of the test run that's a little cleaner. I noticed that the test I was running was actually instantiating 2 connections to two different grpc subservers on the destination server. This version just connects and make a single call. I think the result looks the same as before, but just with a little less noise: |
|
I guess this is what you are referring to on the grpc-js version, when it gets to this part: 173 17.509120 127.0.0.1 127.0.0.1 TLSv1.3 1217 Server Hello, Change Cipher Spec, Application Data, Application Data, Application Data, Application Data, Application Data
vs on grpc: 165 16.769757 127.0.0.1 127.0.0.1 TLSv1.2 912 Server Hello, Certificate, Server Key Exchange, Server Hello Done
|
|
Any ideas? Maybe the encoding on the cert is getting messed up? |
|
I found another thing to check out: now that we know that there's a TLS issue, we might get something useful with the |
|
grpc doesn't give any additional info when run with that flag but here is what I get from grpc-js: Is SSL being terminated at the proxy? Something extra that needs too be set for it to pass through to the destination? One thing that I notice is that the additional tls cyper suites that I enabled do not seem to be included In my code I'm doing this: const grpcSslCipherSuites = [
// Default is ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384
// https://github.com/grpc/grpc/blob/master/doc/environment_variables.md
//
// Current LND cipher suites here:
// https://github.com/lightningnetwork/lnd/blob/master/lnd.go#L80
//
// We order the suites by priority, based on the recommendations provided by SSL Labs here:
// https://github.com/ssllabs/research/wiki/SSL-and-TLS-Deployment-Best-Practices#23-use-secure-cipher-suites
'ECDHE-ECDSA-AES128-GCM-SHA256',
'ECDHE-ECDSA-AES256-GCM-SHA384',
'ECDHE-ECDSA-AES128-CBC-SHA256',
'ECDHE-ECDSA-CHACHA20-POLY1305',
// BTCPay Server serves lnd behind an nginx proxy with a trusted SSL cert from Lets Encrypt.
// These certs use an RSA TLS cipher suite.
'ECDHE-RSA-AES256-GCM-SHA384',
'ECDHE-RSA-AES128-GCM-SHA256',
].join(':')
// Set up SSL with the cypher suits that we need.
if (!process.env.GRPC_SSL_CIPHER_SUITES) {
process.env.GRPC_SSL_CIPHER_SUITES = grpcSslCipherSuites
}Perhaps grpc-js is not including these when running via a proxy? |
|
For comparison, here is the debug info from tls handshake when connecting to the destination directly over clearnet (no proxy, no tor) |
|
Perhaps something like this is needed: With this, the tls handshake looks a lot more promising. It looks like it does establish a proper connection when I'm doing |
|
BINGO! I've just updated #1369 with an additional commit which actually gets this working correctly! I've probably broken some other stuff in the process so you should take a close look @murgatroid99 but essentially I have patched |
|
Here are the logs from a successful run with #1369 in place. However still seems to be an issue with the cert because this only works if I also set |
|
Right, after a little more hackery I now have a version that works including properly validating the certs. See updated #1369. Again, the way I have done it in that PR is probably wrong and certainly needs some cleanup and more knowledgable eyes on it, but in summary what I believe needs to happen is:
|
# for free
to join this conversation on GitHub.
Already have an account?
# to comment
Problem description
Hi. I am trying to connect to a server over Tor via grpc and haven't had any success. Is this because Tor has
http/2disabled for now? Has anyone had any success with grpc + Tor? Any pointers would be super helpful.Reproduction steps
Attempt to connect to a server over Tor
The text was updated successfully, but these errors were encountered: