[Self-Signed] Ability to ignore cert checks for upstream servers

[sam-kleiner]

Is your feature request related to a problem? Please describe.
I am unable to use proxy.py when the upstream is using a self signed cert when using mitm mode. I have no issues using this with properly signed certs.

Describe the solution you'd like
An option to ignore upstream certificates when using mitm mode.

--ignore-upstream-ssl

Describe alternatives you've considered
An option to specify certs to verify against when calling the upstream on a per domain basis

--upstream-verify localhost=/path/to/cert --upstream-verify nginx.local=/path/to/cert2

Additional context
The following error occurs when trying to mitm an upstream with an unsigned cert.

proxy-http-mitm            | 2021-04-08 14:07:18,681 - pid:9 [E] run:359 - ssl.SSLError
proxy-http-mitm            | Traceback (most recent call last):
proxy-http-mitm            |   File "/usr/local/lib/python3.8/site-packages/proxy/http/handler.py", line 353, in run
proxy-http-mitm            |     teardown = self.run_once()
proxy-http-mitm            |   File "/usr/local/lib/python3.8/site-packages/proxy/http/handler.py", line 338, in run_once
proxy-http-mitm            |     teardown = self.handle_events(readables, writables)
proxy-http-mitm            |   File "/usr/local/lib/python3.8/site-packages/proxy/http/handler.py", line 152, in handle_events
proxy-http-mitm            |     teardown = self.handle_readables(readables)
proxy-http-mitm            |   File "/usr/local/lib/python3.8/site-packages/proxy/http/handler.py", line 298, in handle_readables
proxy-http-mitm            |     upgraded_sock = plugin.on_request_complete()
proxy-http-mitm            |   File "/usr/local/lib/python3.8/site-packages/proxy/http/proxy/server.py", line 352, in on_request_complete
proxy-http-mitm            |     return self.intercept()
proxy-http-mitm            |   File "/usr/local/lib/python3.8/site-packages/proxy/http/proxy/server.py", line 524, in intercept
proxy-http-mitm            |     self.wrap_server()
proxy-http-mitm            |   File "/usr/local/lib/python3.8/site-packages/proxy/http/proxy/server.py", line 551, in wrap_server
proxy-http-mitm            |     self.server.wrap(text_(self.request.host), self.flags.ca_file)
proxy-http-mitm            |   File "/usr/local/lib/python3.8/site-packages/proxy/core/connection/server.py", line 44, in wrap
proxy-http-mitm            |     self._conn = ctx.wrap_socket(
proxy-http-mitm            |   File "/usr/local/lib/python3.8/ssl.py", line 500, in wrap_socket
proxy-http-mitm            |     return self.sslsocket_class._create(
proxy-http-mitm            |   File "/usr/local/lib/python3.8/ssl.py", line 1040, in _create
proxy-http-mitm            |     self.do_handshake()
proxy-http-mitm            |   File "/usr/local/lib/python3.8/ssl.py", line 1309, in do_handshake
proxy-http-mitm            |     self._sslobj.do_handshake()
proxy-http-mitm            | ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate (_ssl.c:1123)
proxy-http-mitm            | Exception in thread Thread-5:
proxy-http-mitm            | Traceback (most recent call last):
proxy-http-mitm            |   File "/usr/local/lib/python3.8/threading.py", line 932, in _bootstrap_inner
proxy-http-mitm            |     self.run()
proxy-http-mitm            |   File "/usr/local/lib/python3.8/threading.py", line 870, in run
proxy-http-mitm            |     self._target(*self._args, **self._kwargs)
proxy-http-mitm            |   File "/usr/local/lib/python3.8/site-packages/proxy/http/handler.py", line 365, in run
proxy-http-mitm            |     self.shutdown()
proxy-http-mitm            |   File "/usr/local/lib/python3.8/site-packages/proxy/http/handler.py", line 167, in shutdown
proxy-http-mitm            |     self.flush()
proxy-http-mitm            |   File "/usr/local/lib/python3.8/site-packages/proxy/http/handler.py", line 210, in flush
proxy-http-mitm            |     self.selector.register(
proxy-http-mitm            |   File "/usr/local/lib/python3.8/selectors.py", line 352, in register
proxy-http-mitm            |     key = super().register(fileobj, events, data)
proxy-http-mitm            |   File "/usr/local/lib/python3.8/selectors.py", line 241, in register
proxy-http-mitm            |     raise KeyError("{!r} (FD {}) is already registered"
proxy-http-mitm            | KeyError: "<socket.socket fd=14, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('172.31.0.5', 8899), raddr=('172.31.0.1', 62848)> (FD 14) is already registered"

[abhinavsingh]

@sam-kleiner Thanks for bringing this to my notice. Indeed currently hostname check is hardcoded here

proxy.py/proxy/core/connection/server.py

Line 42 in 06a84ae

ctx.check_hostname = True

In future release I will look to expose these via existing (or new) proxy plugin callbacks. I think we must also add

ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE

but need to experiment before confirmation.

Can you try and hardcode ctx.check_hostname = False for now and see if it works for self-signed upstreams? If it doesn't also try to add ctx.verify_mode = ssl.CERT_NONE.

Please let me know, will look into how to expose them via plugins.

[abhinavsingh]

@sam-kleiner Another quick option which came to me while updating title of this issue:

1. We can add a boolean command line flag which indicates whether to bypass cert check for self-signed upstreams
2. Core can then detect such cases internally and do the needful
3. This is simply avoid need of exposing this configuration dynamically via plugin callback.

Ofcourse, if we opt for such a solution, this is a hardcoded use case hidden behind a flag rather than a dynamic configuration ability. Wdyt?

[abhinavsingh]

--upstream-verify localhost=/path/to/cert --upstream-verify nginx.local=/path/to/cert2

This is an even better option :). Looking to address this for upcoming v2.4.0 release.

[lyz04551]

hi, do we have a boolean command line flag which indicates whether to bypass cert check for self-signed upstreams now? thanks for your reply. @abhinavsingh

[abhinavsingh]

hi, do we have a boolean command line flag which indicates whether to bypass cert check for self-signed upstreams now? thanks for your reply. @abhinavsingh

Hi Lyz,

Unfortunately no. I never got to supporting this officially. Having said that, I will try to spend sometime coming weekend and see what can we do to get this out quickly.

Thanks for bring this back to my attention.

Best
Abhinav

[lyz04551]

@abhinavsingh Thank you very much for your timely reply. This project is very meaningful and helpful to my project, so if you updated the code of cert check for self-signed upstreams, please remind me. Thank you！

[thiagobenine]

Any update on this?

[abhinavsingh]

@thiagobenine @lyz04551 Thank you for bringing this back. It got off the radar somehow. Lemme jump onto this over the coming weekend. Hopefully we'll have something out soon. Best

[noorul]

@thiagobenine We have a use case to do this on condition. Any plans to work on this?

[abhinavsingh]

@noorul @thiagobenine @lyz04551 @sam-kleiner PR #1446 will address this concern.