Multiple concurrent client certs broken with v2.32.3

[jeffreytolar]

We use requests with multiple mTLS client certificates - each certificate is signed by the same CA, but they have different subjects - each subject has different permissions. Each distinct client cert is used by a different requests.Session
Additionally, we make use of ThreadPoolExecutor to make many requests in parallel.
When client certs are in use, urllib3 will load the cert into the SSL context, which, with concurrent requests, will cause the shared SSL context to get modified while it's in use.

The reproducer actually fails with an exception - when we first encountered this, we were seeing the wrong certs get used. (That was with different versions of python and openssl, however, and as mentioned above, they're all signed by the same CA, unlike the reproducer)

Expected Result

The reproducer below, when run with requests-2.31.0:

$ ./bin/python3 test.py
/tmp/py/test.py:24: DeprecationWarning: X509Extension support in pyOpenSSL is deprecated. You should use the APIs in cryptography.
  crypto.X509Extension(b"subjectAltName", False, b"DNS:localhost, IP:127.0.0.1")
/tmp/py/test.py:50: DeprecationWarning: ssl.PROTOCOL_TLSv1_2 is deprecated
  self.ssl_context = ssl.SSLContext(ssl.PROTOCOL_TLSv1_2)
Server started on port 8443...
  OK client2: CN: client2 / URL: /client2
  OK client3: CN: client3 / URL: /client3
  OK client1: CN: client1 / URL: /client1

Actual Result

When run with requests-2.32.3:

$ ./bin/python3 test.py
/tmp/py/test.py:24: DeprecationWarning: X509Extension support in pyOpenSSL is deprecated. You should use the APIs in cryptography.
  crypto.X509Extension(b"subjectAltName", False, b"DNS:localhost, IP:127.0.0.1")
/tmp/py/test.py:50: DeprecationWarning: ssl.PROTOCOL_TLSv1_2 is deprecated
  self.ssl_context = ssl.SSLContext(ssl.PROTOCOL_TLSv1_2)
Server started on port 8443...
FAIL client1: HTTPSConnectionPool(host='127.0.0.1', port=8443): Max retries exceeded with url: /client1 (Caused by SSLError(SSLError(116, '[X509: KEY_VALUES_MISMATCH] key values mismatch (_ssl.c:3926)')))
FAIL client3: HTTPSConnectionPool(host='127.0.0.1', port=8443): Max retries exceeded with url: /client3 (Caused by SSLError(SSLError(116, '[X509: KEY_VALUES_MISMATCH] key values mismatch (_ssl.c:3926)')))
  OK client2: CN: client2 / URL: /client2

Reproduction Steps

Gist: https://gist.github.com/jeffreytolar/ea05b3092df12dc6e5b518e58e6821ad ; this generates a few sets of key/certs, hackily sets the default CA bundle, and then makes a few concurrent requests, each using a distinct client cert.

With:

$ pip freeze
certifi==2024.2.2
cffi==1.16.0
charset-normalizer==3.3.2
cryptography==42.0.7
idna==3.7
pycparser==2.22
pyOpenSSL==24.1.0
requests==2.32.3
urllib3==2.2.1

(top level dependencies are pyOpenSSL and requests)

System Information

$ python -m requests.help

{
  "chardet": {
    "version": null
  },
  "charset_normalizer": {
    "version": "3.3.2"
  },
  "cryptography": {
    "version": "42.0.7"
  },
  "idna": {
    "version": "3.7"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.11.8"
  },
  "platform": {
    "release": "5.15.0-105-generic",
    "system": "Linux"
  },
  "pyOpenSSL": {
    "openssl_version": "30200010",
    "version": "24.1.0"
  },
  "requests": {
    "version": "2.32.3"
  },
  "system_ssl": {
    "version": "300000d0"
  },
  "urllib3": {
    "version": "2.2.1"
  },
  "using_charset_normalizer": true,
  "using_pyopenssl": true
}

[nateprewitt]

Thanks for the report, @jeffreytolar. It does looks like we're not checking the certs provided by the Session before opting into the default context, I've put together fe251aa to disable the default context when certs are present.

However, with testing I'm seeing an exception with the cert being self-signed that wasn't present in 2.31.0. I'm looking into that further but would you mind checking the above patch against your current setup so we can decouple the two issues. If your issue is persisting after we've moved the default context out of the hot path, there may be something else at play with the recent CVE fix.

[jeffreytolar]

So far it's looking like that patch is working in our main setup - thanks for the quick commit!

For the self-signed issue still in the reproducer, I think it's that 2.32.x isn't passing a CA bundle to urllib3, whereas 2.31 did that here:

requests/requests/adapters.py

Lines 257 to 258 in 147c851

	if not cert_loc:
	cert_loc = extract_zipped_paths(DEFAULT_CA_BUNDLE_PATH)

; that causes urllib3 to load the OS default, rather than using certifi

To restore the v2.31 behavior, I think maybe a elif verify is True: pool_kwargs["ca_certs (or ca_cert_dir)"] = extract_zipped_paths(DEFAULT_CA_BUNDLE_PATH) might work in _urllib3_request_context ?

[sigmavirus24]

So far it's looking like that patch is working in our main setup - thanks for the quick commit!

For the self-signed issue still in the reproducer, I think it's that 2.32.x isn't passing a CA bundle to urllib3, whereas 2.31 did that here:

requests/requests/adapters.py

Lines 257 to 258 in 147c851

	if not cert_loc:
	cert_loc = extract_zipped_paths(DEFAULT_CA_BUNDLE_PATH)

; that causes urllib3 to load the OS default, rather than using certifi

To restore the v2.31 behavior, I think maybe a elif verify is True: pool_kwargs["ca_certs (or ca_cert_dir)"] = extract_zipped_paths(DEFAULT_CA_BUNDLE_PATH) might work in _urllib3_request_context ?

Yeah, I thought the optimization broke the zipped paths extraction but couldn't prove it easily

[nateprewitt]

Ok, so that's the same issue reported here (#6710 (comment)) this morning. That explains why calling load_default_certs on the SSLContext fixes it.

Let me take a closer look tomorrow, I'm a little worried if we do it only for verify is True that we'll start the whole custom SSLContext issue over again. Thanks for pointing that out, @jeffreytolar!

[jschlyter]

Any updates on a release including a fix for this @nateprewitt ?