Standard grpc mTLS

[georgeliao]

MULTI-1765
Public side of https://github.com/canonical/multipass-private/pull/719

A few things to note
The make_cert_key_pair function originally handles the generation of both server and client key-certificate pairs . The server_name parameter determines the use case, when server_name is an empty string, it indicated the client key-certificate case. In the standard gRPC TLS scheme, the root key-certificate pair is added.

Now, the role of the make_cert_key_pair function is still generating and holding server and client key-certificate pairs. However, the server certificate has evolved into a signed server certificate. As the result, the server side branch must first generate root key-certificate pair and use it to sign the newly created server certificate.

Previously the constructor of X509Cert only handled server and client certifcate generation, distinguishing between them based on whether server_address was empty. Now, with the introduction of root, server and client certifcate, the dispatch is managed by using the CertType enumeration.

Additionally, the X509Cert constructor has been refined to generate certificates in a standard format. Key differences between before and after include adjustments to the serial number format and the inclusion of X509v3 extensions. To inspect a certificate, you can use the following command: openssl x509 -in <cert_path> -noout -text

The certificate paths are as follows:

• Server certificate:/root/.local/share/multipassd/certificates/localhost.pem
• Client certificate: /home/<user name>/.local/share/multipass-client-certificate/multipass_cert.pem
• Root certificate: /usr/local/share/ca-certificates/multipass_root_cert.pem

Snap environment considerations
In the snap environment, the root certificate is stored at /var/snap/multipass/common/data/multipassd/certificates/multipass_root_cert.pem
Unlike /root/, /var/snap/ directory allows other users view its files, making this setup feasible.

Certificate regeneration and migration
The root certificate and server key-certificate pairs area automatically regenerated if either is missing. This mechanism ensures a smooth server key-certificate pair migration when updating Multipass. Upon upgrading, the server startup process will automatically generates root key-certificate pair and use it to sign a fresh server certificate. Consequently, the original server key-certificate pair will be overwritten, enabling successful verification under the new standard grpc TLS.

The Multipass upgrade process should be included in the functional testing as well, both the cmd and gui clients should be tested.

Unit test adaptations
The unit tests have been modified to accommodate changes in the gRPC TLS verification process. The key adjustments include:

1. Mocking the get_root_cert_path to allow usage of string-based certificates.
2. Merge the two server and client key-certificate pairs into one, With MockCertProvider being used on both server side and client to provide key-certificate pair. In the unit testing environment, they can be the same.

Following the commit history and messages is also a helpful way to understand changes that have been made.

[codecov]

Codecov Report

Attention: Patch coverage is 90.27778% with 14 lines in your changes missing coverage. Please review.

Project coverage is 89.13%. Comparing base (2424799) to head (bb0154b).

Files with missing lines	Patch %	Lines
src/cert/ssl_cert_provider.cpp	89.28%	12 Missing ⚠️
src/daemon/daemon_config.cpp	0.00%	1 Missing ⚠️
src/utils/utils.cpp	66.66%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3909      +/-   ##
==========================================
+ Coverage   89.11%   89.13%   +0.02%     
==========================================
  Files         255      255              
  Lines       14603    14663      +60     
==========================================
+ Hits        13013    13070      +57     
- Misses       1590     1593       +3     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[andrei-toterman]

Just a quick pass with some nitpicks, didn't test functionally yet.

[xmkg]

LGTM, except for one small nit.

[xmkg]

Is there a reason for using auto* instead of just plain auto? Both should result in const char*.

[georgeliao]

Yes, they are equivalent in this case. But normally I prefer appending * or & to add a bit expressiveness of data type.

[andrei-toterman]

Hey, @georgeliao! So now, in the local environment, the GUI works fine both when upgrading from the old certificates and when starting with no certificates. But in the case of the CLI, it works when starting with no certificates, but if I upgrade from old certificates, I get the following error

E0211 14:23:49.406977684   36809 ssl_transport_security.cc:1316]       Handshake failed with fatal error SSL_ERROR_SSL: error:0A000086:SSL routines::certificate verify failed.

And in the snap, both when starting fresh or when upgrading, I get the following

[error] [client] Caught an unhandled exception: failed to open file '/var/snap/multipass/commondata/multipassd/certificates/multipass_root_cert.pem': No such file or directory(2)

[georgeliao]

E0211 14:23:49.406977684 36809 ssl_transport_security.cc:1316] Handshake failed with fatal error SSL_ERROR_SSL: error:0A000086:SSL routines::certificate verify failed.

About this root certificate is unsync with server certificate issue in development environment, maybe we can add a check at here, which checks not only the existence of the certificates but also whether the root certificate is the one who signed the server certificate. If not, everything will be re-created. This check can be done by openssl c-api. However, not sure the juice worth the squeeze.

[error] [client] Caught an unhandled exception: failed to open file '/var/snap/multipass/commondata/multipassd/certificates/multipass_root_cert.pem': No such file or directory(2)

This is fixed in the latest version of the PR.

[andrei-toterman]

Yeah, for the dev environment, I don't think it's worth doing the effort. We already have to remove stuff manually, so this is just an extra thing to remove.

[andrei-toterman]

Great, now both the CLI and the GUI work as expected, both locally and in the snap, both when upgrading or installing fresh. Thanks, @georgeliao!

Now, one thing to discuss with everyone: the purpose of these changes was so that we could remove our grpc patches, and that goal is accomplished by this PR. And now the CLI verifies the server certificate, which couldn't be avoided without the patches. But the GUI can avoid verifying the server certificate, using the plain vanilla grpc dart library. My question is: should the GUI also verify the server certificate, to be in line with the CLI, or should we keep the existing, functioning behavior of not having the GUI verify the server certificate?

[ricab]

As discussed elsewhere, let's have the GUI also adhere to the new scheme, preferably in a separate PR.

[ricab]

Thanks @georgeliao, lots of good work in here!

I am only doing a tertiary review in this case, so I glossed over most of the scary low-level certificate stuff. @andrei-toterman and @xmkg, we're relying on you on that front 💪 If you could provide assurance that you've verified sanity of all the nitty-gritty, that would be much appreciated. All of this would be for nothing if security were somehow broken on the foundation...

Other than that, I have some proposals for path derivation and the initialization. Let me know what you think.

[ricab]

Was this case only ever hit in clients, before?

[georgeliao]

Are you talking about the original cn_name_from function? That one was used both by the server certificate and client certificate. server_name non-empty indicates the server case. In our new implementation, this dispatch is preserved by the line

        const auto cn = as_vector(cert_type == CertType::Root     ? "Multipass Root CA"
                                  : cert_type == CertType::Client ? mp::utils::make_uuid().toStdString()
                                                                  : server_name);

[ricab]

OK, it might be good to add a comment next to the signature of the SSLCertProvider constructor. Something like: // leave server_name empty for clients; choose a (non-empty) name for servers.

[xmkg]

We can get rid of the magic numbers here in favor of something more readable:

    constexpr std::chrono::seconds one_year = std::chrono::hours{24} * 365; 
    constexpr std::chrono::seconds ten_years = one_year * 10; 
    const auto valid_duration = cert_type == CertType::Root ? ten_years : one_year;
    X509_gmtime_adj(X509_get_notAfter(cert.get()), valid_duration.count());

We can also simplify this even further when we switch to C++20 with the help of "std::chrono::years".

[georgeliao]

Yep, it is definitely better with chrono time units and automatic unit conversion.

[georgeliao]

It is a bit funny that hours is the biggest unit before c++20.

[ricab]

We have a constant for the name of the daemon.