server: gRPC server does not set NextProtocols

[tbg]

Describe the problem

While upgrading gRPC from v1.56.3 to v1.68.0, I noticed that post-bump CRDB was unable to connect to pre-bump CRDB nodes, the nodes using gRPC at v1.68.0 would print

W241128 13:28:08.615942 205 server/init.go:402 ⋮ [T1,Vsystem,n?] 25 outgoing join rpc to ‹127.0.0.1:29000› unsuccessful: ‹rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: credentials: cannot check peer: missing selected ALPN property"›

This check is new in v1.68.0 (as in, it's not yet there in v1.56.3). So this problem likely persists, but it has not caused any issues. This new check was added to provide a better failure mode. Apparently every HTTP2 server needs to support ALPN, but our HTTP2 server is gRPC and it is a bit of a mystery how we end up failing this check, especially given that everything works once we disable the check, which we can do via an env var.

See the full initial analysis here: #136278 (comment)

With a fork of grpc-go that turns the check off, #136278 can merge. This issue serves as a reminder to get to the bottom of this problem, as future versions of gRPC are likely to hard-code the check, and we don't want to have to be on a fork forever.

To Reproduce

See #136278 (comment)

Jira issue: CRDB-45001

[tbg]

Oh I understand it now. Our TLS config uses GetConfigForClient:

/pkg/security/certificate_manager.go#L416-L421

func (cm *CertificateManager) GetServerTLSConfig() (*tls.Config, error) {
	if _, err := cm.getEmbeddedServerTLSConfig(nil); err != nil {
		return nil, err
	}
	return &tls.Config{
		GetConfigForClient: cm.getEmbeddedServerTLSConfig,

in v1.56.3, gRPC does add h2 to NextProtocols, but only for the config itself. It doesn't make sure that the configs returned by GetConfigForClient do the same:

// NewTLS uses c to construct a TransportCredentials based on TLS.
func NewTLS(c *tls.Config) TransportCredentials {
	tc := &tlsCreds{credinternal.CloneTLSConfig(c)}
	tc.config.NextProtos = credinternal.AppendH2ToNextProtos(tc.config.NextProtos)
	return tc
}

In v1.68.0, this has been rectified in a fairly recent change: grpc/grpc-go#7813

https://github.com/grpc/grpc-go/blob/v1.68.0/credentials/tls.go#L201-L215

The long and short of it is that we need to make sure to disable this ALPN check until every CRDB version our binary might need to communicate with contains the v1.68.0 upgrade. That should be straightforward, and thanks to mixed-version testing it'll be pretty obvious if we get it wrong either. I'll sprinkle comments in go.mod to that effect.