grpc: Advertise h2 support in ALPN

[inahga]

A compliant HTTP/2 server advertises support for h2 in the TLS ALPN
field. See https://datatracker.ietf.org/doc/html/rfc7540#section-3.1.

grpc-go did not always check for the presence and validity of this
extension. It did so in a relatively recent version.

This didn't affect boulder, because neither a boulder client nor boulder
server was advertising the ALPN.

However, now that we're trying to update Consul, we run into a problem
with failing healthchecks. Consul now uses an upgraded grpc-go with ALPN
enforcement. When Consul makes a gRPC healthcheck against boulder, it
rejects the healthcheck due to the missing ALPN.

Fix this problem by appropriately advertising the h2 ALPN.

Normally, grpc-go would handle this for us, but because we're handling
a raw tls.Config struct, we have to do it ourselves.

[inahga]

This is ready to be reviewed, but I still need to verify whether or not this requires a three legged deploy. So don't merge it just yet.

[jsha]

I somewhat prefer to do this in cmd/config.go TLSConfig.Load() where the *tls.Config is initially constructed:

	return &tls.Config{
		RootCAs:      rootCAs,
		ClientCAs:    rootCAs,
		ClientAuth:   tls.RequireAndVerifyClientCert,
		Certificates: []tls.Certificate{cert},
		// Set the only acceptable TLS to v1.3.
		MinVersion: tls.VersionTLS13,
		NextProtos: []string{"h2"},
	}, nil

The place where you've added NextProtos for the client side seems fine, though.

Also worth noting that WFE and SFE don't use TLSConfig.Load() for their HTTPS servers and so wouldn't be affected by this.

[inahga]

Sounds good. I did try to hoist it out as far as I could but missed this :)

[inahga]

I'm quite confident this requires a three-legged deploy, just from a reading of the ALPN spec. https://www.rfc-editor.org/rfc/rfc7301.html#section-3.2

   It is expected that a server will have a list of protocols that it
   supports, in preference order, and will only select a protocol if the
   client supports it.  In that case, the server SHOULD select the most
   highly preferred protocol that it supports and that is also
   advertised by the client.  In the event that the server supports no
   protocols that the client advertises, then the server SHALL respond
   with a fatal "no_application_protocol" alert.

So consider during a deploy: if a newer boulder client advertises h2 to an older boulder server, the server will respond with no_application_protocol.

We'll have to turn off ALPN enforcement via environment variable fleetwide first, before taking this change. I don't think this is something we can get around with feature flags.

[jsha]

In https://pkg.go.dev/crypto/tls#Config:

// NextProtos is a list of supported application level protocols, in
// order of preference. If both peers support ALPN, the selected
// protocol will be one from this list, and the connection will fail
// if there is no mutually supported protocol. If NextProtos is empty
// or the peer doesn't support ALPN, the connection will succeed and
// ConnectionState.NegotiatedProtocol will be empty.
NextProtos []string

(emphasis mine)

I think this means we could set NextProtos on the server side but not the client side, deploy, and then set it on the client side. Indeed, adding this minimal patch locally and running integration tests passes:

diff --git i/cmd/config.go w/cmd/config.go
index e006a12ad..8a32d6622 100644
--- i/cmd/config.go
+++ w/cmd/config.go
@@ -218,6 +218,7 @@ func (t *TLSConfig) Load(scope prometheus.Registerer) (*tls>
                Certificates: []tls.Certificate{cert},
                // Set the only acceptable TLS to v1.3.
                MinVersion: tls.VersionTLS13,
+               NextProtos: []string{"h2"},
        }, nil
 }

[inahga]

Fascinating. I even tried the reverse (client advertises while server doesn't). And that passes tests as well. I can coerce tests to fail by setting, say, the server to advertise h2c, and the client to h2.

Therefore indeed a single legged deploy ought to work. I wonder if we even need to take the client change at all then.

Thanks for finding that!

[aarongable]

I like that plan. Still a three-sided change, but the first side (adding H2 on the server side) should unblock the Consul upgrade, and we can follow up with the client side change in a future tag.

[inahga]

Ah hang on, #8696 is spitting at that change.

[inahga]

Hmm, that seems to have been a flake. I reran the tests on both these PRs a few times now and haven't seen it again. Let's proceed?