Quic Support

Most Lighthouse users know that Lighthouse by default listens on two ports, TCP 9000 and UDP 9000. Since our last release, we've added a third one (UDP 9001). In this post we'll explain why and if you're a Lighthouse user, how to support it and hopefully lower some of your system resources.

If you read nothing else in this post, the only thing you really need to know is if you are running Lighthouse, update to v4.5.0 and add a port-forward in your router for UDP 9001.

QUIC

It all begins with QUIC. QUIC is a general purpose transport layer network protocol allowing two computers to exchange data in a reliable and secure way. For the technical reader interested in the specifics, the wiki page gives a good overview.

Up until the previous release, Lighthouse's primary form of communication with other nodes on the network was via the TCP transport. This is the tried and true method most computers on the internet communicate with. Recently (in the scheme of things) work was done to build a new protocol to optimise this communication. One effort is the QUIC protocol.

To give a high level overview, let's first describe TCP and then introduce QUIC and how it improves on TCP. As networking applications have evolved over recent years, greater and greater complexity has been built on top of the base communication protocols, such as TCP. In many modern applications, these protocols are stacked one on top of the other. As an example, consider connecting to a website.

1. This connection first requires us to establish a TCP connection, which requires 3 messages back and forth between our computer and the webserver (SYN, SYN + ACK and ACK).

2. Once the TCP connection is established, we usually want the information sent on the new connection to be encrypted, authenticated and checked for integrity. We therefore set up some kind of cryptographic protocol like TLS, which requires additional messages (3 in the case of TLS 1.3) between both parties on top of the established TCP connection.

3. On top of this we can then start exchanging information with the web server. In applications (like in Lighthouse) we may want to use the same connection for multiple protocols, a concept known as multiplexing, requiring yet another protocol layer on top of the original TCP+TLS connection.

In the case of TCP the cryptographic handshake (step 2 above) can only run after the connection has been establised (step 1). This requires at least 2 round-trips. QUIC improves on this design by combining both the connection establishment handshake (step 1) and the cryptographic handshake (step 2) into one, thus requiring 1 round-trip only. This is a significant reduction in latency.

Lastly QUIC also combines multiplexing (step 3), enabling various optimizations, e.g. preventing head-of-line blocking on the connection level.

The Rust libp2p implementation has been working at integrating QUIC into their library and performing some initial benchmarks comparing QUIC to the TCP transport we have been using. Some of the results look quite promising.

The initial benchmarks show significant gains in latency and throughput compared to the TCP transport stack. The latency gains are due to QUIC's handshake improvements itself, i.e. improvements through better protocol design. The throughput gains are due to the more sophisticated multiplexer implementation in the Rust QUIC implementation (namely Quinn). Compared to the multiplexer implementation used in our TCP stack, namely the Rust Yamux implementation (implementation level), the QUIC implementation can better saturate a connection. For the avid reader, you can read more in https://github.com/libp2p/rust-yamux/issues/162. Overall the results are compelling enough that we thought we'd test it out in practice by introducing QUIC into Lighthouse.

Lighthouse and QUIC

We have introduced QUIC into Lighthouse in v4.5.0. If you have updated your Lighthouse node to the latest version, chances are you have QUIC enabled. There are some caveats in utilising this transport however. Firstly (and probably most importantly), Lighthouse now listens on a new UDP port (9001 by default). It can be set via the CLI flags --quic-port and --quic-port6 (for IPv6). QUIC can also be disabled via the --disable-quic CLI flag.

As Lighthouse now listens on a new UDP port, in order to be contactable users must now ensure their nodes are contactable on this extra UDP port in order to take advantage of the QUIC transport. This might mean adding a new port forward mapping in your router, and/or opening up your local firewall.

Secondly, it should be mentioned that this upgrade is fully backwards compatible. We still support TCP and this is the preferred connection with all other clients (as the current specification dictates). QUIC will only function and attempt connections with other Lighthouse nodes, so we should only start seeing benefits between Lighthouse <-> Lighthouse connections. The more Lighthouse peers you have, the more QUIC connections you should have (provided your peers have upgraded to the latest Lighthouse version and port forwarded their QUIC port).

Since the v.4.5.0 release, we have been tracking the kinds of connections we have been seeing on mainnet. An example from a single mainnet node over the last week is given below.

This particular node has on average (over the last week) 50 Lighthouse peers. If all of these peers were on the latest version of Lighthouse and connected over QUIC, we would expect to see around 50 QUIC connections. Since the release the number of QUIC connections across the network has risen, but it still indicates that a majority of Lighthouse users either haven't updated their clients yet or haven't made their nodes contactable via the QUIC port. We have been hoping to watch the QUIC connections in this graph grow over time, but it is currently stagnating.

We are in the process of analysing these connections to statistically determine if there are significant performance improvements for these kinds of connections and the more Lighthouse nodes on the network supporting these connections the greater the data we have to analyse.

Summary

If you are a Lighthouse user and want to either help with our little QUIC experiment or want to try and get a more performant Lighthouse node, please update to v4.5.0 and port forward UDP port 9001. Once we have collected and reviewed enough data we'll make a follow up post with the results of how performant these connections are and maybe aim to encourage spec adoption if they are significantly positive.