Lighthouse targets its first release, introduces node discovery using discv5 and adds a new, experimental fork-choice algorithm

Summary

Quite a few exiting developments have occurred these past weeks. As a simple summary, we have:

• Lighthouse is targeting it's first release v0.0.1 with a tentative release date of 15/7.
• An initial discovery v5 libp2p module has been built and integrated into lighthouse. Nodes now discover each other using this protocol.
• Implemented a new reduced-tree fork choice algorithm, which (with current testing) has shown ~5x speed improvements (without additional overheads) to previous implementations.
• Passing all v0.6.3 tests. Intend to skip v0.7.x and go straight to the frozen spec, once released.

Next Steps

• Update Lighthouse to the frozen specification.
• Improve and stabilise the syncing mechanism.
• Update the libp2p ETH2-RPC to handle substreams for request/response.
• Add extra network testing/tooling for testing network protocols at larger scales.
• Start pushing the changes in our core BLS library to the Apache Milagro upstream library.

Lighthouse Releases

With the upcoming spec-freeze, we're pleased to announce that Lighthouse will now be producing releases. The first release, v0.0.1 (Genesis) has a tentative date of the 15/7, and will be primarily targeted for researchers and developers. Expected items that will be built in the release can be found here. Continuing on from v0.0.1will be more structured development through releases as Lighthouse begins its interoperability phase with other clients.

Reduced Tree Fork Choice

During the annual IC3 event at Cornell University, a team lead by Vlad Zamfir developed a LMD GHOST fork-choice mechanism dubbed "reduced-tree". This algorithm is directly applicable to Eth2 and we have implemented it in Lighthouse with great results. For those unfamiliar with fork choice algorithms, we discuss them in Update #08. The slides from the presentation can be found here and the full membership of the group is: Vlad Zamfir, Aditya Asgaonkar, Nate Rush, Carl Beekhuizen, Hsiao-Wei Wang & Paul Hauner.

Each Eth2 full-node naturally has some form of database that stores all the blocks in the Beacon Chain so they can be provided to users or used to validate other blocks. Several implementations, Lighthouse included, keep another fork-choice-dedicated block-tree that acts as a cache to reduce the time taken to run the fork choice algorithm.

The key observation from the IC3 group was that the fork-choice algorithm does not need to store information about every single block in the block-tree. Instead, it only needs to be concerned about blocks that are the target of a validators latest vote (message) or blocks that are a decision point between the two. The idea that fork-choice can safely bypass some blocks is the basis of the name "reduced-tree".

The reduced-tree has the advantages that it is smaller to store and requires less operations to iterate through. However, compared to the full-tree, we must define more complex add/remove/prune operations to maintain the reduced-tree. We have not performed detailed analysis of reduced-tree versus full-tree, however we have observed significant performance improvements in our testnets. Notably, there are several optimisations proposed for full-tree fork-choice that are also applicable to reduced-tree, so we can expect to see further efficiency improvements.

New Discovery

As was mentioned in previous updates, we have been working on a discovery v5 implementation. This has been built in a branch of our rust-libp2p library and can be found here. This protocol implements the high level RPC messages FINDNODE, PING and PONG. These are sufficient to use the protocol as a discovery mechanism to find other nodes on the network and determine external IP's for NAT traversal. The libp2p integration allows for libp2p PeerId's to be discovered and connected to once peers are found on the UDP discovery mechanism. Currently topic advertisement and registration have not been implemented and are postponed until further testing of the fundamental implementation is complete.

Fuzzing updates

Fuzzing continues on the new and existing protocols being built on lighthouse. In particular, we have begun testing some of the higher-level functionality within lighthouse, such as state transitions.

We're now targeting block processing by loading a minimal state file and fuzzing attestation, attestation slashing and block header processing functions. Code coverage is optimised by providing libFuzzer a valid corpus (i.e. examples of valid blocks, headers, attestations, etc.) and fuzzing speed is greatly enhanced by loading the beacon state from file.

Our naive per block processing fuzzer has also been updated to use blocks with valid signatures as input.

Give us a hand!

As always, we warmly welcome anyone who is interested to come join us in developing Ethereum 2.0. Find us in the lighthouse Gitter, or on GitHub.