First Lighthouse testnet, development approaches, and progress updates.

Summary

We are proud to announce that Lighthouse now produces a functional testnet!

Like many of the Eth 2.0 teams, we adopted a goal to have a working testnet by the end of March 2019. We have been working hard over several months to reach this goal, which marks a major milestone for the Lighthouse project.

In this Lighthouse update we describe our approach to development so far and identify how we have been focusing our time. We then describe the exact nature of the testnet that we have achieved, with an overview of each component, followed by a quick development update.

Our Approach

Teams behind various Serenity clients have adopted a range of approaches to development. When undertaking such an enormous software project, it is necessary to build a "scaffold" of code within which intended features are built.

Sometimes, gaps in the scaffolding are filled with temporary stopgap solutions, with the intention of replacing them later. The advantage to this approach is a shorter turnaround to achieving a running executable that can be tested in the wild, expediting the release of usable public betas. The disadvantages of this approach is that components must be (re)built more than once, potentially leading to significant code restructures down the line.

An alternative approach is to carefully complete as components within the scaffolding with code that represents the final state as closely as possible. While this approach reduces the amount of code-rewriting, it naturally takes longer to achieve a working prototype.

There is no correct way to approach this, both approaches are perfectly valid and all projects are a blend of these approaches.

Our intention is to lean toward the latter approach as much as possible, developing as many of the internal components as we can, with the aim of producing most feature complete code possible at the time. On one hand this has caused us difficulties with the constantly changing spec requiring constant updating, however on the other hand, it has pushed us to produce a more complete networking stack with our extensions to rust-libp2p.

My First Testnet

Over the last month, we have been heads-down working on Lighthouse as part of a big push towards having a testnet ready by the end of March. Our hard work has paid off and we are proud to announce that we now have an operational single-client, private, short-lived testnet. (For a description of what this means, please refer to our Update #07 blog, where we set the expectations regarding Serenity testnets in March.)

Below, we provide further details on what each component of Lighthouse does, and its role within the testnet.

Beacon Node

Presently, the main binary in the lighthouse project is the beacon_node. Upon starting, it will use libp2p to listen on TCP port 9000 for incoming P2P connections, and if a bootnode is specified it will attempt to establish a direct connection with it. Discovery is not yet implemented, so it will not look for additional peers and instead immediately starts syncing a blockchain via the phase0 RPC.

Paul has invested a huge effort getting the beacon node ready and has ensured that it is up-to-date with the Eth2.0 spec version: v0.5.0. While we have a patch to assist with the bugs that were fixed in v0.5.1, we are yet to integrate the correct solution as described by the spec.

Blocks discovered from network peers, or signed by the validator client, are processed by the beacon node and its state is updated. Presently, the state transition function must re-compute and re-merkleise the entire state for every slot, which is highly inefficient. Paul is currently working on an implementation of tree hash caching, which will drastically reduce the amount of computation required for each slot, making everything a lot faster.

If the beacon_node is started with the --rpc flag, then it will also listen on localhost port 5051 for incoming connections from Lighthouse validator clients. This internal RPC enables the validator_client to request data pertaining to blocks and attestations, sign them, and subsequently publish them.

Validator

The validator_client binary runs as a stand-alone process, communicating with a running instance of beacon_node. The validator is responsible for signing blocks and attestations during the appropriate slot; we refer to these actions as "duties". The validator is informed of its duties for the current epoch by performing calls each slot to the GetValidatorDuties endpoint on the running beacon node. This ensures the validator client knows when to sign/produce blocks/attestations at any time, given the validators it holds a private key for.

Communication between them occurs by means of a gRPC connection over 127.0.0.1:5051, for which the protocol definition can be found at lighthouse/protos/src/services.proto.

Upon startup, the validator begins by establishing a connection to a running instance of beacon_node, and uses the Info RPC call to check that it is connected to a beacon node with an appropriate configuration (i.e. a sensible chain_id and genesis_time).

The validator will then request its duties for the current epoch, and keep track of the slot in which it must perform each duty. At the appropriate time, the validator will produce a signed block or attestation, as the duty requires.

At present, the keys used by the validator in this first iteration of the testnet are pre-configured and generated deterministically. In future, they will be generated securely by a separate binary and loaded from disk.

Networking

Adrian has been working hard expanding upon the existing rust-libp2p library, implementing the gossipsub protocol and our own custom implementation of the phase0 wire protocol.

Our testnet demonstrates the following protocols in rust-libp2p: - identify - ping - gossipsub - A p2p network routing protocol, to be used in Eth2.0 - phase0 RPC - Custom RPC for Eth2.0 node inter-communication.

Demo

We have published a video on YouTube, where Adrian demonstrates:

1. Starting the beacon_node using the --rpc flag (so that the validator can connect)
   • libp2p listens on localhost port 9000 with a new PeerId
   • The node subscribes to the beacon_chain gossipsub topic
   • The RPC listens on localhost port 5051.
   • Slots are generated
2. Starting the validator_client
   • Connects to beacon_node on port 5051
   • Obtains duties for the loaded set of validators
   • Continues to produces blocks and attestations for the current slot
3. Another beacon_node is started, but with libp2p on port 8080
   • Connects to first beacon_node process with HandshakeSuccess
   • Receives notification of new block over gossipsub: NewGossipBlock
   • Retrieves block data from first beacon_node process: BlockBodiesResponse
   • Continues to sync blocks from the other beacon_node

Progress Update

This month

We are very excited to have the talented Michael Sproul join the Sigma Prime team. His contributions so far have been excellent, and we are confident that his background in formal verification and experience with Rust will help shape Lighthouse into world-class software.

Our development sprint on Lighthouse this month saw our recent recruits Kirk and Michael leap over any learning curve before them, identify bugs, and push code.

Adrian and Paul have led the pack this month, driving everything forward with networking, syncing, spec updates, and validator client development.

Next Steps

Where the previous month has been a big push towards the March testnet milestone, with many late nights and most weekends spent on Lighthouse (props to Adrian and Paul), we will now be returning to a more sustainable Lighthouse development cycle.

We expect Lighthouse development to temporarily slow down in April while we engage with other Eth 2.0 developers and researchers, who will be in Sydney for EDCON. Paul and Adrian will also be presenting on Lighthouse at EDCON, so be sure to catch them on stage if you can make it! The Easter holidays are also in April, so some members of the team will be taking a much needed break.

While development may slow this month, it will not stop. We expect to release the tree hash caching implementation, alongside all kinds of improvements and bug fixes, as we optimise and stabilise our testnet.

The next major milestone for us will be achieving interoperability with other Serenity clients, transforming our single-client testnet into a multi-client testnet.

Contributors

With all the development going on, it may have been more difficult than usual for external contributors to keep up. Now that we're back to business-as-usual, Lighthouse code should change less frequently, and we will dedicate ourselves to on boarding anyone who would like to contribute. If you would like to to get involved, please find a "good-first-issue" and leave a comment, or drop us a line on the gitter channel.

P.S.

This is not an April fools joke!