SSZ upgrades, HTTP Rest API Standardisation and v0.6.1 spec implementation progress.

Summary

Eth 2.0 development at Lighthouse has been steady over the last fortnight, seeing hefty updates to some core aspects of Lighthouse as new networking components are defined and the specification is updated.

In summary form:

• SSZ updated to include SOS-style offsets (SOS is the name given to the encoding scheme devised by @karalabe from the geth team).
• A new parser has been built for parsing test formats defined by the Ethereum Foundation and our ssz, tree_hash and cached_tree_hash crates are passing all specified tests.
• Further development towards v0.6.1 spec compatibility
• Standardisation towards a RESTful HTTP API service to be provided by all beacon nodes.
• Further development on bringing DiscoveryV5 (discv5) to life inside rust-libp2p.

Next Steps

• Complete the update to spec v0.6.1.
• Migrate the current gRPC interface between the beacon node and validator client to the specified RESTful HTTP API.
• Complete the development of the discv5 protocol in rust-libp2p and integrate into Lighthouse.

What you need to know

For the interested readers, this section provides further details about the development progress/discussions happening inside the Lighthouse project and the broader Eth 2.0 ecosystem.

Design decisions and the road ahead

Recently, an Eth 2.0 interoperability workshop was held in New York where various client teams and researches discussed design choices and a path towards client interoperability. A live stream was available, however the link appears to be down at the time of writing. We will update this post when a new link is available.

This article will discuss a few topics that were covered in the workshop, however we recommend watching the recorded live stream for further information.

RESTful HTTP API

There has been quite a bit of discussion amongst the client teams and researchers about how to best interface with a beacon node (the software that connects to the chain, the Eth 2.0 analogue of geth or parity-ethereum). Currently, Ethereum clients use JSON-RPC which is widely supported, however there have been some drawbacks which we have an opportunity to address in the design of Eth 2.0 clients. A discussion of the pros/cons of using JSON-RPC can be found in this issue.

There are some interesting advantages to using a RESTful HTTP API in place of a JSON-RPC interface. In addition to being widely supported by most Internet applications, its stateless design allows for easier load-balancing, easier firewall configuration (specific HTTP requests can be rate-limited or blocked), and the API can be easily standardised using a single YAML file.

The Lighthouse team agree with making the switch to support a RESTful HTTP API and as such have started efforts towards standardising such an API. Luke has created PR-1069 which proposes an API that clients will ideally support when Ethereum 2.0 goes live. With a standard interface amongst clients, this could also lead to interoperability between validator clients and beacon nodes from different client implementations.

Discovery v5

Ethereum 2.0 is based on a peer-to-peer network. In such networks, nodes need a mechanism to find each other without relying on some central service. This is typically referred to as a discovery mechanism, of which there are a variety of approaches. P2P networks (such as BitTorrent) typically use a DHT (Distributed Hash Table) to track other peers in the network. This forms a global collection/database of peer addresses where small subsets of of the global collection are stored individually on each node of the network. A common protocol that facilitates this is Kademlia.

Clients that support present-Ethereum use an adaptation of this protocol known as DiscoveryV4 (discv4). A number of proposed improvements to discv4 has lead to a new protocol known as DiscoveryV5 (discv5). In summary, it adds encryption, topic advertisement, and a few other nice features that we'd like to see in a next generation Ethereum client. Security considerations with discovery mechanisms are very important in P2P networks since significant attack vectors can arise as a direct result of a poorly designed mechanism. The eclipse attack is a common example of this, where nodes can be lead to believe in a false head of a blockchain.

We believe that discv5 is a good fit for Ethereum 2.0, however, no implementations are currently able to perform tests and benchmarks using this protocol. Since Eth 2.0 clients will be using the libp2p networking stack, it would be ideal to have this mechanism built into libp2p. To this end, we have been working on a rust-libp2p discv5 implementation and expect to have an initial working version by the next development update.

SSZ Spec-related decisions

As mentioned earlier, v0.6.x of the specification includes significant modifications to SSZ. During these updates, the research team has decided to not include an explicit definition of SSZ de-serialisation. Instead, they leave it to the reader to derive their own method based upon the supplied definition of SSZ serialisation.

This is completely reasonable and supports the concept of a succinct specification. However, it does necessitate that implementers think carefully about edge-cases and attacks which may not be obvious at first glance. During our implementation we found several such cases and produced a HackMD document to assist other implementations and security researchers.

Lighthouse-specific development details

Specification testing

In a effort lead by @protolambda, the new v0.6.x specification came with a vastly more comprehensive test suite. Passing these tests is a critical stepping-stone for a multi-client testnet and we have been hard at work implementing these tests.

The tests are delivered in YAML files and are consistent enough for us to build a framework that only requires a small amount of unique code for each test. We have successfully implemented the generic framework and have implemented generic SSZ tests in less than 100 lines of unique code.

Specification constants

There are many (>30) "constants" that shape an Ethereum 2.0 chain, some of the most significant being SLOTS_PER_EPOCH and SHARD_COUNT. The specification is defined across a fixed set of constants, however it is possible to alter these constants to achieve a chain with different properties. For example, we have been using SHARD_COUNT = 8 and SLOTS_PER_EPOCH = 8 to create chains for testing that run comfortably with small validator counts (e.g. 16 validators in total).

It is a matter of choice for each implementation as to whether or not these "constants" are indeed compile-time constants or if they are really variables defined dynamically at runtime.

With Lighthouse, we have elected to avoid fixing a single set of constants at compile time so that we can support running chains with different constants without requiring a re-compile. This is consistent with how existing, current-Ethereum clients may switch between Ropsten and mainnet without having to recompile.

Whilst we had originally intended to support any arbitrary set of constants declared at runtime (e.g. from a JSON file), we found this added significant complexity to our codebase. As a result, we struck a compromise that we will support any arbitrary set of constants, as long as they are defined at compile-time; however the set that is used can be selected dynamically at runtime. We can declare sets of constants very easily (1-10 lines of code per set), so we may define many sets of constants which will achieve code-simplicity as well as providing flexibility to our users.

Lighthouse's immediate future

In the next fortnight, Lighthouse will undertake significant core library development. As discussed in this post, we are focusing on upgrading our spec-related libraries, overhauling our networking stack to include discv5, and converting our current gRPC beacon-node API to a RESTful HTTP API.

As always, we warmly welcome anyone interested to come join us in the developing Ethereum 2.0. Find us in the lighthouse Gitter, or on GitHub.