Up-to-specification and pushing forward; introducing chaos to achieve stability; preparing for interoperability.

Summary

It's been crunch time over the last month with major progress being made on all fronts. We have focused on improving test-net stability and our network stack as we prepare to achieve cross-client interoperability.

To summarise what we've been up to:

• We are up to date with the spec freeze at v0.8.1.
• Introduction of a basic chaos monkey to improve real-world testing.
• Significantly improved test-net stability, reduced storage requirements, and more efficient memory management.
• Further testing on fork-choice rules, fixing bugs along the way.
• Updated network stack, conforming to the current libp2p rpc proposal.
• Working on SSZ merkle partials, thanks to ConsenSys R&D (the Quilt team specifically) for their PR.
• Specification of our REST API, collaborating with potential users to ensure completeness.
• Sigma Prime welcomes a new full-time developer to the team, dedicated to Lighthouse contributions.

Next Steps

• Finalise the libp2p networking specification and implementation.
• Further optimisation of state storage efficiency.
• Integration of the Eth1 deposit contract.
• Complete all REST API endpoints.
• Collate all changes, and merge into a v0.0.1 release of Lighthouse.

Lighthouse Releases

In our previous update we announced that we are targeting a v0.0.1 release in July. While working towards a release continues to be the plan, we have identified a series of optimisations and networking improvements that we believe should be included Lighthouse's first release. We will be ready to release after merging the various feature branches that have been produced this month, which we expect to be complete by mid-August. Our absolute deadline for this release is September 4th, right before we head to Canada for some face-to-face Eth2 collaboration efforts.

Up-to-date With The Frozen Spec ❄️

Michael has been hard at work keeping Lighthouse up-to-date with the latest Eth2 specification, v0.8.1. Our beacon node is now successfully passing all tests . As part of this work Michael also discovered a small bug in the specification.

A Monkey Takes Residence In Lighthouse

The concept of chaos engineering has been around for some time. One of the earliest implementations developed by Netflix was Chaos Monkey. We have begun using Pumba: Chaos testing tool for Docker for our test-net as the first iteration of our very own chaos monkey, and we are already producing very useful results. So rather than a chaos monkey, maybe our chaotic savanna creature is instead a chaos warthog.

Stability & Efficiency Improvements

After achieving a functional test-net a few months ago, we are now focusing on improvements related to the stability of the test-net, as well improving efficiency around resources such as disk and memory.

• Reduced Tree Fork Choice Testing - As discussed in our previous update, an improved fork choice algorithm was incorporated in Lighthouse and Eth2 more broadly. Further testing has identified some bugs and has led to considerable improvements in efficiency, especially around disk utilisation.

• State Storage Reduction - As we've learned with Eth1, minimising the storage footprint of a client is highly desirable. Database efficiency in Eth2 is a relatively new endeavour and we're exploring the solution space. In the coming weeks we're planning to implement two database schema changes that will significantly improve BeaconState storage efficiency.

Networking Updates

Adrian has been working on this PR in an effort to standardise the way different clients build their libp2p implementations, particularly around the individual messages exchanged in P2P communication.

Lighthouse has re-written its libp2p RPC implementation to align with the above PR. As the PR is being finalised, we expect further updates to our RPC.

The past week has also seen increased focus on client interoperability. In this vein, we have discovered some discrepancies between the rust-libp2p and go-libp2p implementations. In particular, the go-libp2p implementation of floodsub is not compatible with the rust-libp2p version. Since we modelled our gossipsub implementation on the existing floodsub implementation, our gossipsub implementation had to be modified so as to enable compatibility with the go implementation.

In the coming weeks, we expect to finalise the libp2p standardisation effort, have our RPC conform to this standardisation, have a fully go-compatible libp2p networking stack, and have initial short-lived communications with clients that are also compatible with the go libp2p network stack.

Lighthouse API

It's not much use having Eth2 clients talking to each other, if we can't talk to them. We previously spoke about our decision to implement a RESTful HTTP API, which has seen significant progress this month. The specification has been extended to cover everything we think would be necessary for a phase zero test-net. This can be found in the Lighthouse repository under docs/rest_oapi.yaml, or on SwaggerHub.

The API implementation remains a work-in-progress, and so the specification will be most useful to any forward-looking developers who intend to consume the API in future. If you notice something missing from the API, or think of a feature that you'd really like to see, we'd love for you to create an issue with your suggestion. Or, if you're feeling really keen, a PR on the OpenAPI spec. would be even better!

Fuzzing & Security Updates

Fuzzing continued this month and successfully identified a bug that was independently discovered by @hwwhww. This is the first case of our fuzzer affirming bugs which were related to the Eth2 specification; which in our view affirms the need for our fuzzing effort.

While our fuzzing efforts mature, our computation requirements increase. As such, we are exploring options to use a sustainable alternative to maintaining our own dedicated computational resources.

A notable vulnerability has been identified in the popular SmallVec Rust library this month. We have examined Lighthouse and believe that it is immune to this vulnerability, however testing is still ongoing.

Finally, we look forward to integrating Lighthouse into the Eth2 differential fuzzer currently being developed by the Ethereum Foundation. In fact, the block_header and shuffle fuzzers already include Lighthouse!

Give us a hand!

As always, we warmly welcome anyone who is interested to come join us in developing Ethereum 2.0. Find us in the lighthouse Gitter, or on GitHub.