This article describes how to correctly configure and debug your NAT settings to maximize the effectiveness of the Lighthouse client.

This is a practical guide, but I've added some basic theory and information around how Lighthouse functions to aid with users understanding of the issues addressed in this post. Feel free to skip to the most relevant section for you.

What is NAT?
Lighthouse and NAT
IPv4 NAT Configuration
IPv6 Configuration
Debugging NAT Issues

What is NAT?

Network Address Translation (NAT) is a method typically employed on most home routers. It allows your ISP to allocate your entire household a single IPv4 address. All computers within your house then use that single IP address, typically known as a "global IPv4 address".

The reason this is done, is because there are limited (order 3 billion) public IPv4 addresses and there are a lot more devices in the world than this, so every device cannot have a public IPv4 address.

To see how it works in practice, lets consider a typical home network, with standard router. When a packet is sent from a computer in this network (lets say it has the IPv4 address 192.168.0.10) to the internet (say a Google server), the internal computer first sends the packet to its gateway, i.e the router. The router is the device that holds the single public IPv4 address that your ISP has assigned to your account. The router forwards that message on to the Google server, but it changes the source address of the packet from 192.168.0.10 to the routers public IPv4 address. This way when the packet reaches Google's server, it knows to send the response back to the IPv4 address of the router. Once the router receives the response, it knows to forward it on to the originating computer, 192.168.0.10. This is fundamentally how NAT works and your typical router does this for all computers in its internal network.

Lighthouse and NAT

Lighthouse can function behind routers imposing NATs, however it will perform significantly better when the router is properly configured to allow traffic through its NAT and firewall. The following sections will describe the best way to do this, whereas this section will focus on why Lighthouse works better and why we recommend configuring NAT for Lighthouse.

Discovery

Lighthouse uses a protocol called discv5 in order to find peers that are suited for the network we are running on. By default, it opens UDP port 9000 (on IPv4 and IPv6, if its enabled). Discovery works by initially connecting to a predefined set of nodes called bootnodes. From these nodes we ask if they know any other peers. In a recursive and somewhat complicated manner, we do this search until we have collected and connected to our desired number of peers.

Peers can advertise themselves through a signed record called an ENR. This is just an object that effectively contains a peer's IP address and port on which we can connect to. These ENRs are what are transmitted through the discovery protocol. If a node sends us an ENR (think IP address and port) and we find that we cannot connect to them (i.e the IP address is wrong, or their firewall or NAT is not configured to allow incoming connections) then we reject their ENR and therefore will not forward it to other peers who ask us for known ENRs. Therefore non-contactable ENRs should not propagate through the discovery network.

What this means is that, if you do not correctly open your UDP 9000 port (i.e configure the firewall/nat on your router to forward this port) then your node will not be advertised to others and you will not receive incoming connections from other peers (I will explain why this isn't great in the following libp2p section).

One further complication related to discovery is the ability for a Lighthouse node to correctly identify and subsequently advertise its external (think global) IP address. It is not as simple as just checking our local IP address, because packets from our computer can be behind one or more NATs which change the source IP address of our packets. In a conventional home IPv4 network, our external IPv4 address is typically our routers IPv4 address and not our local one. For IPv6, we typically have a globally routable IPv6 address (I leave IPv6-focused discussions to another post).

The way we determine our external IP address in Lighthouse is that initially the client will start up without any IP address or port set in its ENR. This means on startup, we do not advertise our IP/PORT on discovery. Once running, Lighthouse continually tries to determine its external IP address. This is done by connecting to other nodes on the network (via discovery, i.e UDP 9000) and asking them what IP address and port they are seeing as the source of our packets. If enough nodes agree on a specific IP/PORT pair, we then set this in our ENR and see if we get incoming connections. If we get incoming connections, we know we have set our external IP address correctly and all the configuration in our router is working as expected. If we don't see incoming connections we reset our ENR to having no IP/PORTs set and try again later.

A complication with this process is that routers can perform their NAT translation differently. Even if you have forwarded your ports correctly (meaning inbound traffic is directed to your Lighthouse node), outbound traffic can be mapped to random ports on our router for each connection. This means our peers will each individually see a unique set of IP/PORT as the source of our packets. If this is the case, then we will not see enough nodes agreeing on what is the source of packets coming from your Lighthouse node (because the outbound ports are randomized). To correctly resolve this, an extra configuration in routers is sometimes required. This configuration is called an SNAT rule. This instructs the router to map an external port for all traffic coming internally from an internal host and port (i.e our Lighthouse node and UDP 9000). I'll talk about this in the configuration and debugging sections below.

Libp2p

Lighthouse, by default, will listen on port TCP 9000 and UDP 9001 (for QUIC support, see our blog on QUIC support). We use these ports for direct communication and data transfer to other peers on the Ethereum consensus network. Lighthouse has a concept of target-peers. This is a CLI configuration parameter and at the time of writing this post, defaults to 100. Lighthouse uses discovery to search for peers until we have connected to this number of peers. To allow new nodes to join the network and to maintain a healthy peer set, Lighthouse allows 10% more than this number to connect to us. In the default case, this is 10 extra peers.

Every 30 seconds, Lighthouse then evaluates it's peer set. If we have more than our target number of peers, we prune the excess. This process is not random. We prune peers that are less useful to us (i.e may not be helping with our attestations, have been slow to respond to messages or simply are just not as performant as the others). Over time this allows Lighthouse to maintain a very useful and distributed (in terms of peers that we need for attestations and other metrics) set of peers. This fundamentally improves the efficiency of the Lighthouse client.

Setting up correct port-forwards on our libp2p ports will aid in new connections and increasing our peer count beyond the target. This excess is healthy for Lighthouse in order to do the pruning process mentioned above. Although Lighthouse will still function without incoming connections, it may cycle through peers via this pruning process a little slower and we therefore recommend everyone correctly configure their network to get the optimal performance from their Lighthouse node. As a side benefit, it helps the connectivity of the network as a whole.

The remaining sections of this article will discuss how to do this and how to debug possible errors.

IPv4 NAT Configuration

This section aims to describe how to setup a conventional router and some possible pitfalls that can occur in the process.

Port Forwarding

This is the process of telling a router that incoming traffic on a specific port should be redirected to a specific internal host and port. We want to port-forward all our Lighthouse node's ports. By default these are:

UDP 9000
TCP 9000
UDP 9001

Most routers have an option in their firewall called port-forwarding. If it doesn't exist you may need to lookup your specific router on how to setup port forwards. Typically, you need to specify an external port and map it to an internal host and port. We recommend mapping the external port to the same internal port, to avoid confusion and make it easier for debugging. The Lighthouse book provides a guide on how to setup port-forwarding.

As an example, if your Lighthouse node has an IPv4 address of 192.168.0.15. You should set up a port forward that maps external port UDP and TCP 9000 to internal host 192.168.0.15 on port 9000. And the same for UDP port 9001.

SNAT

SNAT (Source NAT) is also required on some routers for discovery to correctly identify your external IPv4 socket and therefore advertise your Lighthouse node on the network. Again, routers often have a variety of ways to configure this and searching your router for "how to configure an SNAT rule" may aid in how to do this. Fundamentally, this involves setting up an outbound rule that tells the router that traffic from our Lighthouse node originating from UDP port 9000, should be mapped to output port UDP 9000 on the router.

If you cannot setup SNAT rules or your router doesn't support this, you can tell Lighthouse to not automatically find your IP address, instead you can set your ENR fields manually, via the CLI flags --enr-address, --enr-port and --disable-enr-autoupdate.

IPv6 Configuration

With IPv6 it is common that your internal computers will have global IPv6 addresses. Therefore, IPv6 will not be NAT'd by your router (this is one of the main motivations for IPv6). So there are no port forwarding rules required to set up. However, for security reasons, your router will not allow arbitrary external traffic to reach your internal computers, even if their global IPv6 addresses are known. A router will typically have default rules to prevent this traffic. Therefore, to set up IPv6 support for Lighthouse, you need to open a port in your firewall. This will just be an IPv6 firewall rule that allows external traffic on specific ports to reach our specific internal Lighthouse computer.

For Lighthouse we need to allow traffic on IPv6 ports TCP and UDP 9000 and UDP 9001 (by default), for the specific host running Lighthouse. Router firewalls usually have a range of settings and again it might be worth searching your specific router for allowing IPv6 traffic. I will however give an example here in the hope it is useful.

Lets say we have a Lighthouse node with IPv6 support in an internal network. By default if the CLI --listen-address is not specified (Lighthouse versions >= 7.0.0 and computer has a globally routable IPv6 address) or --listen-address :: is specified, Lighthouse will have IPv6 enabled. In linux, you can run:

$ ip -6 addr

Which should show you a list of local IPv6 addresses. Globally routable IPv6 addresses exist in the 2000::/3 range, which means they start with a 2 or a 3. You'll likely see 200x at the start of your IPv6 address to identify it as global. You can also check with curl ifconfig.me or go to https://whatismyipaddress.com to see if you have a globally routable IPv6 address. Lets assume in this example we have an address of 2001:5555::10. We then need to set an IPv6 firewall rule in our router that allows traffic originating from the internet (typically the WAN interface) on the ports we need, that also has a destination address of 2001:555::10.

Once the firewall rules are set up, Lighthouse should receive IPv6 traffic and register that it's IPv6 ports are open (we discuss how to determine this in the following section).

Debugging NAT Issues

Determine if Network Configuration is Working

There are two main ways Lighthouse reports if it has a correct network configuration.

HTTP API

If you run Lighthouse with the --http flag, Lighthouse exposes its http api. One method queries Lighthouse about its NAT configuration. Assuming you are running this command on the same machine as the Lighthouse node, you can run:

curl http://localhost:5052/lighthouse/nat

The result should be something of the form:

{"data":{"discv5_ipv4":true,"discv5_ipv6":false,"libp2p_ipv4":true,"libp2p_ipv6":false}}

If you have jq installed, it's a bit nicer:

curl http://localhost:5052/lighthouse/nat | jq

With output:

{
  "data": {
    "discv5_ipv4": true,
    "discv5_ipv6": false,
    "libp2p_ipv4": true,
    "libp2p_ipv6": false
  }
}

Which tells you for each IP version whether Lighthouse thinks its contactable or not (i.e is NAT configured correctly or not). In this example, IPv4 has NAT configured correctly but not IPv6. This will be the same result if IPv6 is not supported or running in Lighthouse.

Metrics

Lighthouse has a suite of metrics to allow users to monitor their nodes. See Lighthouse metrics for further instructions on how to set them up and add dashboard. The Network dashboard shows information about the NAT status of the Lighthouse node.

nat-metrics

This example shows a correct configuration for IPv4 and IPv6.

Debugging

If Lighthouse is reporting a NAT port closed that you are expecting to be open, this section lists a number of ways to debug the root cause of the problem.

A Note on CGNAT

It is becoming more common for ISPs to put their customers behind a Carrier Grade NAT (CGNAT). This is where the ISP also does NATing on their end and it is impossible for you to set up working port forwards on your own home router. If you have set up port forwards and Lighthouse is reporting all ports still closed, it could be the case that you are behind a CGNAT. One way to determine if you are behind a CGNAT is to check the IP address in your router and compare it to curl ifconfig.me or the address obtained from https://whatismyipaddress.com. If the addresses do not match, it is likely you are behind a CGNAT. You will need to contact your ISP to opt out of the CGNAT or get them to port-forward the ports for you.

ENR Advertisement

In order for us to get peers, we need to advertise our IP address. As discussed above this requires discovery to find its external IP address. This requires UDP port 9000 to be open (by default), and sometimes an SNAT rule set up in the router. There are a few ways to check if we correctly discovered our external IP address.

Firstly, we can check our ENR and see if it has an IP address listed.

The ENR can be found (by default) at:

~/.lighthouse/<network>/beacon/network/enr.dat

alternatively via the HTTP API via (assuming jq is installed, if not just reading the field from the localhost:5052/eth/v1/node/identity endpoint should work):

curl localhost:5052/eth/v1/node/identity | jq '.[] | .enr'

We can view an ENR by either using the online tool enr-viewer or by using a cli tool enr-cli (obtain it via cargo install enr-cli). An example of viewing an ENR with the enr-cli tool installed is via:

enr-cli read $(cat ~/.lighthouse/mainnet/beacon/network/enr.dat)

This should output something like:

ENR Read:
Sequence No:11070
NodeId: a2af888a72b392136d03aa3f8adc9473b1f3601afc88cfd08cc4958f37ea3fd4
EnodeId: enode://a2af888a72b392136d03aa3f8adc9473b1f3601afc88cfd08cc4958f37ea3fd4@x.x.x.x:9000
Libp2p PeerId: 16Uiu2HAkzVy1Zvayo1pjK1e6CzDiLoTEuZPStexHDwutktZgHujf
IP:x.x.x.x
IP6:x:x:x:x:x::x
TCP Port:9000
TCP6 Port:9000
UDP Port:9000
UDP6 Port:9000
Eth2 Field:
    Fork digest: 6a95a1a9
    Next fork version: 04000000
    Next fork epoch: 18446744073709551615
    SSZ Bytes: 6a95a1a904000000ffffffffffffffff
Known multiaddrs:
/ip4/x.x.x.x/udp/9000
/ip4/x.x.x.x/tcp/9000
/ip6/x:x:x:x:x::x/udp/9090
/ip6/x:x:x:x:x::x/tcp/9090

If there is an IP address here (which there is in this example), it means Lighthouse has identified an external IP address and port. We should check that it matches our actual external IP address, i.e via https://whatismyipaddress.com.

This can also be witnessed by looking through the Lighthouse logs for Address updated. Logs can be found (by default) at:

~/.lighthouse/<network>/beacon/logs/beacon.log

For example we can search via:

grep "Address updated" ~/.lighthouse/mainnet/beacon/logs/beacon.log

If no IP address is specified (or no log is returned), it means there is something wrong with our UDP port-forwards and/or SNAT rules.

We can debug this further, by looking at discovery logs specifically. We need access to extended discovery logging and this requires a restart of Lighthouse. To get access to enhanced discovery logs run the Lighthouse binary with the prefix RUST_LOG=discv5=debug.

For example:

RUST_LOG=discv5=debug lighthouse --network mainnet ...

This will enable debug logs for discv5. We don't recommend running this for extended periods of time as it generates a lot of logs and will consume more resources to process and store them.

With this running, we can examine the discovery logs. Specifically we want to search for PONG messages which tell us what our peers see our external IP address and port as.

We can examine these logs via (for example):

cat ~/.lighthouse/mainnet/beacon/logs/discv5.2025-02-10.log | grep "Pong"

This provides logs of the form:

2025-02-10 11:01:24 DEBUG  Received RPC response PONG: Enr-seq: 424, Ip: 2403:6322:a6f2:3ddd::8770,  Port: 9000 PING: enr_seq: 276 Node: 0x64ce..8c9e, addr: [2a01:4f9:3071:1163::2]:9000
2025-02-10 11:01:39 DEBUG  Received RPC response PONG: Enr-seq: 2870, Ip: 124.251.22.13,  Port: 43754 PING: enr_seq: 276 Node: 0xea8d..7d61, addr: 184.82.210.230:9000
2025-02-10 11:01:40 DEBUG  Received RPC response PONG: Enr-seq: 8, Ip: 124.251.22.13,  Port: 51332 PING: enr_seq: 276 Node: 0xe762..af80, addr: 130.61.84.237:9000
2025-02-10 11:01:40 DEBUG  Received RPC response PONG: Enr-seq: 8978, Ip: 124.251.22.13,  Port: 16106 PING: enr_seq: 276 Node: 0xea63..199d, addr: 136.38.76.232:9000
2025-02-10 11:01:40 DEBUG  Received RPC response PONG: Enr-seq: 1736163570943, Ip: 124.251.22.13,  Port: 40341 PING: enr_seq: 276 Node: 0xea70..c0ff, addr: 185.209.176.61:27403
2025-02-10 11:01:40 DEBUG  Received RPC response PONG: Enr-seq: 34, Ip: 124.251.22.13,  Port: 37385 PING: enr_seq: 276 Node: 0xea1b..31b7, addr: 40.117.231.135:9000
2025-02-10 11:01:40 DEBUG  Received RPC response PONG: Enr-seq: 1719430811736, Ip: 124.251.22.13,  Port: 55887 PING: enr_seq: 276 Node: 0xeaca..ecca, addr: 75.144.254.101:30303
2025-02-10 11:01:40 DEBUG  Received RPC response PONG: Enr-seq: 938, Ip: 124.251.22.13,  Port: 5375 PING: enr_seq: 276 Node: 0xeb1c..da60, addr: 136.27.5.47:9105
2025-02-10 11:01:40 DEBUG  Received RPC response PONG: Enr-seq: 1720963819678, Ip: 124.251.22.13,  Port: 21979 PING: enr_seq: 276 Node: 0xe810..3766, addr: 3.8.4.18:30303
2025-02-10 11:01:40 DEBUG  Received RPC response PONG: Enr-seq: 1288, Ip: 124.251.22.13,  Port: 14960 PING: enr_seq: 276 Node: 0xeabc..11a9, addr: 74.201.216.109:9000
2025-02-10 11:01:40 DEBUG  Received RPC response PONG: Enr-seq: 4, Ip: 124.251.22.13,  Port: 17470 PING: enr_seq: 276 Node: 0xeb85..836b, addr: 44.222.172.44:9000
2025-02-10 11:01:40 DEBUG  Received RPC response PONG: Enr-seq: 930, Ip: 124.251.22.13,  Port: 31664 PING: enr_seq: 276 Node: 0xeae6..0ea9, addr: 46.4.159.90:9000
2025-02-10 11:01:40 DEBUG  Received RPC response PONG: Enr-seq: 1721289462507, Ip: 124.251.22.13,  Port: 24825 PING: enr_seq: 276 Node: 0xeb26..70e7, addr: 172.207.83.77:30303

Notice the IP address and ports being returned. If the port is fluctuating or changing (like in the above example) it means your router is using random external ports to send traffic. This means you need to add an SNAT rule, or manually specify your ENR as discussed in the configuration section.

Also notice that only one IPv6 address was returned. This is not based on your machine, rather the number of other peers on the network you have contacted that support IPv6. At the time of writing, there are more IPv4 nodes, so these are the most common kind of responses.

If both the IP address and ports are identical in the logs, this then indicates potentially a firewall issue. You should check your local node's firewall and router firewall to ensure traffic is allowed on the specific ports. You may need to add these yourself.

Libp2p Debugging

The Libp2p NAT ports can appear closed if the discovery ports are closed. This means we need to make sure the discovery port(s) are open before we can reliably debug the Libp2p ports. This is because we determine the port as being open or closed based on if we get incoming connections or not. As we will not get incoming connections if we are not advertising our node on discovery, the Libp2p ports can appear closed even if they are correctly configured in the network.

Assuming the discovery port(s) are open, it may take some time for peers to connect to our node on Libp2p for them to show the status of open. This can take a few minutes on mainnet. There is not a whole lot to debug here. If Lighthouse is marking these ports as closed, then we are not receiving incoming peers on the Libp2p ports. Ensure your Lighthouse node is listening on the correct interface (typically "0.0.0.0" or "::"). One way to do this is via:

netstat -ntlp

And checking to see if Lighthouse lists the ports expected. Double check ports have been forwarded in your router on TCP and UDP for the Libp2p ports. Typically if the discovery ports are registered as open, then the equivalent IP version Libp2p port should register as open. If not, you may need to check if any software can reach your internal node on these TCP ports with Lighthouse turned off. There are a few articles on how to check if a port is open, and I'll leave the exact details as out of scope for this post.

Further Help

If you've tried everything here and Lighthouse is still reporting that your NAT ports are closed, reach out to us on our discord.

We'll try and help if we can :).

Lighthouse NAT Configuration

Adrian Manning