DHT and discovery behind NAT

[bvvinai]

Hi, I am new to libp2p and was working on a chat app. I still dont understand how the peer discovery should be done!

Setup:
I have a laptop running a node (behind a NAT) and an AWS Instance (with public IP) running another node.

1. The node in my laptop doesnt connect to bootstrap nodes when im on home wifi. But when I change to mobile data, it works.
2. Even if I connect through mobile data, I am not able to find my AWS peer with dht.FindPeer(). But if I use RoutingDiscovery with my app name as rendevouz string, I am able to find the peers. Is it how it should be done? Or dht.FindPeer() should work?
3. I think Auto Relay is enabled by default when you create the libp2p host. But still my node behind NAT is unable to join the network.
4. Are bootstrap nodes really necessary? What if I dont want a whole p2p network but just manually connect with my friends by exchanging QR codes? Is it possible without bootstrap nodes?

I have gone through a lot of docs and website, but couldnt get my head around it!

[Stebalien]

1. What bootstrap nodes did you configure?
2. dht.FindPeer should work as long as your AWS peer is a part of the DHT. Make sure you're calling it with the right peer ID.
3. Auto-relay isn't guaranteed to work, unfortunately. It can also take a few minutes to bootstrap.
4. You need to connect to at least one node with a known IP, that's how you find the rest of the network. But you can run that node yourself. Even better, to prevent your nodes from accidentally joining the "mainnet" DHT, you can change the DHT protocol prefix (construct the DHT with the ProtocolPrefix("/your-prefix") option).

Also note, if you're running this in a test scenario, you may need to wait a bit between each step to let your nodes bootstrap, etc.

[Stebalien]

I will definitely look into the protocol prefix!

Note, if you do use a protocol prefix, you'll need to run your own bootstrap node because you'll be creating a completely separate network.

[bvvinai]

There has been some progress. I am able to connect to my AWS instance from my laptop(with mobile data) with dht.FindPeer(), but not able to do vice versa.

The first 4 logs are the bootstrap nodes!
This is the logs of AWS instance :

{QmNnooDu7bfjPFoTZYxMNLWUQJyrVwtbZg5gBMjTezGAJN: [/dnsaddr/bootstrap.libp2p.io]}
{QmQCU2EcMqAqQPR2i9bChDtGNJchTbq5TbXJJ16u19uLTa: [/dnsaddr/bootstrap.libp2p.io]}
{QmbLHAnMoJPWSCR5Zhtx6BHJX9KiKNN6tpvbUcqanj75Nb: [/dnsaddr/bootstrap.libp2p.io]}
{QmcZf59bWwK5XFi76CZX8cbJ4BhTzzA3gU1ZjYZcYW3dwt: [/dnsaddr/bootstrap.libp2p.io]}
{QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ: [/ip4/104.131.131.82/tcp/4001]}
This node :  12D3KooWJ3sm2GZhwmmB28kz1Nd2JBwgg96grdrHVzoT7XKo2wGq
Finding peer :  12D3KooWQfBE9wUrCNvk81vw8a3vho8sBKG9DRoA9WwKSd9bUNGW
Could not find peer!
No connection established!

This is the logs in my laptop :

{QmNnooDu7bfjPFoTZYxMNLWUQJyrVwtbZg5gBMjTezGAJN: [/dnsaddr/bootstrap.libp2p.io]}
{QmQCU2EcMqAqQPR2i9bChDtGNJchTbq5TbXJJ16u19uLTa: [/dnsaddr/bootstrap.libp2p.io]}
{QmbLHAnMoJPWSCR5Zhtx6BHJX9KiKNN6tpvbUcqanj75Nb: [/dnsaddr/bootstrap.libp2p.io]}
{QmcZf59bWwK5XFi76CZX8cbJ4BhTzzA3gU1ZjYZcYW3dwt: [/dnsaddr/bootstrap.libp2p.io]}
{QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ: [/ip4/104.131.131.82/tcp/4001]}
This node :  12D3KooWQfBE9wUrCNvk81vw8a3vho8sBKG9DRoA9WwKSd9bUNGW
Finding peer :  12D3KooWJ3sm2GZhwmmB28kz1Nd2JBwgg96grdrHVzoT7XKo2wGq
Connected to node :  12D3KooWJ3sm2GZhwmmB28kz1Nd2JBwgg96grdrHVzoT7XKo2wGq

This is what I am getting when I connect from my home wifi (behind NAT) :

{QmNnooDu7bfjPFoTZYxMNLWUQJyrVwtbZg5gBMjTezGAJN: [/dnsaddr/bootstrap.libp2p.io]}
failed to dial: no good addresses
{QmQCU2EcMqAqQPR2i9bChDtGNJchTbq5TbXJJ16u19uLTa: [/dnsaddr/bootstrap.libp2p.io]}
failed to dial: no good addresses
{QmbLHAnMoJPWSCR5Zhtx6BHJX9KiKNN6tpvbUcqanj75Nb: [/dnsaddr/bootstrap.libp2p.io]}
failed to dial: no good addresses
{QmcZf59bWwK5XFi76CZX8cbJ4BhTzzA3gU1ZjYZcYW3dwt: [/dnsaddr/bootstrap.libp2p.io]}

I have attached my code for reference!

package main

import (
	"context"
	"fmt"

	badger "github.com/dgraph-io/badger/v4"
	"github.com/libp2p/go-libp2p"
	dht "github.com/libp2p/go-libp2p-kad-dht"
	"github.com/libp2p/go-libp2p/core/crypto"
	"github.com/libp2p/go-libp2p/core/host"
	"github.com/libp2p/go-libp2p/core/peer"

	drouting "github.com/libp2p/go-libp2p/p2p/discovery/routing"
	relayv2 "github.com/libp2p/go-libp2p/p2p/protocol/circuitv2/relay"
	"golang.org/x/crypto/bcrypt"
)

var hostNode host.Host
var idht *dht.IpfsDHT
var db *badger.DB
var appPeers <-chan peer.AddrInfo

func main() {

	initHost("username", "password")
	fmt.Println("This node : ", hostNode.ID())
	connectToPeer("12D3KooWJ3sm2GZhwmmB28kz1Nd2JBwgg96grdrHVzoT7XKo2wGq")

	select {}
}

func connectToPeer(peerid string) {

	fmt.Println("Finding peer : ", peerid)
	peerID, err := peer.Decode(peerid)
	if err != nil {
		panic(err)
	}
	peerAddrInfo, err := idht.FindPeer(context.Background(), peerID)
	if err != nil {
		fmt.Println("Could not find peer!")
	}

	connerr := hostNode.Connect(context.Background(), peerAddrInfo)
	if connerr != nil {
		fmt.Println("No connection established!")
	} else {
		fmt.Println("Connected to node : ", peerid)
	}
}

func initHost(username string, password string) {

	db, _ = badger.Open(badger.DefaultOptions("./badger"))

	var hostKey crypto.PrivKey
	get_err := db.View(func(txn *badger.Txn) error {
		item, err := txn.Get([]byte("priv"))
		if err != nil {
			return err
		}
		err = item.Value(func(val []byte) error {
			hostKey, err = crypto.UnmarshalPrivateKey(val)
			return nil
		})
		return err
	})

	if get_err != nil || hostKey == nil {
		fmt.Println(get_err)

		hashedPassword, err := bcrypt.GenerateFromPassword([]byte(password), bcrypt.DefaultCost)
		if err != nil {
			panic(err)
		}

		priv, _, err := crypto.GenerateKeyPair(
			crypto.Ed25519,
			-1,
		)
		if err != nil {
			panic(err)
		}

		hostKey = priv

		privBytes, err := crypto.MarshalPrivateKey(priv)
		if err != nil {
			panic(err)
		}

		txn := db.NewTransaction(true)
		defer txn.Discard()

		txn.Set([]byte("username"), []byte(username))
		txn.Set([]byte("password"), hashedPassword)
		txn.Set([]byte("priv"), privBytes)
		txn.Set([]byte("peerid"), []byte(hostNode.ID()))
		txn.Commit()
	}

	hostNode, _ = libp2p.New(
		libp2p.Identity(hostKey),
		libp2p.NATPortMap(),
		libp2p.EnableNATService(),
		libp2p.EnableRelayService(),
		libp2p.EnableRelay(),
	)

	idht, _ = dht.New(context.Background(), hostNode)
	if err := idht.Bootstrap(context.Background()); err != nil {
		panic(err)
	}

	_, err := relayv2.New(hostNode)
	if err != nil {
		panic(err)
	}

	for _, addr := range dht.DefaultBootstrapPeers {
		pi, _ := peer.AddrInfoFromP2pAddr(addr)
		fmt.Println(pi)
		err := hostNode.Connect(context.Background(), *pi)
		if err != nil {
			fmt.Println(err)
		}
	}

	dhtDiscovery := drouting.NewRoutingDiscovery(idht)
	go func() {
		for {
			_, err := dhtDiscovery.Advertise(context.Background(), "chatapp-bvvinai")
			if err != nil {
				fmt.Println("No peers discovered!")
			}

			appPeers, err = dhtDiscovery.FindPeers(context.Background(), "chatapp-bvvinai")
			if err != nil {
				fmt.Println("No peers discovered!")
			}
		}
	}()

}

[Stebalien]

So, one issue is that you should construct a "routed" host (a host that'll use the DHT for routing): Instead of constructing the DHT yourself, pass libp2p.Routing(YourDHTConstructorFunction) to the libp2p constructor. Alternatively, you can follow the example in examples/routed-echo (which does this manually).

That will let you call use host.Connect to directly connect your peers without having to do a manual DHT lookup. But I don't think that's your main issue.

This is what I am getting when I connect from my home wifi (behind NAT) :

This is confusing. It seems like you may have a DNS issue? Can you lookup the TXT record of _dnsaddr.bootstrap.libp2p.io (from your home wifi, ideally using a CLI tool like drill or dig).

This is the logs in my laptop :

From behind your NAT? Or are there two different cases here?

[bvvinai]

Yes, i have tried with libp2p.routing() as a parameter, but still there was no change.

This is my dig output:

; <<>> DiG 9.16.28 <<>> TXT _dnsaddr.bootstrap.libp2p.io
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 25270
;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;_dnsaddr.bootstrap.libp2p.io.  IN      TXT

;; ANSWER SECTION:
_dnsaddr.bootstrap.libp2p.io. 150 IN    TXT     "dnsaddr=/dnsaddr/am6.bootstrap.libp2p.io/p2p/QmbLHAnMoJPWSCR5Zhtx6BHJX9KiKNN6tpvbUcqanj75Nb"
_dnsaddr.bootstrap.libp2p.io. 150 IN    TXT     "dnsaddr=/dnsaddr/ny5.bootstrap.libp2p.io/p2p/QmQCU2EcMqAqQPR2i9bChDtGNJchTbq5TbXJJ16u19uLTa"
_dnsaddr.bootstrap.libp2p.io. 150 IN    TXT     "dnsaddr=/dnsaddr/sg1.bootstrap.libp2p.io/p2p/QmcZf59bWwK5XFi76CZX8cbJ4BhTzzA3gU1ZjYZcYW3dwt"
_dnsaddr.bootstrap.libp2p.io. 150 IN    TXT     "dnsaddr=/dnsaddr/sv15.bootstrap.libp2p.io/p2p/QmNnooDu7bfjPFoTZYxMNLWUQJyrVwtbZg5gBMjTezGAJN"

;; Query time: 0 msec
;; SERVER: 192.168.1.1#53(192.168.1.1)
;; WHEN: Wed Jun 05 09:26:46 India Standard Time 2024
;; MSG SIZE  rcvd: 474

The logs from my laptop was when I connected to my mobile data through my phone. But when I connect with my home router, it say "failed to dial: no good addresses"

[Stebalien]

1. Try pinging am6.bootstrap.libp2p.io. Does that work?
2. If you run nc am6.bootstrap.libp2p.io 4001 (you'll need to install netcat), you should see a string with /multistream/1.0.0 in it. Do you?

By the way, what version of libp2p are you using?

[bvvinai]

I am able to ping am6.bootstrap.libp2p.io :

Pinging am6.bootstrap.libp2p.io [147.75.87.27] with 32 bytes of data:
Reply from 147.75.87.27: bytes=32 time=164ms TTL=52
Reply from 147.75.87.27: bytes=32 time=163ms TTL=52
Reply from 147.75.87.27: bytes=32 time=163ms TTL=52
Reply from 147.75.87.27: bytes=32 time=164ms TTL=52

Ping statistics for 147.75.87.27:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 163ms, Maximum = 164ms, Average = 163ms

I am getting the output for "nc am6.bootstrap.libp2p.io 4001" :
/multistream/1.0.0

I am using the version v0.30.0

[Stebalien]

This really sounds like a DNS issue. Specifically, I believe the DNS records for those bootstrappers may be too big (doesn't fit into a single UDP packet).

Try drill TXT _dnsaddr.ny5.bootstrap.libp2p.io.

Also try explicitly setting your laptop's DNS provider to 1.1.1.1 (cloudflare). If that works, we know it's DNS.

[bvvinai]

When i tried drill TXT _dnsaddr.ny5.bootstrap.libp2p.io, i got :
Error: error sending query: Could not send or receive, because of network error

And when I change my dns to cloudflare, im getting the output :

;; ->>HEADER<<- opcode: QUERY, rcode: NOERROR, id: 21659
;; flags: qr rd ; QUERY: 1, ANSWER: 11, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;; _dnsaddr.ny5.bootstrap.libp2p.io.    IN      TXT

;; ANSWER SECTION:
_dnsaddr.ny5.bootstrap.libp2p.io.       0       IN      TXT     "dnsaddr=/dns4/ny5.bootstrap.libp2p.io/tcp/443/wss/p2p/QmQCU2EcMqAqQPR2i9bChDtGNJchTbq5TbXJJ16u19uLTa"
_dnsaddr.ny5.bootstrap.libp2p.io.       0       IN      TXT     "dnsaddr=/dns6/ny5.bootstrap.libp2p.io/tcp/443/wss/p2p/QmQCU2EcMqAqQPR2i9bChDtGNJchTbq5TbXJJ16u19uLTa"
_dnsaddr.ny5.bootstrap.libp2p.io.       0       IN      TXT     "dnsaddr=/ip4/136.144.51.25/tcp/4001/p2p/QmQCU2EcMqAqQPR2i9bChDtGNJchTbq5TbXJJ16u19uLTa"
_dnsaddr.ny5.bootstrap.libp2p.io.       0       IN      TXT     "dnsaddr=/ip4/136.144.51.25/udp/4001/quic-v1/p2p/QmQCU2EcMqAqQPR2i9bChDtGNJchTbq5TbXJJ16u19uLTa"
_dnsaddr.ny5.bootstrap.libp2p.io.       0       IN      TXT     "dnsaddr=/ip4/136.144.51.25/udp/4001/quic/p2p/QmQCU2EcMqAqQPR2i9bChDtGNJchTbq5TbXJJ16u19uLTa"
_dnsaddr.ny5.bootstrap.libp2p.io.       0       IN      TXT     "dnsaddr=/ip4/139.178.65.157/tcp/4001/p2p/QmQCU2EcMqAqQPR2i9bChDtGNJchTbq5TbXJJ16u19uLTa"
_dnsaddr.ny5.bootstrap.libp2p.io.       0       IN      TXT     "dnsaddr=/ip4/139.178.65.157/udp/4001/quic-v1/p2p/QmQCU2EcMqAqQPR2i9bChDtGNJchTbq5TbXJJ16u19uLTa"
_dnsaddr.ny5.bootstrap.libp2p.io.       0       IN      TXT     "dnsaddr=/ip4/139.178.65.157/udp/4001/quic/p2p/QmQCU2EcMqAqQPR2i9bChDtGNJchTbq5TbXJJ16u19uLTa"
_dnsaddr.ny5.bootstrap.libp2p.io.       0       IN      TXT     "dnsaddr=/ip6/2604:1380:45d2:8100::1/tcp/4001/p2p/QmQCU2EcMqAqQPR2i9bChDtGNJchTbq5TbXJJ16u19uLTa"
_dnsaddr.ny5.bootstrap.libp2p.io.       0       IN      TXT     "dnsaddr=/ip6/2604:1380:45d2:8100::1/udp/4001/quic-v1/p2p/QmQCU2EcMqAqQPR2i9bChDtGNJchTbq5TbXJJ16u19uLTa"
_dnsaddr.ny5.bootstrap.libp2p.io.       0       IN      TXT     "dnsaddr=/ip6/2604:1380:45d2:8100::1/udp/4001/quic/p2p/QmQCU2EcMqAqQPR2i9bChDtGNJchTbq5TbXJJ16u19uLTa"

;; AUTHORITY SECTION:

;; ADDITIONAL SECTION:

;; Query time: 22 msec
;; SERVER: 172.24.208.1
;; WHEN: Thu Jun  6 14:37:46 2024
;; MSG SIZE  rcvd: 1268

Now I am able to bootstrap through my home wifi and I am also able to connect to my AWS instance.

But is changing the DNS server feasible for all the peers? How would it work in a large scale scenario?

Also sometime i get this in my aws instance :

2024/06/06 09:49:16 failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details.

I manually increase the buffer size! Then it works. Is there any work around for this?

[Stebalien]

Wait:

I manually increase the buffer size! Then it works. Is there any work around for this?

You're saying that dialing your home node works if you manually increase your buffer size?

I have a new experiment.

1. Start your home node.
2. Go make a cup of coffee.
3. Then start the AWS instance and have it dial your home node.

I'm guessing waiting a bit and restarting the AWS node is what actually fixed it. That or just trying again.

[bvvinai]

Yes, the experiment worked. I reset my buffer size to default and tried it. Waiting a bit and restarting the AWS node was able to dial the home node. So, I guess setting the buffer size was of no use. What would be the reason for this?

After only after I changed my dns to cloudfare, I am able to dial my home node, else it never worked. Should every node change its dns to 1.1.1.1? What would be the possible solution?

And on top of this, there is NAT traversal and relay servers...I'm starting to think if I want to continue with the app or not! Pls guide me where I can find the latest documentation (preferably user friendly).

[Stebalien]

So:

1. Changing your DNS servers allowed you to resolve/connect to the bootstrappers. The libp2p devs are looking into a fix now.
2. You can ignore the UDP buffer size message, that's a red herring.
3. You had to wait a bit because it takes your home node a bit to connect into the network, discover its external address, discover a some relays, and advertise these relay addresses. If you try to connect to your home node to early, it may not yet have chosen/announced its relays.

Concretely, I'd modify your app to:

1. Repeatedly try to re-connect if connecting fails and/or the nodes disconnect (on an exponential backoff).
2. I'd remove the explicit relayv2 instantiation. Unless I'm mistaken, that shouldn't be necessary.

At this point, it sounds like ever

[bvvinai]

Thanks a lot for helping me out! The suggested modifications sounds good and will try it out.

[Stebalien]

Take a look at Kubo's "peering" module: https://github.com/ipfs/boxo/blob/625ba769263c2beeec934836f54bbd6624db945a/peering/peering.go. Really, I'd advise just importing it or stealing it wholesale. You can then configure both nodes to "peer" to each other and they will try to remain connected.