Regression in opnsense NAT/NAT-PMP with using tailscale on client devices behind opnsense router with CGNAT WAN

[mdedetrich]

Important notices

Before you add a new report, we ask you kindly to acknowledge the following:

• I have read the contributing guide lines at https://github.com/opnsense/core/blob/master/CONTRIBUTING.md
• I am convinced that my issue is new after having checked both open and closed issues at https://github.com/opnsense/core/issues?q=is%3Aissue

Describe the bug

A clear and concise description of what the bug is, including last known working version (if any).

Tip: to validate your setup was working with the previous version, use opnsense-revert (https://docs.opnsense.org/manual/opnsense_tools.html#opnsense-revert)

To Reproduce

Steps to reproduce the behavior:

1. Setup tailscale on another machine on an entirely separate network
2. Follow the guide at https://tailscale.com/kb/1097/install-opnsense#nat-pmp and install tailscale on a client machine behind opnsense
3. Try to connect to the previously setup tailscale network
4. Run tailscale ping node (this checks to see if your connection is working)
5. See error, specifically 2024/11/19 00:40:30 error looking up IP of "node": lookup node: no such host

Expected behavior

Tailscale works, i.e. its able to establish a proper connection.

Describe alternatives you considered

Pure IPv6 doesn't work (this was tried by disabling the IPv6 interface on my network interface). I used https://ipv6-test.com/ to test that my IPv6 connection is working purely correctly.

Note that aside from entirely skipping opnsense by hotspotting on my phone, there isn't any alternative that I have available to me. Switching ISP's is not an option as I only have a single ISP that offers high speed internet and they use CGNAT (and their cost for a static IPv4 address is insanely high).

I have also tried completely opening up (i.e. disabling) the firewall and it didn't make a difference.

Another thing to note is that my ISP's router in non bridge mode works with tailscale fine, this issue only started occurring when I put my router into bridge mode and put it behind an opnsense router.

Relevant log files

This is the last few relevant lines of tailscale netcheck

2024/11/19 09:31:50 portmap: [v1] Got PCP response: epoch: 31988
2024/11/19 09:31:50 portmap: PMP probe failed due result code: {OpCode:128 ResultCode:NetworkFailure SecondsSinceEpoch:31988 MappingValidSeconds:0 InternalPort:0 ExternalPort:0 PublicAddr:invalid IP}

Report:
	* UDP: true
	* IPv4: yes, <REDACTED>:41842
	* IPv6: yes, <REDACTED>:61493
	* MappingVariesByDestIP: false
	* PortMapping: PCP
	* CaptivePortal: false
	* Nearest DERP: London
	* DERP latency:
		- lhr: 162.8ms (London)
		- fra: 165.4ms (Frankfurt)
		- par: 167.2ms (Paris)
		- ams: 213.4ms (Amsterdam)
		- waw: 213.7ms (Warsaw)
		- mad: 213.8ms (Madrid)
		- nyc: 213.8ms (New York City)
		- ord: 216.4ms (Chicago)
		- mia: 216.4ms (Miami)
		- tor: 216.4ms (Toronto)
		- dfw: 232.6ms (Dallas)
		- den: 234.5ms (Denver)
		- blr: 264ms   (Bangalore)
		- dbi: 267.3ms (Dubai)
		- sfo: 317.5ms (San Francisco)
		- lax: 322.9ms (Los Angeles)
		- sea: 323.3ms (Seattle)
		- sin: 323.4ms (Singapore)
		- nai: 324.6ms (Nairobi)
		- hkg: 340.1ms (Hong Kong)
		- sao: 340.2ms (São Paulo)
		- hnl: 340.2ms (Honolulu)
		- jnb: 340.2ms (Johannesburg)
		- tok: 345.6ms (Tokyo)
		- syd: 390.5ms (Sydney)

Additional context

So the main context of this to read is at https://forum.tailscale.com/t/pfsense-nat-pmp-failures/2300/6 (I have the exact same error). When reading the whole thread, the overall impression is that

• This used to work with an earlier version of pfsense/opnsense (specifically pfsense 2.4.4). This was also when the earlier linked tailscale kb guide worked (see https://forum.tailscale.com/t/pfsense-nat-pmp-failures/2300/10). Note that the reason why this issue is being linked as a bug report is because according to that post (and the fact there is a knowledge base article on tailscale), it did work at one point. The user seems to be saying that the static IP port NAT outbound rules are no longer applying.
• There seems to be an issue with NAT-PNP being able to properly use the port. Even if randomizeClientPort on Tailscale's host side is set (which causes it to use multiple random ports instead of just 41641) it still doesn't work, see https://forum.tailscale.com/t/pfsense-nat-pmp-failures/2300/6
• The most pertinent thing to point out here is that the whole premise of tailscale is that its meant to "just work" with double-NAT's which is becoming increasingly common because of CGNAT's (a facet that will become increasingly common due to IPv4 address exhaustion). So while its a given that opnsense will need some configuration for tailscale to work since its a hard nat (i.e. enabling NAT-PMP, using fixed static port's in NAT outbound translation), it should just work once configured. As mentioned before, my ISP's router (before it was put into bridging mode) also had double-NAT and it worked fine with tailscale with zero configuration.

Environment

Software version used and hardware type if relevant, e.g.:

OPNsense 24.7.8-amd64
FreeBSD 14.1-RELEASE-p6
OpenSSL 3.0.15
Intel(R) X553 N NIC / Intel(R) I226-V
Intel Atom Denverton C3758
64GB ECC Memory

The following NAT outbound rules are applied

And UpNp is setup as follows

My devices are behind a Ubiquiti USW Pro 8 PoE switch but since its being used as a pure switch it shouldn't be making a difference. Incase its relevant (which I don't think it is), I do have a bridge setup (since I am using the multiple NIC's as a virtual switch) and that bridge is what is being assigned to LAN

[mdedetrich]

So I am going through tailscale open issues and it appears that this issue is somewhat common. According to tailscale/tailscale#10866 (comment), setting up a STUN server/port can help and it did make a difference with NAT-PMP but tailscale ping node still fails.