[ BE 23.10 ... BE 24.10_7-amd64 ] Automatic fail-over to a Fallback Gateway still fails

[Manfred-Knick]

Important notices

Before you add a new report, we ask you kindly to acknowledge the following:

• I have read the contributing guide lines at https://github.com/opnsense/core/blob/master/CONTRIBUTING.md
• I am convinced that my issue is new after having checked both open and closed issues at https://github.com/opnsense/core/issues?q=is%3Aissue

Describe the bug

Loosing the primary connection [ M-Net Premium IP, "Dual Stack" IPv4 + (dynamic) IPv6 ] ,
automatic fail-over to another IPv4 Fallback Gateway fails.

Hint:

Although "Loss = 100%" and "Status=Offline",
the IPv6 part of the WAN interface does not get recognized as "defunct". <----- !

To Reproduce

Simplest: by un-plugging the DSL connection / MoDem cable.

Expected behavior

The lost primary IPv6 connection should not remain "active", but result into "defunct" proper

Describe alternatives you considered

Disabling the broken IPv6 in "System: Gateways: Configuration" does not help.

Possible work-around

A) Manually delete IPv6 default gateway via "route -6 delete -net default",
afterwards re-start the Fallback IPv4 Gateway, e.g.:
-> System: Gateways: Configuration,
. . . select Fallback Gateway -> Edit, -> Save, ->Apply
to allow configuration of its alternative default route.

B) Reboot

Confirmation

Re-plugging the DSL connection / MoDem cable properly re-instantiates the primary connection without any further intervention.

Additional context

Pre-decessor: #7335

Probably related: #5630

Environment

OPNsense Business Edition 24.10_7-amd64
Processor: Intel Haswell I3-4360T
Memory: 32 GiB
Network:
. Intel I218-V
. Intel I350-T2 v2
. Intel I350-T4 v2

[Manfred-Knick]

Background

Connection: FttB ; DSL into Flat ; "M-Net Premium IP" :
"Dual Stack" IPv4 + (dynamic) IPv6 :
ISP provides IPv6 prefixes only, but no static IPv6 interface address

Connections of this type have caused problems before;
these were addressed in #5630 :
Many thanks to @meyergru, @kevinchalet and @fichtner, the situation has definitely improved a lot !

Details

1. Setup WAN interface "MNET" :
   . "IPv6 Configuration Type" = "DHCPv6"
   . "Use IPv4 connectivity"
   . "Prefix delegation size" = 56
   . "Request prefix only"
   . "Send prefix hint"
   . "Assign prefix ID" = 0x10

2. Note: no other interface is configured as
   . "IPv6 Configuration Type" = "Track Interface" yet

3. Result:

   • IPv6 Gateway is setup
   • IPv6 Monitor IP := Gateway

4. ssh into the FW

5. Result:

   • ping -6 { IPv6 Gateway IP }
   • ping -6 to Provider-internal name servers
   • . . . dns01.mnet-online.de
   • . . . dns02.mnet-online.de
   • . . . ipv6-only.m-online.net <--- AAAA Records only
   • ping -6 to external addresses
   • . . . 2a0f:fc80:: ( = dns0.eu )
   • . . . 2620:fe::fe ( = Quad9 )
   • DNS -6 host lookup
   • . . . dns01.mnet-online.de, ipv6-only.m-online.net
   • . . . dns0.eu, Quad9, heise.de,

[x] Check: All of this is working as to be expected :-)

6. Un-plug the connection cable to the upstream DSL MoDem

7. Check -> System: Gateways: Configuration:

   • The IPv6 part of the WAN interface does not get recognized as "defunct",
   • the state is still "active"
   • it's higher priority prevents the Fallback Gateway from stepping in into first row of priorities being used

[Manfred-Knick]

Additionally enabling IPv6 for LAN:

. . . "IPv6 Configuration Type" = "Track Interface"
. . . "Assign prefix ID" = 0x11

Test Site Results

German Test Site: "wieistmeineip.de"

. . . Ihre IPv4-Adresse lautet: xxx.xxx.xxx.xxx
. . . Ihre IPv6-Adresse lautet: 2001:yyy:yyyy:yyyy:yyyy:yyyy:yyyy:yyyy
 
. . . Test IPv4: "OK"
. . . Test IPv6: "OK"
. . . Test Dual Stack: "OK"

Hope
that these details help to diagnose,
and perhaps others with a similar type of ISP connection for comparison during setup.

Kind regards
Manfred

[Manfred-Knick]

Version History

OPNsense 24.10 business edition is based on the OPNsense 24.7.6 community version.

Roadmap for 24.7 contained

. </> "Interfaces"
. . . "Interfaces: allow tracking the WAN itself in DHCPv6 mode *"

(*) pointing to above named #5630

as "Completed".

[Manfred-Knick]

Completely dis-abling IPv6:

. . . WAN interface "MNET" :
. . . . . . "IPv6 Configuration Type" = "DHCPv6"

Flint_GW (active) "111 (upstream)"
MNET_PPPOE "defunct (upstream)"
MNET_DHCP6 still exists "defunct (upstream)"

BUT:
ssh -> "netstat -r" : no default route has been created at all ! <--- !

Hint:
. . . "netstat -r" quickly shows IPv4 information,
. . . but (reproducibly) takes a long time to show IPv6 information.

Re-start the Fallback IPv4 Gateway results into proper fallback default IPv4 route.

REBOOT:
. proper fallback default IPv4 route

RE-CONNECT:
. re-creates main DSL connection with correct IPv4 default route
. MNET_DHCP6 still exists as "defunct (upstream)"
. "netstat -r" still takes a long time to show IPv6 information

DIS-CONNECT:
. same failure as above:
. . . no default route created
. . . re-start the Fallback IPv4 Gateway helps again
. . . "netstat -r" takes a long time to show IPv6 information again

RE-CONNECT:
. quickly re-creates main DSL connection with correct IPv4 default route

[Manfred-Knick]

In -> System: Settings: General,
a (priority) list of DNS servers is configured:
. primary connection:
. . . MNET_PPPOE --> IPv4 (p/s)
. . . MNET_PPPOE --> IPv6 (p/s)
. fallback connection:
. . . Flint_GW --> IPv4 (p/s)

Even after re-starting the Fallback IPv4 Gateway,
the corresponding DNS servers are not being taken into service!

Even ssh -> : "host ..." delivers, but "ping ..." fails

Although -> Services: Unbound DNS: General : "Enable Unbound"

[Manfred-Knick]

Probably related:
"Topic: Multi-WAN failover not bringing link back up properly"

[Manfred-Knick]

Further observations:

. -> System: Gateways: Configuration : Disable Failover Gateway:

• only results into status "pending"
• netstat -r : corresponding entry persists

. -> Interfaces: Disable Interface:

• Gateway status changes to "Malconfigured Gateway",
• but still does not change to "defunct", keeping its Priority
• netstat -r :
• • corresponding IPv4 entry finally removed
• • corresponding IPv6 entry still persists

.-> un-plug connection:

• dpinger: only one <- fallback-GW (correct)
• netstat -r : main IPv6 default persists, all same as before

[fichtner]

@Manfred-Knick I'm sorry to say it's very hard to follow your report. Can you break this down into the initial issue and the relevant logs? Otherwise I'm unable to follow.

[Manfred-Knick]

Hallo, Franco!

Kurzfassung, aktueller Stand:

Loosing the primary connection [ Dual Stack IPv4 + dynamic IPv6 ] ,
automatic fail-over to another IPv4 Fallback Gateway fails.

System: Gateways: Configuration:
.- Primary IPv4 default gateway [Prio 101] --> "defunct" (correct)
.- Primary IPv6 default gateway [Prio 102] --> not "defunct", but should be
.- Primary IPv6 default gateway [Prio 111] --> "active" (should not be)
Routes:
.- Primary IPv6 default still persists
.- Fallback IPv4 default not created

System: Log Files: General
2024-11-22T10:26:29 Notice opnsense-business /usr/local/etc/rc.newwanip: Failed to detect IP for interface wan
2024-11-22T10:25:23 Notice kernel <6>igb0_vlan40: link state changed to DOWN
2024-11-22T10:25:23 Notice kernel <6>igb0: link state changed to DOWN
2024-11-22T10:24:55 Notice syslog-ng Configuration reload finished;
2024-11-22T10:24:55 Notice syslog-ng Configuration reload request received, reloading configuration;

Mein Eindruck:
.- surviving Primary IPv6, with it's higher priority, prevents the Fallback Gateway from stepping in

Work-around:
.- von Konsole : " 11) Reload all services "

System: Gateways: Configuration:
.- unverändert, s.o.

Routes (correct now):
.- IPv4: default created -> Fallback GW
.- IPv6: no default route

Mein Verdacht:
.- evtl. Umfeld "pppoe plus DHCPv6 ohne eigene feste IP"?

Grüße aus einem frisch verschneiten München
Manfred