Routing traffic between netifs #537
Comments
|
Do you want to route (L3; i.e. on IP level?; still might be possible, will try to dig it out) or is L2 bridging also good? For the latter case, see this issue: #508 |
|
Ah unfortunately bridging seems to work only when the wifi is in AP mode though, probably not your use-case... |
Ideally on L3? But really I don't care so long as the receiving netif knows to automatically take L2 packets and forward them to its IP layer? I'm not sure about that. I'll look at everything you sent. On this issue and the other |
|
What I need is to be able to route/bridge my wireguard netif that's been created in the C layer with the ethernet "router" netif I created in rust. Ideally this would be done in rust but I'm starting to believe I'm going to have to make my own wrappers here. I would just like to know what functions I should be using for this, or if it's a lost cause, just a nudge really. |
let mut eth_netif = EspEth::wrap_all(
eth_driver,
EspNetif::new_with_conf(&NetifConfiguration {
flags: esp_netif_flags_ESP_NETIF_DHCP_SERVER | esp_netif_flags_ESP_NETIF_FLAG_AUTOUP,
got_ip_event_id: NonZeroU32::new(ip_event_t_IP_EVENT_ETH_GOT_IP as _),
lost_ip_event_id: NonZeroU32::new(ip_event_t_IP_EVENT_ETH_LOST_IP as _),
key: "ETH_DEF".try_into().unwrap(),
description: "eth".try_into().unwrap(),
route_priority: 50, // Higher is better
ip_configuration: Some(IpConfiguration::Client(IpClientConfiguration::Fixed(IpClientSettings {
ip: Ipv4Addr::new(10, 10, 10, 1),
subnet: Subnet {
gateway: Ipv4Addr::new(10, 10, 10, 1), // This is the gateway advertised by the dhcp server
// real_gateway: IpV4Addr::new(69, 69, 69, 69) // This would be the gateway this if routes its packets to
mask: Mask(30),
},
dns: None,
secondary_dns: None,
}))),
stack: NetifStack::Eth,
custom_mac: None,
})?,
)?;To recenter the problem, I need a way to differentiate the gateway that the dhcp server advertises to the network, and the gateway that the netif itself will use to connect to the internet. Here, that would be the ip of my wireguard netif. Otherwise I'm a bit stuck. |
OK now clear. But then your use case is definitely then routing and not bridging, in that you are creating a mini router (+ quite possibly, NAT). Also, I think for a proper router you absolutely need two netifs, and not one:
Now, honestly I'm not sure how to do the routing & NAT excercise. But there is something in ESP-IDF called "NAPT" which is badly documented, but used to work back in time in a now-archived old project of mine. I think this was a simplistic "NAT+router", but you need to google a bit on the Internet as to exactly what it was as I am actually having a hard time recollecting my memories. For one, I'm not sure why I don't have any code that connects two netifs together (the LAN and WAN one) when enabling NAPT. It could well be, that the two netifs in question are the Wifi STA and AP netifs, hence no need for any explicitness. But you probably want instead eth as LAN and Wifi STA as WAN (or another eth as WAN). |
|
Thanks i'll look all this up. I had another idea elsewise, though I don't know what's it's worth. What if i create the eth netif with gateway=ip as I did before, advertise that to the pc, then sneakily replace the netif with another that's got the gateway of what I need? So long as we don't say anything, the pc should be none the wiser, no? And have the right gateway to boot. The only issue I can think of is when the lease expires we're kinda screwed. And lwip only allows the lease to go up to an hour. |
|
The other idea was to go to L2 and try to bridge the two netifs there to bypass the whole DHCP+routing problem but I don't know how feasible that is. |
Bridging means a single ip for all netifs in the bridge which is not what you want? I don't think either this or the earlier hack would work. After all, you want to correctly pass packets between two different networks so you have to properly do routing. And moreover - likely nat too. |
I'm being stupid with my language, what I meant by "bridge" here is "make callbacks between the two netifs so that they forward packets to each other" |
Sure BUT:
Above might be possible but I wonder why not just giving NAPT a try? |
Oh I certainly will, I was just throwing ideas in the wind first. |
|
But wait so we have a wrapper to enable napt but not to do anything with it? Or am I blind? |
|
Well that clearly wasn't it. I'm kinda lost. I did this: log::info!("Enabling napt on eth netif..");
// Necessary for routing packets between subnets.
eth_netif.netif_mut().enable_napt(true);Now what? Nevermind wrappers, i don't even understand what C functions I have to use now that I enabled napt. |
You don't have to use any extra C functions. Just make sure you have another netif (wifi or eth) which is configured to your 67.67.67.67 network and the routing should just start happening automatically between the two the moment you call |
|
I can't clone the eth driver i'm wrapping to make the new ethernet netif. |
I do not understand what you are trying to achieve with that. Of course you can't and should not clone. And of course you can't and should not have multiple netifs assig ed to the same eth driver. If you explain what setup you are actually trying to achieve (say, with terminology as if you were using a regular pc networking) I might be able to help /suggest something, but otherwise I cannot explain to myself why you are not doing two drivers / netifs as I already suggested. As in your rmii eth + wifi sta. Or your rmii eth + an spi eth. Unless you use vlans, you anyway need two physical mediums for routing. |
|
Ok so what I have, is a setup like this: What I need is to be able to send packets from eth to wg0 and back. I don't really care how so long as they can speak to each other. |
|
I will know at runtime what the ip and subnet of the wgnetif is so it being variable should not be a problem, i just dont really know how to forward packets between the two netifs |
|
What is WgNetif in the PC world please? Tun? Tap? Something else? |
|
Wireguard virtual interface, so TUN. |
|
Then you might need something like this on the esp32 too. But what I don't get it is ... do you plan to run the wireguard vpn layer on the esp32 itself? Isn't it a bit underpowered for that? As in like 300kb ram (not counting psram) and relatively slowish? |
|
Hm, apparently it is possible, not sure how fast it is though: https://github.com/ciniml/WireGuard-ESP32-Arduino |
|
Separate from how the whole wireguard networking works (I need to read on that) you might have to tap into mbedtls or else it would be painfully slow. The mbedtls impl for esp idf have some if the algorithms implemented using hardware, so much faster (or to put it another way, not unbearably slow). |
|
But I think you are at a point where you need to explain your idea in more detail, as I'm at a loss as to what device you are trying to build. Like, is this some sort of wireguard vpn "device" and if yes, what is the point? |
|
The idea is to make a plug and play usb key that encapsulates both connectivity and vpn as transparently as possible for the user. Nothing must run on the computer. I've managed to get the tunnel to work but I don't understand how to make the wg netif speak to the eth netif so that the computer itself can speak to the internet through the vpn. |
|
Both the eth and the wifi drivers do have TX/RX callbacks through which you can push/pull packets (look at EthDriver and WifiDriver), but I'm not sure / don't remember whether you were supposed to push/pull whole ethernet frames, or just IP packets. Also, Line 1102 in 326a3e1 It also does have TX/RX so that you can push/pull packets. Whether you can still use the NAPT thing in this scenario I don't know. It seems to me it won't work by just trial and error. You probably need at least:
T.b.h. I've never done such a setup. Might be quite possible, but it would require learning (ESP IDF netif C code and LWIP C code) and experimentation I guess. |
|
Alright thanks for the info, I will look into it! |
|
So after a lot of research I found there's this in lwip: https://www.nongnu.org/lwip/2_1_x/group__bridgeif.html fn create_config() -> anyhow::Result<*mut esp_netif_config_t> {
let bridge_info = Box::new(bridgeif_config_t {
max_fdb_dyn_entries: 1,
max_fdb_sta_entries: 1,
max_ports: 2,
});
let inherent_config = Box::new(esp_netif_inherent_config {
flags: esp_netif_flags_ESP_NETIF_FLAG_IS_BRIDGE,
mac: [0x02, 0x00, 0x00, 0x00, 0x00, 0x01], //Needs to be unique, first 0x02 = LAA
ip_info: ptr::null(),
get_ip_event: 0,
lost_ip_event: 0,
if_key: CString::new("br0")?.into_raw(),
if_desc: CString::new("bridge")?.into_raw(),
route_prio: 30,
bridge_info: Box::into_raw(bridge_info),
});
let bridge_config = Box::new(esp_netif_config_t {
base: Box::into_raw(inherent_config),
driver: ptr::null_mut(),
stack: unsafe { _g_esp_netif_netstack_default_br },
});
Ok(Box::into_raw(bridge_config))
}
pub fn start(eth_netif: Arc<Mutex<EspEth<'static, RmiiEth>>>) -> anyhow::Result<*mut esp_netif_t> {
unsafe {
let bridge_handle = esp_netif_new(create_config()?);
let eth_handle: *mut esp_netif_t = eth_netif.lock().unwrap().netif_mut().handle();
let wg_handle: *mut netif = (*WG_CTX.lock().unwrap().0).netif;
esp!(esp_netif_bridge_add_port(bridge_handle, eth_handle))?;
esp!(esp_netif_bridge_add_port(bridge_handle, wg_handle))?; //Not working
Ok(bridge_handle)
}
} |
|
Hey - and putting aside your concrete issue - I thought it is clear that you need a router, not a bridge, as we kind of figured out relatively early in this thread? |
|
I've spend only 5 minutes browsing the internet as to how WireGuard works and also looking a bit into the LwIP WireGuard existing impl and here are my findings / hypothesis:
|
|
Hmmm
Actually... I take my words back. let eth_handle: *mut esp_netif_t = eth_netif.lock().unwrap().netif_mut().handle();
let wg_handle: *mut netif = (*WG_CTX.lock().unwrap().0).netif;
esp!(esp_netif_bridge_add_port(bridge_handle, eth_handle))?;
esp!(esp_netif_bridge_add_port(bridge_handle, wg_handle))?; //Not workingIt is not working because |
|
T.b.h. I'm not really sure where this "67.67.67.67" network you were mentioning earlier even fits into the picture?
That's it? |
|
Btw you can do the above purely with Wifi, as the Esp can operate simultaneously in an AP+STA mode. No need for Eth or anything like that. |
|
This is a very heavyweight way to avoid converting Line 970 in 2968fa1 You just need to create a regular This means you need to do adaptation of the LwIP Wireguard code, but at least this is solving the "netif"-to-"esp_netif_t" problem? |
Yeah that was merely an exemple i used to illustrate the problem, it has no basis in reality.
Should be what I want. I think. The http server on the side would still see the request anyway right? The request would just be duplicated and also sent to the bridge which would then drop it or do whatever, which we don't care about. (I think.)
I need eth because i'm building an usb key, I have a schematic that converts usb to eth frames that are interpreted by the esp32.
I'll look into it, thanks. My main gripe with the whole conversion thing was I don't want to have to go through the thousands of lines of wireguard C code I have in the lib to modify everything to be esp_netif_t. If there's a way to wrap it in the rust layer, i'm all for it, otherwise i'll bite the bullet. |
Clear now.
That's also exactly how I understand it (or rather - hope it would work).
Fair enough. And really a side topic. Risking the derail the conversation even further: there is this s3 chip which I think was able to act as a real USB host. Where I'm going with that is that maybe you can even do it without any schematics, by implementing the USB-to-Ethernet purely in software with that chip. But yes, side topic.
Got it. But then I really don't know how easy it would be to wrap a Speculating: the other thing is, if you plan to have this device for commercial usage, it might be that this wireguard C code would anyway has to be changed or even completely rewritten (and then better in Rust I guess) if it is not fast enough. |
Do you still have the link for that? I'm very interested.
Yeah, that sounds like pain. Part of the reason I dread the first option not working.
Thanks I'll look at it. I'm a student though, this is all just a torture project thought up by my professor to rate us as either "passable" or blights upon humanity. So this isn't gonna be commercialized anytime soon if ever. And optimizing data flow rates is something I'll look at if I manage to make everything work seamlessly. Because yeah, iirc, the lib I use for wireguard writes its own implementation of the chacha/poylwhatever crypto functions. So maybe hooking everything to mbedtls instead wouldn't be the worst idea. An idea for later. |
Example with ethernet over usb: https://github.com/espressif/esp-idf/tree/master/examples/peripherals/usb/device/tusb_ncm
Fair enough! |
|
Ok so this whole thing won't work. Because apparently this works purely on an L2 level (should've known, what with the name and all..) and mac stuff gets beheaded at the subnet boundary. So back to figuring out how the hell nat works.. |
If I can't get the "easy" nat implementation to work I'll give this a try. |
|
Can you remind me how and in what context napt is supposed to be used here? You said I just had to enable it and then the netif would magically speak to each other but there has to be some assumed restrictions to this yes? It seems quite magical that they would do this without specific routes being set too.. And isn't napt supposed to be to communicate with the outside? why are we even using this to speak between subnets? I'm so deep into it I don't even remmeber why I started this |
|
The problem was I need a way to move traffic from my 10.10.10.0/30 subnet that contains the PC and the esp's ETH netif to the let's say 50.50.50.0/30 subnet (completely random made up number we don't care) that contains my WG netif and the other guy's netif.
|
That's what I thought initially and that's why I was constantly throwing at you the fact that such a bridge is an L2 thing, but I no longer think a bridge will NOT work. I now think it WILL work. Imagine that you have:
Then you bridge these two ^^^ and assign to the bridge netif to be a Router with IP subnet 10.10.10.0/30 and have the 10.10.10.1 gateway. What do you achieve this way? A lot, in my opinion:
|
|
So your idea is something like eth <-> bridge <-> wg netif but I don't understand why the bridge has the ip/subnet it does? Currently the idea is that my ethernet netif can serve an ip to the pc, that won't be possible with this design will it? And so when we say bridge here we don't actually mean the weird IEEE L2 bridge provided by esp idf do we? It's just the term we use for "random netif we hijack to do our bidding" yeah? |
When you put multiple network interfaces in a bridge, they are assigned the same ip address and in fact all netifs in the bridge do get even the same mac. From the pov of the other network peers the bridge is a single entity. That's why bridges are l2.
That's exactly what I hope you can use. What is so weird about it? |
|
Ok leaving that aside for a second, if I am to do any of the above, I need to be able to wrap my (*mut netif) into a (*mut esp_netif_t), and, ideally do better than that, which is to create the netif in rust directly using EspNetif::new_with_conf. struct esp_netif_obj {
// default interface addresses
uint8_t mac[NETIF_MAX_HWADDR_LEN];
esp_netif_ip_info_t* ip_info;
esp_netif_ip_info_t* ip_info_old;
// lwip netif related
struct netif *lwip_netif;
err_t (*lwip_init_fn)(struct netif*);
esp_netif_recv_ret_t (*lwip_input_fn)(void *input_netif_handle, void *buffer, size_t len, void *eb);
void * netif_handle; // netif impl context (either vanilla lwip-netif or ppp_pcb)
netif_related_data_t *related_data; // holds additional data for specific netifs
#if ESP_DHCPS
dhcps_t *dhcps;
#endif
// io driver related
void* driver_handle;
esp_err_t (*driver_transmit)(void *h, void *buffer, size_t len);
esp_err_t (*driver_transmit_wrap)(void *h, void *buffer, size_t len, void *pbuf);
void (*driver_free_rx_buffer)(void *h, void* buffer);
// dhcp related
esp_netif_dhcp_status_t dhcpc_status;
esp_netif_dhcp_status_t dhcps_status;
bool timer_running;
// event translation
ip_event_t get_ip_event;
ip_event_t lost_ip_event;
// misc flags, types, keys, priority
esp_netif_flags_t flags;
char * hostname;
char * if_key;
char * if_desc;
int route_prio;
#if CONFIG_ESP_NETIF_BRIDGE_EN
// bridge configuration
uint16_t max_fdb_dyn_entries;
uint16_t max_fdb_sta_entries;
uint8_t max_ports;
#endif // CONFIG_ESP_NETIF_BRIDGE_EN
// mldv6 timer
bool mldv6_report_timer_started;
#ifdef CONFIG_ESP_NETIF_SET_DNS_PER_DEFAULT_NETIF
ip_addr_t dns[DNS_MAX_SERVERS];
#endif
};This is in components/esp_netif/lwip/esp_netif_lwip_internal.h; except what we see in the bindings.rs file is this: #[doc = " @brief Type of esp_netif_object server"]
#[repr(C)]
#[derive(Debug, Copy, Clone)]
pub struct esp_netif_obj {
_unused: [u8; 0],
}
pub type esp_netif_t = esp_netif_obj;That's it. And it gets it from components/esp_netif/include/esp_netif_types.h: /** @brief Type of esp_netif_object server */
struct esp_netif_obj;
typedef struct esp_netif_obj esp_netif_t;Is it just not importing any include files that aren't in the include folder of a component? In which case we have a big big problem because this is just not doable. Or maybe I'm stupid. You tell me. I hope I'm stupid, this is painful enough as it is. |
|
Ideally, I'd do something like |
|
Welp, from what I've seen there's exactly 0 ways to do this except by extending the bindings ourselves, which is stupid, so I guess I'm modifying the entirety of the wireguard implementation so that it takes in esp_netif_t instead of the raw lwip netif, everywhere. That's going to take a while. |
|
Actually I don't even know if it's possible. A lot of the functions used depend on having direct access to information that just isn't exposed with the wrapped esp_netif_t |
Hi,
I'm trying to wrap my head around how to route traffic from this netif to another? Since it's set as a dhcp router it doesn't seem like it's at all possible. I'd need to set the gateway of this netif to the ip of the other netif I want to route traffic through but that doesn't seem possible? The gateway here is only to set the ip of the netif itself inside its subnet from what I understand.
Is there a way to make the DHCP server run independently and have this configuration be a client instead?
The text was updated successfully, but these errors were encountered: