How to connect to a BIRDNETPI in the field via LTE router without public IP ?

[DD4WH]

I have a BirdNET-Pi in the field where there are many birds, where there is electricity, but no Internet.
So, I bought an LTE router with a cheap prepaid SIM card. However, I noticed, I am not given a public IP address. So, the standard way of bringing BirdNET-Pi into the internet did not work.
So, I set up vpnc on the Pi to connect it to my router at home via VPN.
My question is now: how do I manage to put the BirdNET-Pi (that is connected via VPN to my home network) into my home network, so I can view it over the browser? Is it really as simple as to enter the local IP into the Settings page, or do I have to take care of something else? Tomorrow I will have the chance to get into the field and install a brand-new version, so that's why I am asking 😀

UPDATE WITH THE FINAL SOLUTION HOW TO ACHIEVE THIS (Thanks to Patrick !):

Sharing the BirdNET-Pi over the internet is quite easy due to Patricks´ efforts in programming. So, principally, it is easy to access a remote BirdNET-Pi over the internet. However, my special problem is, that the internet connection of the BirdNET-Pi (in the field) is achieved by an LTE router with a SIM card. In my case this leads to a public IP issued by my internet service provider, that is shared with other internet users, which means I cannot use this IP adress for a webserver (like the BirdNET-Pi).

Solution:

• connect the BirdNET-Pi to the internet over the LTE router
• use vpnc to connect to my home router over VPN, so that the remote BirdNET-Pi receives a local IP in my home VPN
• then I can connect to the remote BirdNET-Pi page over the browser from every device in my home network

Workflow for the setup:
1.) Install vpnc on the RPi: sudo apt-get install vpnc
2.) Configure the conf-file: sudo nano /etc/vpnc/fritzbox.conf

IPSec gateway xxxxxxx.myfritz.net
IPSec ID your_VPN_name
IPSec secret xxxxxxxxxxxxxxxxxxxx
IKE Authmode psk
Xauth username xxxxxx
Xauth password xxxxxxxxxxxxxxxxxxxx
local port 0
DPD idle timeout (our side) 0

3.) Connect the VPN to test the setup: sudo vpnc fritzbox.conf
4.) Disconnect the VPN: sudo vpnc-disconnect
5.) Create a script that checks the VPN connection of the remote BirdNET-Pi and reconnects to the VPN, if it does not find the home router:
sudo nano ~/BirdNET-Pi/scripts/checkVPN.sh

#!/bin/bash
ping -c4 IP.OF.HOME.ROUTER > /dev/null
if [ $? != 0 ]
then
sudo vpnc fritzbox.conf
fi
exit

6.) make the script executable:
sudo chmod 755 ~/BirdNET-Pi/scripts/checkVPN.sh

7.) Create a link to the file:

• Navigate to directory /usr/local/bin
• while in this directory, execute:
  ln -s ~/BirdNET-Pi/scripts/checkVPN.sh checkVPN.sh

8.) Make the VPN check execute every 5 minutes:
sudo nano /etc/crontab

• in the crontab file, insert the following line:
  */5 * * * * pi /usr/local/bin/checkVPN.sh &> /dev/null

9.) Go into the settings of the BirdNET-Pi and enter the VPN IP adress, that your home router issues for the VPN-connected BirdNET-Pi

Now, the VPN connection is checked every 5 minutes and reestablished, if necessary.

At home, I can now access the remote BirdNET-Pi in the field by using the VPN IP of my BirdNET-Pi in my browser.

[mcguirepr89]

This answer is in regards to the sqlite branch, which will become the main branch eventually.

Is it really as simple as to enter the local IP into the Settings page, or do I have to take care of something else?

I'm pretty sure it is going to just be that simple. In the settings page, enter the IP address that the VPN provides, click "Update Settings" and the settings should be loaded within a few seconds. Check the site at http://IPADDRESS and everything should be working.

Let me know if you have any issues at all with this. 👍

But could you please send the output of the command below when you have access to that machine tomorrow?? I'm super super curious what the system has to say about not being issued a public IP address. (You can email this to me if you are worried about this information, but it shouldn't be too sensitive.)

echo -e "Public: $(curl -s4 ifconfig.co)\nLocal:$(hostname -I|awk '{print $1}')"

[mcguirepr89]

I like to use diagrams for scenarios like this because networking often needs them to elucidate what is really going on:

                    Where can I find my BirdNET-Pi?
                    ______________________________
                   |                              |
    ---------------|          (Internet)          |-----------
    |              |                              |          |
    |              | BirdNET-Pi IP=172.168.1.100  |          |
    |              |______________________________|          |
    |                                                        |
    |                                                        |
    |              ________________________________          |
    |             |                                |         |
    |             |             (VPN)              |         |
    |-------------|                                |---------|
    |             | BirdNET-Pi IP=10.123.12.100    |         |
    |             | Frank's Computer=10.123.12.101 |         |
    |             | Frank's Phone=10.123.12.102    |         |
    |             |________________________________|         |
    |                                                        |
 ______________________                  ________________________________
|                      |                |                                |
|     (LTE Modem)      |                |          (Frank's Home)        |
|                      |                |                                |
| BirdNET-Pi           |                | Frank's Computer=192.168.2.101 |
| Public=172.168.1.100 |                | BirdNET-Pi=172.168.1.100       |
| Local=192.168.0.100  |                |                                |
|______________________|                |________________________________|

The diagram above is a representation of how you would reach the BirdNET-Pi from different networking environments. You see that in its Local Area Network (the LTE Modem), it can be reached by its Local IP address and by its Public IP address. From the Internet, it can only be reached by its Public IP address. From your Home's network, it can only be reached via the Internet, again using its Public IP address. From within the Virtual Private Network, however, you can only reach the BirdNET-Pi from its VPN IP address. Always think of your VPN as distinct from whatever network is its host (your Home's router), as it is truly a separate network altogether.

So, to understand the nature of your question, the basic idea is that you will want to configure BirdNET-Pi based on the network environment from which you plan to access it.

Taking a look at the diagram above, you'll get the idea that there are (at least) 4 different network environments that you might want to consider:

1. LTE Modem, or Locally: You can use the Public OR Local IP address
2. Internet: You have to use the Public IP address
3. Home Network: You have to use the internet to reach its Public IP address
4. VPN: You have to use the IP address the BirdNET-Pi receives on that network

Ultimately, I would think you will configure the BirdNET-Pi's URL to the http://IPADDRESS given to the BirdNET-Pi from the VPN. It will only be accessible from other devices also connected to that VPN, but that should definitely work for you.

Let me know where I might be more clear and I will do my best answer any questions you have.

My best!

[gitalm]

Dear DD4WH,

i started a similiar project and would use GitHub and Observable. So you don't need to access the RPi, just to the internet.
https://observablehq.com/@gitalm/fetch-data-from-gist-raw-github-url

#163

Probably we can evolve this for a Out-in-the-Field Setup

Best wishes Johannes

[DD4WH]

Thanks a lot, Patrick, for your very detailed and well structured answer!

This makes it a lot clearer for me. I could identify all IPs depicted on your picture except for the two inside the box VPN. I can find the IP adress of the BirdNET-Pi over VPN (because that is the one I use for connecting via VNC), but I do not know how to find the IP adresses of my computer and my phone inside the VPN network. Maybe because my computer and my phone are not connected to that VPN network? But why would I have to connect my computer (that is inside my home network) to the same network again by using VPN? Same question for my phone . . .

But maybe there is a fundamental misunderstanding from my side:
As far as I understand: at the moment, the VPN service on the Pi (vpnc) is the VPN client and my home router is the VPN server (also taking care of the dynamic IP adress I am receiving from my cable company). Thus, my home router is the server, not my computer, and not the LTE router. Maybe this is what I am missing?

I will test your suggestion on the IP adresses in the field this afternoon. Maybe this also makes this network mesh clearer to me.

[DD4WH]

@gitalm Thanks Johannes, that looks like a nice solution for a field setup !

However, at the moment, my system is already reachable via VNC, so I can access it in the field and have a look at the results, the browser on the Pi and download files.

The aim of my recent approach is to really use ALL the features that Patrick built into the web interface and have the BirdNET-Pi running behind an LTE router with its full functionality. Still not sure whether this will be possible, but we will see.

From my point of view, your approach (automatic uploading of data from the BirdNET-Pi to a github page) could be valid for situations where remote access is not possible because of unstable mobile connection. But I also see the danger of much redundant work, because all the nice analyses, database functioning etc. etc. that Patrick built into BirdNET-Pi have been implemented once again by you on the github page. So, for my purposes this would not be an alternative, I think. But I will have a closer look and think more about this.

[gitalm]

@DD4WH and @mcguirepr89
You're absolutely right. The solution from Patrick is wonderful, very easy and is doing everything you can imagine.
Just put BirdNET-Pi on a RPi, connect a microphone and it is working delightfully.

My approach is somehow redundant, but i think necessary, if you put it somewhere out in the field with less power by solar and a not so good internet connection.
You just need for some time the connection to upload the data, but you can see it all times. This safes data cost and energy.

And i use a lot from the great work from Patrick, i could never do on my own, like the real time audio analyzing, converting and so on. So really Thanks for the great work and this is meant as an alternative and not as an competitive project.

[mcguirepr89]

This makes it a lot clearer for me. I could identify all IPs depicted on your picture except for the two inside the box VPN. I can find the IP adress of the BirdNET-Pi over VPN (because that is the one I use for connecting via VNC), but I do not know how to find the IP adresses of my computer and my phone inside the VPN network. Maybe because my computer and my phone are not connected to that VPN network? But why would I have to connect my computer (that is inside my home network) to the same network again by using VPN? Same question for my phone . . .

These are great questions, but unfortunately my response will have to make a few assumptions that could deem it useless in its entirety -- we'll see! Main assumption: You home router hosts the VPN on top of its LAN. Possible alternative (to be explored farther below): Your home router only uses the VPN as the LAN (unlikely).

Without knowing how your home router is really handling the VPN, here is the way that I would assume it functions based on my understanding of networking architecture and the basic needs of a VPN.

• You home receives 1 Public IP address from your Internet Service Provider and provides that IP address and the internet connectivity associated with that connection to your home's modem.
• Your home's modem hands that connection over to your home's router.
• Your home's router is then responsible for creating a Local Area Network. This LAN is distinct from the Internet and is not yet a VPN -- it is simply a list of IP addresses (non-physical), their corresponding MAC addresses (physical devices the IP points to), and some rules about how packets of information can (or cannot) travel between physical devices using that Address Resolution Protocol (ARP) table (the list of IP addresses and their corresponding MAC addresses).
• On the LAN, your home's router will issue devices their IP addresses and will take note of their MAC addresses so that it can handle moving the information packets around. The LAN uses its own ARP table to talk to devices and allow them to communicate with one another, meaning it is only concerned with the Local IP addresses that it issued to its client devices.
• Once it creates this LAN (nearly instantaneously), it will also create a Virtual Private Network that will "sit on top" of the LAN.
• The VPN does everything the LAN just did -- it creates a list of IP addresses (they will be different from the LAN IP addresses, even for the same physical devices), takes note of their corresponding MAC addresses, and handles the rules about how information packets can travel among its clients (not the LAN's clients).

NOTE: I would assume that your VPN is protected with login credentials. If you have to connect to your network (WiFi) AND connect to the VPN afterwards, then this confirms my assumption. IF, however, on your home network, you only connect to the network (WiFi) and do not also have to do any connecting to the VPN, then your home router utilizes the VPN as the LAN.

• Once the VPN is created, it is (usually) configured with a network bridge to its host network (your home's LAN). This is evident when you have internet access from the VPN, as it is using the LAN's internet connection (the router's internet connection via the modem).

Using your laptop connecting via WiFi as an example:
Before connecting to the home network (LAN), your network card (the physical thing that has the MAC address) will not have any IP address at all.
Frank's Wireless Network Card's IP= none
Once you connect to your home's WiFi, your home's router will issue a LAN IP address to the laptop and take note of the MAC address of the network card to which it just issued the IP address.
Frank's Wireless Network Card's IP= 192.168.1.100
When you connect to the VPN (or any VPN), the request is not like the request above. Instead of presenting the MAC address of the wireless network card to the Dynamic Host Configuration Protocol server on the router (the part of any network that issues IP addresses and other network configuration information) and receiving an IP address for that network card, your system creates a virtual network interface for this request. That virtual network interface is issued an IP address from the VPN. That virtual interface is distinct from the interface that received the IP address 192.168.1.100. Indeed, the VPN network interface is fully virtual, so it is not utilizing the standard interface for your wireless network card. This means that your laptop now has 2 IP addresses on 2 distinct network interfaces. Though they share some physical resources (the actual physical network card and the LAN's physical infrastructure for packet travel) the networks are completely isolated from one another and NO information packets can travel between the two (without explicitly configuring a bridge for that purpose).
Frank's Wireless Network Card's IP= 192.168.1.100
Frank's Virtual Wireless Network Interface= 10.182.9.100

My recommendation: Watch these IP addresses get issued and you will have a better understanding of how these networks commingle.
Press your Windows key and type "Command" to open the Windows command prompt. From there, starting out just as in the example above where your machine is NOT connected to any network, issue:

ifconfig

and just take a look at how the network card(s) look without any IP addresses (except you might see something like 127.0.0.1 or 127.0.1.1, which are the loopback interface, which is a tiny little local-only network computers create for themselves to allow disparate processes to share data).

Now, connect to the WiFi and issue ifconfig again and see the new IP address.
Finally, connect to the VPN and issue ifconfig and you should not only see the new IP address, but the new virtual interface that was not there previously. You'll see that you still have the network interface and IP address for the LAN, but now also have a new virtual device (usually called tun) that has the IP address from the VPN.

I'm hoping that the exercise above shows how the LAN and VPN are distinct networks, though they share some resources. This is important to understand well when trying to parse network visibility.

With all of that said, your home router may utilize the VPN as the LAN, but the architecture would still be the same -- a VPN needs a LAN's physical resources to operate, but it operates outside of the LAN entirely, as it creates virtual pathways for information and uses its own set of IP addresses for Address Resolution Protocol.

This same process can be followed on the Pi itself, but instead of issuing ifconfig, you'll issue ip a to get the same results.

Sorry for the epic response, but it is a complex topic and I still didn't really cover everything. (Fingers crossed that you can confirm that you also have to login to the VPN after connecting to WiFi, else most of what I've written will be going on under the hood, so to speak, and will not be observable.)

My best regards,
Patrick!

[DD4WH]

Thanks Patrick, that is a lot of stuff to digest and I think, I can now get a clearer understanding of what is going on!
I spent some time on trying to connect my home PC to the VPN of my home router. I think I managed to do it, however the router does not show my PC in the list of connected VPN devices. Hmm.

I will try some more and then I have to take a break and will come back tomorrow. Seems, network stuff is taking a long time :-).

[mcguirepr89]

I think I managed to do it, however the router does not show my PC in the list of connected VPN devices. Hmm.

When you get a chance while connected to the VPN as in your screen image above, run ifconfig from the Windows Command prompt to see whether your PC is aware of the connection as an interface

[DD4WH]

When I issue ipconfig in the Windows Command line, I can see the different IP adress for VPN: so that is working.

But now this phenomenon is funny to me:

• I am testing entirely in my home network:
• Birdnet-Pi is connected via Wifi to my router AND vpnc is activated --> my router shows the VPN IP adress for the BirdNET-Pi, so it is successfully connected to my home VPN
• the BirdnetPi URL in the settings is configured for this VPN IP adress of the BirdNET-Pi
• now I activate VPN connection of my PC (which is connected to my router via LAN and not by Wifi)
• on my PC, I cannot find BirdNET-Pi when I use the VPN IP adress of BirdNET-Pi
• if I disconnect the VPN connection of my PC, I can connect to the BirdNET-Pi using the VPN IP !
• also, the same for my Android phone
  Hmm, I will think more about that, also taking into account what you wrote in your post.

And if I interpret this correctly, it should be fairly easy in the field to:

• connect the BirdNET-Pi via LTE router to the internet
• connect to my home VPN
• enter the VPN IP adress into the Settings page
• and connect from the PC in my home network
  Why this did not work up to now, is probably caused by the very old version of BirdNET-Pi running in the field (older than 3 months)
  But I will test that in the field tomorrow, its already dark here ;-)

[mcguirepr89]

@DD4WH Frank, it sounds like your home router doesn't utilize the VPN as the LAN, but it does have a network bridge from the LAN to the VPN. In terms of how this would affect the representation of the network in diagrams, this would, in effect, place the "Home Network" and "VPN" boxes into one and the same box. Think of an actual bridge between the two -- the bridge allows info to travel freely from one network to the other, but on either side of the bridge, the networks are different. Regarding you not being able to see the BirdNET-Pi when connected to the VPN on your PC, this could be due to the network configuration of the VPN itself. There is a networking protocol (DHCP option) that is switched on or off for what is called "peer discovery," and that dictates whether or not peers on a network can communicate directly to one another (whether the router/gateway itself will release MAC address information for a client when its IP address is invoked). It is very likely (and common) that your VPN doesn't allow this type of peer discovery on the "VPN" side of the bridge, but that if you request VPN network information (i.e., try to visit a web server that is served within the VPN) while on the "Home Network" side of the bridge, it will allow that information to come through by virtue of the bridge, bypassing the strict rules that live on the "VPN" side of the bridge.

Confusing, I know.

[DD4WH]

Ah, now that makes sense! The router has its DHCP server option activated, but only for that IP range that is NOT VPN IPs. Local IPs are up to 192.168.178.199 and everything higher is given to VPN-connected devices.
There is also a security property "DNS-Rebind" on the router, which refuses all DNS-requests from devices in the same home network. Would that solve the problem, if I enter an exception IP here, ie. would that allow a VPN-connected Smartphone to peer at a VPN-connected LTE router? Probably not, as you explained that web servers on the VPN side can probably only be viewed from the home network side . . .
Thanks a lot, Patrick, for your help and all your effort!

[DD4WH]

Hi Patrick,

I am now facing a completely different problem:

• the script I use for checking whether the Pi is still connected to VPN is not functional any more, it is installed as a crontab -e job
• But there is nothing in the crontab file (except for my 5min checks whether the VPN is still connected), there used to be several BirdNET-Pi jobs in there, so it seems crontab is not being used any more at all by the Pi?

Second problem:

• start vpnc on boot
• I put the following into /etc/rc.local
• sleep 10 vpnc fritzbox.conf
  but it seems vpnc it is not activated at boot
• is rc.local used at all at boot now in the new version?

So, obviously there have been many many changes in the way BirdNET-Pi boots up and checks regular jobs, so I am a bit lost now.

[DD4WH]

Oh, even more problems:

• the file checkVPN.sh was created and edited with sudo nano /usr/local/bin/checkVPN.sh
• it contains the code shown in my previous post in the box with the green background
• I assigned rights with sudo chmod 4755 /usr/local/bin/checkVPN.sh
• But the file is not executed, although it should run every five minutes according to the crontab file . . .
• If I look at the folder /usr/local/bin, it all contains only "symbolic links" [not sure why], and my file is red. I am not sure what that means and why this is so.
  

[DD4WH]

OK, I now understood that the scripts are not inside the folder /usr/local/bin, although they are listed in that directory . . . that is weird. So they seem to be in BirdNET-Pi/scripts and in /usr/local/bin are only their symbolic links . . .
Will try to do the same with my script file checkVPN.sh

[mcguirepr89]

when looking at /usr/local/bin/ is it often helpful to use ls -l as this will show you where the links are and will show you the file permissions. You'll notice your permissions for checkVPN.sh are a little off. The issue here is the mode for the script -- you set the mode to 4755, which set the suid, setgid and "sticky bits". You don't need any of those things and will need them to be set to 0.

So fix your script's mode first:

sudo chmod 0755 ~/BirdNET-Pi/scripts/checkVPN.sh

then let's make sure the symbolic link is in place:

sudo ln -sf /home/pi/BirdNET-Pi/scripts/* /usr/local/bin

and then issue ls -l /usr/local/bin and your script should be green and not have issues running.

[DD4WH]

That was it, it works now and checks every 5 min.
Will make further tests.
Once the whole thing proves to be useful, I will do a complete writeup how this is done.

[DD4WH]

Oh, crossposted!
Thanks, Patrick for your help, that was exactly what was needed!

[DD4WH]

Just a short note:

It works now, I can access the "field station"-BirdNET-Pi webpage via VPN via LTE router inside my home network!

Tomorrow I will write up a workflow how to setup this.

Thanks again for your help, Patrick!

[mcguirepr89]

I'm very glad! I appreciate the work on your end getting a remote system working this way. 👍

[DD4WH]

Thanks Patrick ! 🥇
I have made an edit of the original post and put the final working solution there.
Also, I put a link into the old thread where I described a similar system.

[greiling]

A tipp for german users - or users installing in the field with expensive cellular tariffs - with a LTE router and a busy local network and just VPN active - I got up to 10 GB traffic within 12 hours active VPN connection.
Change the password of the FRITZ!Box VPN user if you want to get connected to your birdnetpi to the right password. Afterwards change to any other invalid password combination. Keeps your remote traffic low ;-)

[srd424]

I've been using https://github.com/fasmide/remotemoe for remote access to my instances, relatively simple set-up for those that are happy at the command line.