Please provide a simple captive portal example

[ballen4705]

I'm looking for a method for non-technical users to configure the WiFi on a headless RP-based device. When the RP first powers up, I'd like it to be in ad-hoc mode, so that a phone or computer can easily connect to it. After a connection is established, the user's phone or computer should bring up a browser with a web page, asking for networking info (or, alternatively, letting them leave the RP in ad-hoc mode). If networking info is provided, the RP should switch to that network, rebooting if needed.

Does the code in sdm-cportal provide this functionality? If so, I wanted to ask if you might add an additional "Quick start" paragraph at the end of the documentation here: https://github.com/gitbls/sdm/blob/master/Docs/Captive-Portal.md, to make it easy to try this. From reading the docs I think that the following might be sufficient, but am not sure if --nowpa is correct.

curl -L https://raw.githubusercontent.com/gitbls/sdm/master/EZsdmInstaller | bash

wget https://downloads.raspberrypi.org//raspios_lite_arm64/images/raspios_lite_arm64-2023-02-22/2023-02-21-raspios-bullseye-arm64-lite.img.xz

xz -d -v 2023-02-21-raspios-bullseye-arm64-lite.img.xz

sudo sdm --customize --loadlocal wifi --nowpa --restart --user myuser --password-user mypassword 2023-02-21-raspios-bullseye-arm64-lite.img

sudo sdm --burn /dev/sde --hostname mypi1 --expand-root 2023-02-21-raspios-bullseye-arm64-lite.img

[gitbls]

Thanks for your interest! While the captive portal does provide the functionality you described, it is a bit "rough" around the edges and I wouldn't recommend it as a solution for non-technical users.

Why? If the Pi only has a single WiFi (the typical case), the user is forced to take a manual step to reconnect to the ad-hoc WiFi after the test for correct credentials has completed. This step is not necessary for Pis that have two WiFi adapters on them, but that's the cadillac case. The single WiFi scenario can be confusing, so I'd encourage you to do some testing if you think it might work for your users.

LMK your thoughts on this...is the above completely off-putting for your usage, or do you think it might still work?

If it's not going to work for you, there is RaspAP that I believe does this (I've not tried it), and a few others. If you pursue other alternatives and find one that works well for you, LMK and I can look into providing an sdm plugin for it if that would be helpful for your usage.

Also, if all the users have iPhones, you could look into btwifiset, which lets the user connect to the Pi from an iPhone via Bluetooth to provide the WiFi credentials. sdm has a plugin for it, so it's quite easy to install the Pi end, and the iPhone app is in the app store. At the current time the btwifiset author has not provided an Android app.

[ballen4705]

Thanks for the fast reply. I am just about to test what I wrote above. Is --nowpa correct?

I am indeed in the Chevy case of only one WiFi. But I'm not so worried about asking the user to reconnect to the adhoc link. That info can be clearly displayed on a web page that the user is looking at just before the connection drops.

I did look at RaspAP, but it seems far too heavy. I don't want an entire access point, just a simple dialog to enter networking info.

I can't go with an iPhone-only solution, because all I can count on is that my users will have some sort of WiFi capable device with a browser.

If --nowpa is correct then I should be testing this later tonight.

[gitbls]

Yes, --nowpa tells sdm to not require a wpa_supplicant.conf. Alternatively, you could provide a "stub" wpa_supplicant.conf containing

country=US
ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
update_config=1

Of course, updating country= for your country. The only reason I'd suggest using a "stub" is so you can verify that sdm-cportal is actually running; it renames /etc/wpa_supplicant/wpa_supplicant.conf to /etc/wpa_supplicant/wpa_supplicant.conf.orig, but it should work fine without it.

One note: I haven't tested this in quite some time, so if you run into any problems/mis-behaviors, LMK and I'll check it out..

I didn't think that btwifiset would be a solution due to iphone-only, but mentioned it just in case.

[ballen4705]

I just did the testing with --nowpa, with raspian 64-bit lite. Results were:

• An adhoc sdm network was visible and I connected to it
• (Note: I had expected that after connection a browser would open and take me automatically to 10.1.1.1. This did NOT happen, so I had to navigate to the IP "by hand".)
• the configuration web page came up
• I entered SSID, password, and country, and proceeded. After about 30 seconds I reconnected to the network, and received a message giving me the IP address and telling me that it had worked. Specifically: "WiFi Setup Results
  Obtained IP Address 192.168.xxx.xxx via WiFi SSID 'lyyyyyy'
  Internet IS Accessible
• However I was unable to connect to the IP that had been given (ssh). If I ping it, there is also no response. However my network router does show that it issued an IP address via DHCP.
  So this is at least partly working ... any idea why I can't connect?

Cheers,
Bruce

PS: if the network connection succeeds, why not switch entirely to that new IP address, and redirect the browser there? You can assume that after the RP drops the adhoc network connection, the user device will connect itself to the same network that the RP is on

[gitbls]

I researched how to get the device to go to the target IP address when I built this, but at the time was unable to find a technique that worked. Probably worth revisiting this.

I'll have to set up a test system to see how it behaves for me. Will try to get to it later today (PDT here).

[gitbls]

Sorry, honey-do list consumed almost all available time other than getting a system set up for testing. Tomorrow...

[ballen4705]

No worries, thank you for the update.

I have been mulling this over. For my device, the following would be ideal. I plan to have a single recessed "factory reset" button. After that is pushed, or when the device is delivered, the network will be adhoc, with no WiFi password. After connecting to that, the users browser should be directed to the web page for the device. The device does not require an internet connection, and may be used in adhoc mode. Alternatively, the web page for the device will also offer an option to input WiFi credentials (SSID, passphrase, country) and if needed could reboot itself after that is selected. Then it would use the general-purpose network and have an internet connection. Providing these two options in this way will simplify matters for me.

I have a constructive suggestion about one point which we discussed previously.

After connecting to the RP's adhoc network, we would like the user's cell phone or laptop to automatically open a browser pointed to the RP's web page. Three ways to do this are documented here: https://en.wikipedia.org/wiki/Captive_portal under "Implementation". The first is:

(1) When a modern, Internet-enabled device first connects to a network, it sends out an HTTP request to a detection URL predefined by its vendor and expects an HTTP status code 200 OK or 204 No Content. If the device receives a HTTP 200 status code, it assumes it has unlimited internet access. Captive portal prompts are displayed when you are able to manipulate this first HTTP message to return a HTTP status code of 302 (redirect) to the captive portal of your choice.[6][7] RFC 6585 specifies 511 Network Authentication Required code.

Methods (2) and (3) involve ICMP and DNS redirects, and are probably not as good as (1).

So: make sure that your server's first response to any HTTP request on the adhoc network is HTTP status 302 and a redirect. References [6] and [7] above, from the Wikipedia article, provide more information. They are articles written by Andrew Wippler in 2016 and 2017, which describe this in detail. For example, the required responses for iOS and non-iOS devices are shown here:

https://andrewwippler.com/2017/04/07/captive-portal-overview/

[gitbls]

Thanks for the info on captive portals, it will be helpful. I (re-)discovered last night that I had done some additional work on this over a year ago, but put it aside. There are also some important code improvements that I never completed.

I'm going to spend today updating sdm-cportal with those changes. Once that's done I'll look into how to make the captive portal automatically pop.

While I do have time to work on this, it could very well take several days, and possibly up to a week or two to get it working reliably. How does this align with your goals/plans?

[ballen4705]

Actually a few days to a week would be good for me. I need to set up a test/development system with a head and perhaps also a hardwired ethernet network connection. It is otherwise too painful to do the development work. So, if you are willing, why don't we use this thread to coordinate? Let me know when you have pushed stuff to GIT that I should try/test. I'll then try it, and give you feedback.

[ballen4705]

My goal is a system that has no keyboard, no monitor, just a WLAN connection (RP CM4). I want it to boot up and create an adhoc network, which can either be used "as is", or alternatively be used to configure the system to connect to an existing WiFi network, which can then be used for access.

I connected a monitor and keyboard to the system which I created using the five commands in my first post above, and found the boot stalled in a query asking me to confirm the user "pi" and then enter a password. After completing this form, I was then able to connect to sdm and fill out the form, but never got a network connection to the wifi network, in spite of the console reporting an IP address correctly obtained from DHCP and that 'Internet is Accessible'. Immediately after that it re-enabled access for the sdm network. The problem may be that when the sdm connection dropped, my cell phone immediately reconnected to the normal WLAN network here. So when the sdm connection came back up, and I reconnected to it, I wasn't able to get back to the correct test page or dialog.

My suggestion: please give me the exact command lines to generate a boot image, perhaps modified from the five commands in my first post above. This will ensure that we have identical starting points for comparison. I'll then test that.

Cheers,
Bruce

[ballen4705]

PS: I think I understand the basic issue. Let's suppose that I am using my cell phone to configure the device. My cell phone first connects to 10.1.1.1 where I enter the network info. The device then drops the connection because it is connecting to the network that I have specified, to get its IP from dhcp. At that point, my cell also drops the connection, since the 10.1.1.1 network no longer exists. The device then establishes the 10.1.1.1 network again, to which I must reconnect manually, in order to learn the device's new IP address. I don't see any way around this with clever redirection.

What I would suggest is that the device, having obtained a valid IP via DHCP, now operates under the assumption that this will be a valid IP in the future. So after the user reconnects to 10.1.1.1, they find a DIFFERENT web page. This new web page displays the IP obtained from DHCP, and provides a clickable link that will (a) redirect the browser to that new IP and at the same time (b) reset the network connection to the DHCP-obtained IP. Hopefully this will then cause the browser to also shift to this new IP when the cell phone reconnects to the public network.

[gitbls]

Not ignoring you, dealing with a few life things and reacquainting myself with the sdm-cportal code. Will get back to you here once I've got things running and remember how this all works 🤣

[ballen4705]

No worries, I am not in a hurry.

I have found a captive portal example which runs on my RP4 and successfully does the redirect of my iPhone. Also works without a WiFi password if desired. You can reproduce this in very short time. Follow the instructions here:

https://github.com/Splines/raspi-captive-portal
and in particular look at the Issue Report that I filed there for a complete 'howto' transcript.

I prefer your compact python-based web server solution to the java one used by Splines. So if you can get this working with sdm it would be nice.

[gitbls]

Thanks for that. I've looked at a few captive portals, and have also been reading Apple and Android docs on all this. I have a few ideas to research and try out to figure out what works.

Like you, I want a simple solution for this, but it seems that the client OS (Apple, Android, and probably Windows) are (rightfully) tightening up their security requirements which of course adds complexity.

So, nothing concrete to report yet, other than simply setting DHCP option 114 (Captive portal URL) doesn't work with Apple. iOS (16.4.1a) doesn't seem to honor it at all, instead opting for captive.apple.com and a few others. Lots to investigate 😲

[ballen4705]

I have been thinking about "how this should work" and can think of only two choices. The first choice is more or less what you have currently described. The second choice makes use of 'Zero Configuration Network' protocols. I am not sure if it would work, but if it does, I think it would be better.

(1)
user attaches to AdHoc network, is redirected, fills in form with SSID/PW
RP drops AdHoc network, connects to WLAN, gets IP
RP drops WLAN connection, and creates AdHoc network again, but remembers and keeps its WLAN IP
user attaches to AdHoc network a second time, but this time receives only the info about what the new WLAN IP will be
user checks box on RP to confirm, "I have this info"
RP drops AdHoc connection, connects to WLAN with the IP previously obtained.
user must now navigate to that IP
On any future boots, RP will go directly to the WLAN and not establish an AdHoc network

This approach has no timers. It requires a hardware reset button to tell RP to forget the WLAN credentials and start an AdHoc network again.

(2)
(a) User attaches to AdHoc network, is redirected, fills in form with SSID/PW.

(b) The form also asks what NAME should be given to the device, and proposes a name like "RP472" where "472" is randomly selected.

(c) RP drops AdHoc network, connects to WLAN, gets IP.

(d) RP uses Multicast DNS (MDNS) to inform the WLAN network "I should be reachable under the name RP472". One can also use DNS-SD for this, I think.

(e) User goes to WLAN network. Even though the user does not know the IP of the RP, the RP will be reachable under the name RP472.

(f) The RP only starts an AdHoc network again if step (c) fails or if the user toggles a 'hardware reset' button.

Possible improvement, just before step (c), RP sends a redirect to user device, directing it to the URL "RP472.local". This eliminates step (e) for the user, since they are automatically redirected.

[gitbls]

There are other techniques, including one that was posted about 2 years ago in the Pi Forums. I'm looking at that one in more detail, and hopefully it will be suitable to use, although I need to prototype it. This method could be a lot simpler.

Besides this, there is the issue of needing to use HTTPS instead of HTTP for the redirected form. Apple requires HTTPS, so this is mandatory.

[gitbls]

Making good progress (ie mostly running) on the new AP technique, which looks like it will keep the AP up while it's testing the SSID/password. This will greatly simplify the end-user experience. There's a fair amount of code cleanup to do since this required ripping out the old NIC handling code and trying various approaches to get it working AND fit with the rest of the code.

Still need to sort out https so that the captive portal site pop works. I think it's doable, but I have a nagging concern about the cert for this. We'll see...

[ballen4705]

I'm excited that it's coming along! Let me know when you have something that you want me to test. You can make a testing branch in GIT that I can pull, if you want.

Making an https server similar to your existing http server should not be hard. See here for a simple example: https://anvileight.com/blog/posts/simple-python-http-server/#:~:text=The%20standard%20Python%20library%20has,minimalistic%20HTTP%2FHTTPS%20web%20server.

Note that the latest Bullseye releases are moving away from the old networking setup and towards a "new" tool called Network Manager. The command line version of this called nmcli. This might offer a simpler (and in the future, better supported and more standard) way to carry out the networking configuration and reconfiguration. According to the docs, only a couple of lines are required to configure and bring up a hotspot.

[gitbls]

I'm glad you think that an https server shouldn't be hard. I've found that to be the case as well, but I previously mentioned that I have a nagging concern about the cert. You didn't ask why I was concerned, but I'll tell you anyhow. Apple is (rightly so) tightening up the whole system to reduce hacking exposure surfaces.

I previously mentioned that HTTPS is mandatory. A captive portal that sends the portal address with an HTTP instead of HTTPS is summarily ignored by iOS, based on my testing.

So, need to use HTTPS. But still uncertain how rigorously iOS checks the cert. If it needs to be a signed cert, this gets much more complex and less friendly to set up by a large swath of people.

As far as Network Manager, I'm very aware of that, of course. I didn't do a deep dive on it yet but I'm pretty sure that NM doesn't do the type of AP that I'm working with now. Besides, it still has to work with dhcpcd unless/until it's removed from RasPiOS. Ultimately, sdm-cportal will have to support both dhcpcd and NM (and maybe dhclient as well).

[ballen4705]

You're correct -- I had not realised that a self-signed cert might be problematic.

I don't know enough about Network Manager (NM) to understand its capabilities. But I hope that (at least eventually) these capabilities become part of its standard portfolio, because they are widely used and widely needed.

[ballen4705]

Any news? Has the iOS requirement for a "proper chain of trust" certificate blocked the way forward?

[gitbls]

My focus has been on getting the wifi to work reliably, and unfortunately, still not there. Have not even started to look at the automatic captive portal popup. The virtual adapter (technique I'm using) mostly works, but sometimes drops, and/or logs other curious errors.

That said, the current portal generally works without automatic pop, altho sometimes requires a manual reconnect from my phone in the middle of the process. But sometimes it works perfectly.

Although I haven't done enough research yet, I'm half-convinced that the Broadcom wifi driver might have an issue with the virtual network adapter, so need to reserarch that before further before posting. If I don't find a solution, I'm back of the mind hoping that someone on the RasPiOS team takes an interest in it with respect to chasing problems from the wifi adapter side of things, since this code provides an easy, standalone repro.

My current thinking is to do some code cleanup on it and post it for you to try, and also to see if anyone in the Pi Forums takes an interest and has any ideas about addressing the wifi issues.

Lots of stuff working:

• Network and hotspot configuration and post-use teardown
• Web page serving and delivery
• SSID/password testing and reporting success/failure back to client
• Logging

We're starting a stretch of beautiful Seattle weather, so progress may be a bit slower 🤔

[ballen4705]

I'd be happy to test. If you want to keep it semi-private, then you could push it to a branch.

FWIW, here in Northern Germany the weather has also gotten nice (:-)

[gitbls]

The portal is working reliably now.

In the interest of moving things forward, I've decided to leave it with non-automatic portal web page pop, so I should have something ready to post in the next couple of days after I add LED flashing (for status), finish some code cleanups, and do some additional testing.

[ballen4705]

OK, looking forward to trying it.

[gitbls]

I posted the new version in the master tree, since it's so much better than what was there IMHO.

There are a lot of notes at the end of the file that will eventually turn into better documentation once I've completed this.

LMK how your testing goes, and I highly recommend a defaults file for that 👍

[gitbls]

The SSID/password picking code had a very fatal flaw in it, now fixed. Now that you've found my typo 🤣 you might want to look at using --defaults so you don't have to deal with that form regularly. I obviously used --defaults one too many times (ok, every time), or I would have caught the problem you uncovered.

Thanks for the script. Will take a look at it to see if there are any concepts that work with dhcpcd, and whether I can use it with network manager.

[ballen4705]

Will test again in a few hours with and without --default and post again here.

Please try running the enable option of my script, to see if iOS devices connect to the AdHoc network without complaining about "no internet access". What I have observed is that they have no internet access but do not complain about it anymore. IMO this is very desirable.

[gitbls]

I looked at your script and did some NM reading. Here's my thinking/plan on NM.

• The portal uses systemd-networkd to control wlan0 and sdm0. sdm0 (default name) is a STA/AP network virtual device created by adding the __ap virtual interface to the wifi phy. This effectively enables 2 simultaneous, independently-managed wifi devices on a single pi. One that connects to the local wifi network (wlan0 for testing ssid/password, and when done for final connection, obviously), and one (sdm0) that is the access point. Both of these exist at once. sd-nd has a couple of really nice features that are important for the portal:
  • sd-nd provides a super-lightweight DHCP server. No muss no fuss, all for one line in a .network file.
  • Message logging is lightweight. I spent a lot of time eliminating order-related messages. It's very clean now.
  • Easy to figure out what's going on
• The portal needs to have fine-grain manual control over the state of the wpa_supplicants for each device (wlan0, sdm0)
• When it's time to test the SSID/Password wpa_supplicant@wlan0 is reconfigured/restarted. In theory sdm0 should not drop, but at least for me, it does. I presume these are wifi firmware bugs, and will report them soon.
• The testing code restarts wpa_supplicant@wlan0 and checks for a successful IP address obtained. Once obtained, internet connectivity is tested, and then results can be returned.
• When the Test Done page has completed, the portal resets the networking to the way it was.

In other words, to switch the portal to using network manager to implement the portal itself would be a complete rewrite. It would also result in a tremendous amount of log file spew (NM is much more verbose than systemd-networkd). I did find a note in the NM docs about using STA/AP concurrency, which is what the portal is already using.

I see no value in reimplementing the portal to use NM with it's system log message bloat "just to use NM"

That said, the portal currently only "supports" dhcpcd, which in this case means that it assumes dhcpcd is the running dhcp client service on the system, and stops/starts it, and reconfigures it as needed to ensure that the portal operates correctly. This code needs to be extended to support "the running dhcp client server", whichever it is. This is on the list at the end of the source: "# Other dhcp clients such as Network Manager".

So, the NM plan is to keep the current implementation, extended to support an active network manager as just mentioned.

[ballen4705]

Just to be clear, I started experimenting with NM because it appeared much easier to create an AdHoc network, and to switch the RP back and forth between that network and a conventional one. Indeed, as my script demonstrates, it is simple to do.

The reason I am calling it to your attention is because, when iOS devices connect to the AdHoc network that NM creates, the iOS devices do NOT display the "no internet connection" warning message. This is the case even though there is, in fact, no internet connection.

The reason must be in some aspect of how NM configures the AdHoc network. If we could identify that aspect, it could probably be easily reproduced in your code, and would eliminate the horrible "no internet connection" warning message. This might also eliminate the fickle behaviour that follows, as iOS loses interest in the connection and shifts to another network link that does offer an internet connection.

[gitbls]

Please update your script to use STA/AP networking and see if "no internet connection" goes away. This is very different than the single-threaded usage of the wifi device your script currently does.

Edit: The reason for using STA/AP mode rather than switching wlan0 between AP and STA mode is to avoid having to take the adhoc network offline to test the SSID/password. If the STA/AP method is causing iOS to drop offline from the AP, I believe that's a bug in the networking stack, and I'm not ready to go rip all this up again until I take a run at getting STA/AP fixed.

In the meantime, there is plenty of other testing to do, I have a few additional features to add, so at this point putting this particular issue (AP/STA vs switching a singular network back and forth) on the back burner to concentrate on this other stuff.

[ballen4705]

(0) Starting with fresh bullseye 64-bit lite install
(1) I am now able to enter the SSID and WiFi password
(2) After cycling the connection on my iPhone, I can navigate to the
"WiFi Configuration Results" page. This reports that the RP
obtained IP address 192.168.123.79
(3) However ifconfig shows that the RP actually has IP address 192.168.123.185
(4) I think the IP obtained in (2) is for the virtual device which has a self-assigned MAC
address that differs from that of the physical wlan0 device, whereas
(5) The IP obtained in (3) assigned to the MAC of the physical wlan0 device

(My vanilla home router is assigning the 192.168.123.xxx addresses via DHCP by hashing the MAC address.)

I would be happy to try to update my script as you suggest, but have not been able to figure out how to do this. Is the idea to create a virtual wireless device that can simultaneously act as an AdHoc network with IP 10.1.1.1 while also connecting to an external WiFi network as (say) 192.169.123.185? Meaning that the physical WiFi device on board the RP is operating on two different networks at the same time?

[gitbls]

I've not seen that at all. If you want me to have a look at it, please do the following:

• Reboot the Pi
• Run the captive portal and step through it with your Phone
• After the portal completes, issue the following command:

sudo journalctl -b | sed -n '/Start Captive Portal/,/Captive Portal Complete/p' > cplog.txt

and post the cplog.txt file here.

JFYI, the CP checks the IP address obtained with: s = gocmd(f"ip -br -o -f inet addr show dev {self.pd.netdev}") , where self.pd.netdev is the name of the wireless device (currently hardwired to 'wlan0'). In any event, the above log segment will disclose what's going on.

You can also grep Command: cplog.txt to see all the commands that the Captive Portal uses. That, along with section 1.4 on this page describes what needs to be done as hints to help you update your script. And yes, the idea is as you described.

BTW, I read somewhere that the just-released iOS 16.5 might have improved WiFi handling. I haven't tested it yet.

Lastly, what do you mean by "hashing the MAC address"? Typically, routers assign IP addresses based on the MAC address of the requesting station, so what does "hashing" do?

[gitbls]

Two quick updates. 1) Confirmed that iOS 16.5 maintains the WiFi connection better than 16.4 did (for me, at least). It still sometimes disconnects when the ssid/password test starts. I haven't looked into this in detail to see if there's a way to ameliorate.

2. I fixed the portal so that it doesn't use dhcpcd or network manager at all. It only needs to stop them if they are started, and when all done, restore prior state. So, as before, only systemd-networkd is involved in the portal, but a couple of operations on dhcpcd (and probably more that would have been required for network manager) are eliminated, which is good!

[gitbls]

I'm on the path to completing this round of updates to the captive portal. It's working quite reliably, and can handle either dhcpcd or network manager (meaning, shuts them off while the captive portal is running, as the portal uses systemd-networkd to control the WiFi device).

If you are keen to try it sooner rather than later, LMK and I'll drop an almost-final update for you.

If not, it will be another week or so to finish the code updates and update the documentation.

[ballen4705]

Congratulations, and thank you very much!

I'm snowed under at work for the next few days, but starting in about a week I'll have time to try it out. So if you have not posted it by then, I'll ping you here.

To answer your earlier question about hashing to get IP addresses:
A hashing function is one that takes a input string or text or bytes and deterministically produces a number. An example of a hash function would be "add up the values of all of the bytes of input, mod 100". So here, for any possible input, there are 100 possible outputs. Typically a hash function is designed so that inputs that are "close together" produce outputs that are not close together. Hash functions in wide use include sha and md5sum.

A typical DHCP server gets hardware MAC address, and needs to assign an IP address. The common way to do this is via a hash function, where the input is the MAC address and the the range of output values is 2^N, where N=8, 16 or 24 is determined by the size of your network/netmask.

For a clever use of hash functions, read this:
https://dl.acm.org/doi/10.1145/3532.315102
Start on page 459 with the paragraph "McIlroy used hashing to represent 30,000 English
words in 27 bits each..."

Cheers,
Bruce

[gitbls]

Of course I know what hashing is.

I was not aware that DHCP servers hashed the MAC address as opposed to simply doing a lookup on it in some internal table. Most DHCP servers have a way to assign a static IP address to a MAC address, so when that system askes for an IP address it always gets the same one.

Similarly, when an unknown MAC address asks for an IP address, the DHCP server first looks to see if that MAC address has had an IP address "recently", and if that particular IP address is free, it re-assigns it. If it's busy or it's a new MAC address, the DHCP server assigns it one from the pool.

Given the above, it's not clear to me how your description of hashing has anything to do with DHCP servers and IP addresses.

[ballen4705]

Not meaning to be argumentative, but the way that the DHCP server first picked an IP address to offer/assign to the client was by hashing the MAC address. This is a simple way to ensure that the IP addresses are uniformly assigned and that the same IP is always assigned to the same physical MAC.

[gitbls]

OK, if that's the case, please provide a pointer to the code in either isc-dhcp-server or dnsmasq (both popular Linux DHCP servers) that actually does that. My experience with isc-dhcp-server leases does not demonstrate what you are saying.

[ballen4705]

Here is a quote from https://linux.die.net/man/5/dhcpd.conf

"The DHCP server generates the list of available IP addresses from a hash table. This means that the addresses are not sorted in any particular order, and so it is not possible to predict the order in which the DHCP server will allocate IP addresses."

[gitbls]

Interesting. I find it amusing that it discusses an internal implementation detail, forced to, in fact, to explain why the addresses aren't assigned sequentially.

However, that section doesn't mention that the MAC is hashed. But, farther on it does mention that hashes are in fact used. Again, internally, and not for address assignment

When the DHCP server issues a client a new lease, it creates a text string that is an MD5 hash over the DHCP client's identification (see draft-ietf-dnsext-dhcid-rr-??.txt for details). The update adds an A record with the name the server chose and a TXT record containing the hashed identifier string (hashid). If this update succeeds, the server is done.

If the update fails because the A record already exists, then the DHCP server attempts to add the A record with the prerequisite that there must be a TXT record in the same name as the new A record, and that TXT record's contents must be equal to hashid. If this update succeeds, then the client has its A record and PTR record. If it fails, then the name the client has been assigned (or requested) is in use, and can't be used by the client. At this point the DHCP server gives up trying to do a DNS update for the client until the client chooses a new name.

I guess that this is one of those:

May 23 13:56:17 pisrv1 named[413]: client @0x7f7812b5d0 192.168.42.3#52719/key dhcp-update: updating zone 'dyn.mydom.com/IN': adding an RR at 'p84.dyn.mydom.com' DHCID AAIBd1K14OoagfprpvsklqySpiHNkEVTW9bPDPeaNbFmX/8=

[ballen4705]

The server must base the IP assignment on the link-layer hardware (MAC) address, because the server has no other information to distinguish the client: see Table 1/Figure 1 in the original RFC 1531 where DHCP is defined:
https://www.rfc-editor.org/rfc/rfc1531.html . The only info available is the chaddr field, and if you search the document you will find "... and the link-layer address specified in the 'chaddr' field."

The choice of a hash table is indeed an internal detail, and I don't think it is required by the spec, but am not certain . One could in principle do it otherwise, but a hash table with collision checking is such a good fit that I don't think that any other approach has been used.

(If you look at Section 6 of RFC 3074 https://www.rfc-editor.org/rfc/rfc3074.html, you will find a MAC hashing function specified, which determines which DHCP server responds to a request. I suspect most DHCP servers use a similar function to assign IP addresses.)

[gitbls]

I'm nearing completion on this, and have been updating the documentation. I went back and re-read much of this thread, and I see now that you might have some confusion about how the captive portal works, so explaining it here, along with a note as to why one of your proposals won't work.

The steps the Captive Portal takes are:

1. Cleanly stop network services (dhcpcd, Network Manager, and systemd-networkd). Note that only one of dhcpcd or NM should be active on a system at any time before the Portal starts.
2. Create and configure the Captive Portal
3. Start systemd-networkd, which starts the Captive Portal
4. Listen for incoming HTTP requests
5. When an SSID/Password have been provided via the Access Point, configure the "normal" WiFi to use those
6. Start the WiFi WPA supplicant to test the SSID/Password, and run the network test
7. When the test succeeds (gets an IP address) or fails (dhcp wait count exceeded) the result is made available to the user
8. After the user retrieves the test result, the Portal either loops back for another try (on failure) or the Portal prepares to exit
9. When exiting the Captive Portal removes the temporary network configuration and restores either dhcpcd or Network Manager, as appropriate

You have stated a few times that the Captive Portal should switch to using the acquired IP address after step 7, when it has obtained said IP address from the local network. At this point, the user's web browser has no outstanding read to the Portal, because if it did, it could very easily time out, since the length of time to get an IP address assigned is indeterminate.

Further, when the user is ready to retrieve the result, how would you expect the Portal to communicate that retrieved IP address to the user's browser to present a fully-formed URL?

If your answer is to connect to the existing Portal (10.1.1.1) to get it, you would be correct. After the Portal displays the obtained IP address, it's done and on to cleanup and exit. So, there's no way to communicate the obtained IP address to the browser ahead of when the user checks results, and after the results are obtained the Portal is done.

Therefore, there's literally no value in switching to using the acquired IP address from the perspective of the Portal.

That said, the Portal DOES restart dhcpcd or NM (whichever was running before the Portal started), and that should cause the Pi to be on the network with the obtained IP address. As this happens, note that the Portal is exiting, as its job is done.

[ballen4705]

Thanks, I'll look at this closely towards the end of the week. Is it correct that in your current implementation, the RP WiFi can function in parallel :
(a) in ad-hoc mode, creating its own network and issuing IP addresses, and
(b) at the same time, connected as a DHCP client to a different network.
I assume that this is what the 'virtual wlan device' provides. Or have I misunderstood?

[gitbls]

These statements are both correct.

[gitbls]

I've checked in the updated Portal as well as updated documentation. I'm interested in your feedback.

At the moment there is no timeout on the captive portal. It will continue running until the WiFi successfully connects or --retries retries through the Fill-in-form/Test-WiFi loop. Looking at adding an overall timeout.

[ballen4705]

I am traveling until Friday, so that's going to be my first opportunity to try this out. Will do so then and report back here.

Regarding your post from two days ago, which starts "I'm nearing completion on this..." I had the following idea regarding redirection. But it may be wrong/infeasible.

To make this concrete, let's use 10.1.1.1 as the captive portal IP used by the RP, and 192.1.2.3 as the IP that is assigned by DHCP to the RP.

(1) Client connects to 10.1.1.1, configures things correctly on RP.
(2) RP sends DHCP request, gets reply assigning it an IP of 192.1.2.3
(3) RP reports success to client and its new IP via a web page.
User is told to confirm this with a click, and then after that, to connect
to the 192.x.y.z WAN.
User confirms this with a click.
At this moment the client is still connected to 10.1.1.1
(4) RP sends client a redirect to 192.1.2.3
(5) RP immediately DROPS its 10.1.1.1 connection and shifts to its 192.1.2.3 IP.
(6) RP listens on 192.1.2.3:80
(7) User reconnects, as instructed in (3), to the WAN
(8) At this point, User's browser, redirected to 192.1.2.3 is able to resolve
that IP and connect.

I don't know if this would work. It depends upon the browser being redirected to to connect to 192.1.2.3 before it has a route to that IP, and then succeeding when that route becomes available. Your thoughts?

[gitbls]

I'm not an http/website person, but based on screwing around with this, my understanding is that the CP (server) can only send something to the client web browser in response to a request. So, you're missing a few steps:

1. Client connects to 10.1.1.1, fills out and submits the form
2. The server has to respond with something without too much delay, so long before it has the "new" IP address
3. Server goes off and tries to configure the wifi and get a dhcp address
4. server gets a dhcp address
5. ??? How does it communicate this address to the client ???

In other words, the point you raised at the end of your message is exactly the problem with this scenario.

Oh, and I'm not clear why rejiggering this, even if it's possible, would be beneficial? The Portal works well today, so there would have to be a huge benefit to justify churning this all up again.

What is that benefit?

[gitbls]

Awfully quiet here. The Captive Portal is now complete, including the missing timeout capability.

I'm interested in your feedback after you've reviewed the documentation and tested it.

[ballen4705]

Apologies, I've had to do some unexpected travel, am away from my system. Feedback as soon as I return...