LoRaWAN: LoRaWAN_ESP32 - session persistence data handling on ESP32, can also manage provisioning data

[ropg]

Thank y'all for working on RadioLib and LoRaWAN specifically!

I think I have built something that may or may not be of use to people wanting to use deep sleep on ESP32 and/or wanting to manage their LoRaWAN / TTN provisioning data in the ESP32 NVS key/value store. This interfaces with the new and wonderful [get|set]Buffer[Nonces|Session]() functionality in LoRaWANNode.

Please check it out

I'd be happy for people to comment on it, test it, and for any feedback generally.

(I'm aware my code now does things that are possibly more generally needed and would, maybe, possibly, be more appropriate to be incorporated in LoRaWANNode, to do with textual representations of LoRaWAN bands (i.e. "EU868") which would be needed for anyone that wants to, say, populate a dropdown with them.)

@StevenCellist @HeadBoffin @jgromes

[StevenCellist]

Hi @ropg, nice work on the library you linked.
However, once again, you are doing stuff at your own pace before we get around to regulating this in a nice manner. As you must have seen, there is a new organisation and repository dedicated to this topic, that, once again, should provide not just an interface for ESP32, but for a collection of platforms. Life is not all about the ESP32. And not about your repositories - instead, there is one within an organization that we can share with devs of all sorts of platforms, not one repository for each platform by one person.

Now if you would have waited for a week or maybe two, you would have gotten a request to collaborate with us on a persistence interface in that repository that would provide an abstraction to be used by at some point a collection of platforms.
As we, and especially @HeadBoffin, use a collection of platforms, we have a clue of what is probably best to add for this interface, what toggles & switches are required, what we should pay attention to.
For example, there are different buffers - which one gets saved to where? Can you specify which one gets saved where? These are some of the points that must be made into a guideline before we throw any random code at it. And which I don't see in your library.

Now, if you can wait, and not update just the points I mentioned here, we can put together some design document with what we think is best & necessary for an interface that does not only fit the needs that you put together for ESP32, but for each and everyone of us.

Note that the collaboration on this is hanging by a silk thread - if you continue working on Open Source like it is a play ground just for you and your ESP32, then this is no place for you.

[StevenCellist]

To be clear - the work you provided in the library does actually look good and useful (just as the find on the persistence), we can probably port it over in some way or another to the proposed library. The commentary provided above is mostly as a reaction to the way it is delivered.

[ropg]

Eh?

The whole idea of me working on ESP32 stuff in its own separate place is to not bother you guys. In fact I was told that all this stuff has no place in RadioLib and that I should go off and do it in its own repository. Clearly you can and should put in RadioLib/LoRaWANNode whatever you like, I am not pressing for anything or at this point even arguing for or against anything. I am nice and polte, I say friendly things and I show you what I made. You're most welcome to criticize, I'm sure it's wrong here or there.

As for the buffers: my code saves the nonces to flash and the session state to RTC RAM. Which is the obvious thing to do as you can lose the session but need the nonces to join for the next session. I think @HeadBoffin 's new example does the same if I'm not mistaken.

What makes you think that I think this is "a playground just for me and my ESP32"? I just want to play with these cool radio boards and for people to be able to build cool stuff, which I think is what drives all of us. Clearly there's a need for something like my library, as distilling what's going on from a massive example s not everyone's cup of tea. Just like there's a need for the example, for those that do want to tinker with that particular part.

I've always been very happy and pleased when people build things that interface with things that I built, and I'm a bit surprised, more than anything, at my reception here. I contribute to other things sometimes, and people are generally quite happy with my work. Don't you think the managed provisioning data feature is even just a litte bit cool and nifty? It comes with easy (and properly done) serial entry and is open in that it allows people to plug it in from other places such as web interfaces or what not. People on ESP32s will need something between their key/value store and LoRaWANNode, no? I mean, not everyone is going to roll their own based on the example, and nor should they. Or?

[ropg]

oh, seeing your reply-to-self only now.

[StevenCellist]

The key idea here is that the LoRaWAN stack in RadioLib is still under active development. We have an active roadmap, we are working on stuff. The other libraries you mention are likely not so actively developed, in which case that is much less of a problem.

The things you build are cool, probably work well and are generally useful, but parts of RadioLib may change, new things are going to end up at the official repository/ies, or are already being worked on. You knew that easy examples were in the making, as the PR was already open for three weeks.
As I remarked before, if there's anything up, please give us a heads-up about something before actually doing it, as we can then arrange or schedule stuff. To which you said "yes I will". The hesitant reaction here is because you propose something that we are also working on, without giving that heads-up you said you would give. Because if you had done so, we would have said "hold on, give us a week, and we'll get some guidelines so it can end up in the official persistence repository". That way it's a win-win, because you can create a nice interface that can be expanded to multiple platforms and is in the official repo, and we get a nice interface in the official repo! And that is the difference that I mean between fully open source and your play ground. By all means, have fun with your Heltec repo - that's all your game. But this is something that is not only your project. So please, keep that in mind!

[jgromes]

I just wanted to drop in here before the discussion has a chance to get heated and people post stuff they regret later (many a good project fell apart due to that particular pitfall).

@ropg I like what you did there! RadioLib is meant to be modified to suit the user needs and it's always great to see projects being built on top of it. @StevenCellist I don't think @ropg proposed anything in particular to be done from our side. So I think the only thing he risks by experimenting is his own time going to waste - but only if his definition of "wasted time" includes "time spent writing code that may not end up in the upstream" and that may not be the case :) I too indulge myself in the occasional experiment (that is how RadioLib ended up with its more silly features, such as SSTV support).

In any case, as @StevenCellist was saying, persistence will get revisited in the future, and that point I'd be more than happy to involve you @ropg, I think everyone (and especially the project) can benefit from that ;)

[ropg]

@jgromes Glad you like it, happy to help whenever possible. And indeed, if I had counted the hours and had suffered mentally every time something I spent days on didn't quite make the mainstream, I'd long be a screaming wreck in an institution. (Although it sure is nice to see something fill a real need from time to time: the Heltec library got 140 stars on GitHub in just a few weeks. 🎉)

One of my goals right now is to have really good tools for people to use on these poor neglected Heltec boards, all these other projects (such as this persist thing) have been spun off from that. (The display library for these OLEDs also has much better Arduino Print support now.) Clearly (in my mind, anyway) people that tinker to create all sorts of solar/battery-powered sensor-thingies need to be able to use deep sleep without a steep learning curve or taking onboard a large amount of code/complexity, and I'd like to provide them with a way to store the provisioning data in flash also. It just seemed selfish to keep that all in the Heltec library when it can, at least potentially, serve more people with RadioLib / LoRaWAN on ESP.

@StevenCellist @HeadBoffin I think the examples are great, and my repository is also quite useful. Anyone that wants to DIY it should take one of the examples and build on it. But my library (or something that it evolves into or that has an interface much like it) also serves a need: for people to have building blocks so that their complexity can be elsewhere. (Or maybe this thing their working on is their very first electronics project, and the examples merely bewilder them.) And if this code turns out to be just a thought experiment on the road to somewhere else, that's fine.

To explain where I am coming from: I often switch between MSVC for the library/libraries I'm working on and Arduino IDE to compile the example sketch that's #include-ing it. Not because I haven't used PlatformIO or Espressif SDK, but because I believe in Eat Your Own Dogfood. Tech grows. For every programmer that can help her/himself with all these grownup tools and steep learning curves are many more young aspiring newbies first trying their hand at this stuff. I like taking cutting-edge things from the hands of geeks like us and give them polished rounded APIs and slick READMEs for inexperienced new techies to be encouraged by. Being told "But it's not ready!" is not entirely unfamiliar terrain for me, and I'll deal if everything changes.

Did not mean to offend anyone. And in fact, thought it was concluded that code for talking to this feature on the different platforms happening at some distance from the main codebase was the desired outcome.

[ropg]

Suddenly reminded of: "Strong software is not released, it escapes!" 😎

[StevenCellist]

Truce. Will check out the confugurator, seems like a nice tool. As mentioned by @jgromes, the persistence will be revisited later.

[HeadBoffin]

Having vaguely nailed jelly to the wall yesterday with an update to the examples with some elements of the API still up for refinement, I can fully understand @StevenCellist's initial reaction and frankly, I'm slightly worried for the universe as I'm meant to be the belligerent one.

The subtext / theme that runs through all of this is that neither of us do "Ta Dah" or self-publicity. So when @ropg's brings out his latest/brilliant/ultimate/whatever widget whilst we are at the bottom of the coal mine in the heat & sweat & dust, it's just a tad more than irritating and given most of my DNA is English, that's me being stiff upper lipped and polite. In theory we could just ignore @ropg's announcements but that's sort of rude, which again isn't a very British sort of thing.

BTW, getting lots of stars on GitHub after offering up your library to Hackaday is just more "Ta Dah" and is not an endorsement. The quantity speaks to either Hackadays reach or the view of the visitors.

The thing that concerns me most of all is the total lack of experience that, by self-admittance, Rop has with LoRaWAN so anything produced has to be scrutinised for detail - like checking sub-band entry is < 255 - or not including AU915 - or checking the combination of frequency plan (not band) & offering up the TTN sub-band as a default & proper restriction on the actual number of channel plans for a region. And the 64 or 128 bits is just a confusion for almost everyone as it's never presented as bits, not entered as bits and for many users they don't know or don't care. Cue people asking for a 64 bit 8 byte EUI and when you give them an EUI they say that it doesn't have 64 bits - because that's the sort of questions that come up.

Rop positioning himself as the savour of the smaller maker from the evils of complexity is also self-aggrandisement of the highest order. What the hell does he think we are doing? LMIC works if you follow the instructions. You have to work hard to break LMIC-node. Neither do sleep for ESP32. So that's why we are here. Plus there is a whole community on TTN answering questions.

And the results several years on are universally agreed, there is a small subset of makers that clearly have a thin grasp of the concepts of coding, doing things like pushing a float in to a char without casting or any comment on potential arising issues, but the vast majority of people struggling are experiencing the reality gap between short range Bluetooth/BLE or WiFi that was/is purposed designed for end users and a stack that requires a whole heap more than sanitised examples & utilities. You learn to juggle with tennis balls, there is no point making fireproof gloves for people to start out with lighted brands when what's actually needed is better information on how balanced objects rotate as they are thrown, so that the move from round items to batons is better understood. But that requires some theoretical learning, not the strong suit of the "Want it Now" generation who can't fix things when broken because they don't understand how they work.

Curious to know how using MSVC & Arduino IDE is eating your own dog food - I'm not aware that you wrote those.

TL;DR: Yes, provisioning over serial is a thing, rarely see it in the maker community as they can just put the details in to the config file, no one in their right mind should be using RadioLib LoRaWAN for a commercial project yet but when it's been hammered to death on various desks including someone who can tell us how much of the certification stack it passes, then serial provisioning may become a thing as people use the stack for devices that are sold.

If you really want to help, read the last three years of the TTN forum and write docs on things like proximity of gateway to device when testing, how antenna gain works for & against the device & gateway, legal issues around duty cycle, applicable uses for LoRaWAN, creating efficient payload formats & transforming data on the device to suit and so forth. And while you are doing that, deploy a several dozen devices of different platforms and half a dozen gateways of different vendors and half a dozen different commercial devices so you can see what is going out outside the Heltec V3 bubble.

[HeadBoffin]

PS, No LoRaWAN ABP examples were hurt in the typing of this message.