Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use a precompiled Portduino version of firmware for python integration testing? #643

Open
geeksville opened this issue Jul 31, 2024 · 7 comments
Assignees
Labels
enhancement New feature or request

Comments

@geeksville
Copy link
Member

geeksville commented Jul 31, 2024

Hi @ianmcorvidae and @thebentern,

I'm making this issue to discuss an idea. In the 'old days' we had a compiled C++ version of the firmware code that ran on linux using the portduino layer. We used that binary in the CI build of the python toolchain to do integration testing - because we could talk to it like a "real" device (because it was 95% the same). This was handy for integration test purposes.

I guess that somewhere over time that got deprecated/removed.

Now that there is slick codecoverage support that someone added (@ianmcorvidae ?) for python: what do you think about using that again?

The reason I ask is that in cleaning up my power measurement stuff for checkin I realized that the PowerStress stuff needs code coverage testing. I could learn how to use the python mocking framework, but I'm worried that even if I use that framework if I mock out the PowerStress class to pretend to be the real device I'm not really testing that much. Because PowerStress is mostly done on the device side.

Instead: (proposal)

  • Use a precompiled linux device firmware (do ya'll already/still do a linux build of the device firmware?), fire it up.
  • Run the python tests as integration tests but have them talk via TCP to the actual device firmware fired up previously.
  • This would allow testing lots more of the python code - and also implicitly testing the device code as well. So it would be a true integration test.
  • Testing all this code would then be seen by the code coverage tool.

Does this seem plausible? Would you like changes?

My current queue is:

  • finishing the power stuff (done next few days?)
  • stress testing nrf52 filesystem stuff to try and reproduce the reported rare corruption and make a robust fix (idk, a couple of weeksish?)
  • do this third?
@geeksville geeksville self-assigned this Jul 31, 2024
@ianmcorvidae
Copy link
Contributor

I've thought similarly, so I definitely agree! There's still a SimRadio system built off portduino, used among other things by the simulator (Meshtasticator), as well as the actual linux-native meshtasticd that can talk to a radio using SPI. So I think it's quite within reach. I'm not who added the code coverage testing -- I think it's nice for knowing what's being tested to some extent, but I'm not sold on the broad usefulness of striving for 100% coverage. I've worked a little bit with hypothesis, a property-based testing system (if you've ever touched Haskell -- QuickCheck is property-based testing), and I'd love if a lot of our tests could move to something like that which works more off stating assumptions and checking that they hold, with something like what you're talking about doing integration-type testing (which is what a lot of our current stuff is doing, albeit by way of a lot of mocks that aren't always that current and a lot that aren't even functional -- so something running actual firmware code would definitely be better).

As far as implicitly testing the firmware code, I think that's also good and we might even be able to build this such that it can optionally use real devices (somewhere far away from defaults/the public mesh), which could uncover device-specific bugs, potentially. I don't imagine we could systematize that without a lot of resources (especially given how many devices we support these days), but it'd be a cool option to have.

Basically: yes, I like this!

@geeksville
Copy link
Member Author

geeksville commented Aug 1, 2024

ooh Meshtasticator looks great (I didn't even know this hotness existed - awesome work @GUVWAF !) and is probably an indirect descendent of that old linux build I was mentioning. awesome!

@geeksville
Copy link
Member Author

geeksville commented Aug 1, 2024

ooh the docker 'auto build and run the native build' container stuff is slick.

@thebentern
Copy link
Contributor

I'm game! I think some of the nominal testing that was originally there shook loose from the 1.3/2.0 days when we moved all of the config protos around.

Long term, I'd actually like to have real IoT hardware in play for e2e testing. I think ESP32 based targets could work the easiest, since we have esptool access already in python. I wonder if a USB hub with programmatically switchable ports exists, because inevitably when something goes wrong, power cycling could save us some flakey test heartache.

@thebentern thebentern added the enhancement New feature or request label Aug 1, 2024
@GUVWAF
Copy link
Member

GUVWAF commented Aug 1, 2024

While I'm not proficient with Python testing, I'm happy to help with some Python glue code to create a basic test scenario of 3 Portduino nodes that talk to each other with a hop in between. I believe with that you can cover a lot of test cases for mesh communication (broadcasts, DMs with ACK, traceroute, etc.) already.

@ianmcorvidae
Copy link
Contributor

Hi @GUVWAF -- I was thinking of digging into this a little bit soon possibly. I can probably adapt things to work with pytest fixtures if you're able to give me an idea of the setup code that needs to be run (and the constraints on it -- I guess primarily that these will need to run on Linux and not other platforms). So when you've got a chance, let me know -- we can chat in Discord or such too if that would be helpful. I'm thinking that with something simple in place I can get us some simple behavior tests (send DM A -> C, confirm C gets it and sends back an ack to A; send traceroute and check how the route is shown, etc.) to get this process started. Thanks for the offer of help; sorry to get back to it so late!

@GUVWAF
Copy link
Member

GUVWAF commented Sep 28, 2024

Hey @ianmcorvidae, in short you'll indeed have to run it on a Linux machine, or in a Docker container. You'll have to run multiple instances of the program, all listening to a different TCP port as specified with -p. Then you open a connection to each instance and subscribe to simulator packets. Then, depending on who transmitted a packet (you can map it to a node based on the TCP port you received it on), you can decide who will receive this and forward the packet to those nodes. And then you can just use the connections as normal to send DMs and check who received it, etc.

Hope this helps to start, feel free to ask more questions here or on Discord.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

4 participants