Noob autotest question

[laat0003]

Hi,

I'm trying to run the autotests after making changes to a few files and I don't think I have everything necessary to run them in my environment but I'm not 100% sure. I get "module not available errors" when starting the autotests before it dies. If I have all of the dependencies to use Pyemu should I also have everything needed for the autotests or are there other dependencies (aside from pytest) needed?

Thanks.

[laat0003]

All good. It was flaky and modflow-devtools that were missing. I do seem to get a huge number of fails when doing all the autotests though most of which appear to have nothing to do with my changes. Should I ensure zero fails before submitting a PR?

[briochh]

Hi @laat0003,
It really depends on the nature of the fails. Autotesting for pyemu is a little complicated due to dependancies on PESTPP exes for the full stack tests. It might be these that are failing, if you don't have the PESTPP exes or bins in the expected location. This gets done automagically when the CI is run on GitHub Actions but for local runs we need to make sure that these are inplace ourselves. Could this be where your tests are failing?

[laat0003]

Thanks for that @briochh. I'm sure this is probably the reason. I'm in the process of moving house at the moment so won't have the time to look into it but if turns out not to be the problem then at least I'll have more of an idea why.

[rhugman]

Hey - I'm currently spending more time in getting the local versus github action autotests to play nicely, than you know writing the code itself :).

Any guidance on a pyemu autotest-noob-proof workflow?

[laat0003]

Hi @rhugman. Apologies for the delay in response. Unsure about a complete workflow but I can let you know some of the issues I had just getting off the starting line. The readme page information is great for people who already know about and have some experience with autotesting. I didn't have pytest so had to install it and I also read the basic example information on their specific page so that I had some idea of what was going on.

It was right about then then that I thought life would be so much easier if just once I got a job at a place where people already knew all this stuff. Perhaps that should be your first recommendation for doing the autotests :)

Anyway, after installing I still wasn't 100% sure what to do but just launched my env and navigated to the autotest directory then typed pytest and hoped for the best. It failed immediately but was nice enough to leave sufficient hints about missing libraries, so I had to install flaky and modflow-devtools. I still wasn't 100% sure if just typing pytest was correct but it seemed like it was doing the right thing so went for it again. Lots of red F's appeared and some red E's - which I assumed was bad. Some green dots, which I assumed was good. Some orange 's' and 'x' which I figured might be the equivalent of participation awards, kind of like, "Nice try, but you still suck".

If I was impatient, I would have thought that it was finished when doing en_tests.py because it seems to sit there for ages like it had stalled after a few F's. I tried to move all the exe's I thought would be used into the Windows sub-directory in the bin folder as suggested by @wkitlasten but this didn't help me out in anyway. I still get 132 fail, 75 pass, 3 skip, 1 xfail, 427 warnings and 8 errors. But it does seem to get through the tests slightly quicker each time I do it - so I've got that going for me. :)

I haven't really had a chance to sit down with it again since because - well life. Hope some of this of use.

[briochh]

Hey @rhugman,

Like most things I am no expert on pytest but I hacked through a way of making it work well for me.

As @laat0003 points out there are a few dependancies for the autotests (actually more dependancies for the tests than for the main library). So getting the right env set up is defo the first step. Not sure if the environment.yml in etc covers of everything for the tests. There is also a list in the pyproj.toml.

As for running the tests. There are a few options. Some of the tests will still run the classic way, uncommenting them in the __main__ context in each test.py and just running that script (however, you like to do that, e.g. in debug). However, some of the tests now make use of the tmp_path workspace stuff in pytest (which is handy when running tests in parallel and need them to not interfere), as well as some generic functions so we don't have to repeat ourselves so much in the tests. You should be able to run them using pytest from the command line (within your python env) e.g. pytest en_test.py run from inside the autotest directory should run all the tests in that script (default is all function that start or end with test). There are a few args that can be useful --basetemp=<yourtempd> will run tests that use pytests tmp_path inside , which is pretty handy for seeing the files generated by the test. -n auto is good for running tests in parallel (which is what happens on GHA). You can also be specific about which test to run, e.g. pytest en_test.py::add_base_test

Personally, I tend run tests from within pycharm, using point and click to run (or debug) specific tests or files. I needed to install the pytest imp plugin to pick up all the (startwith and endswith test) functions (hangover from nosetests). I think I added autotest/pytest.ini to Settings → Tools → Python Integrated Tools → py.test to make use of the custom test namings. I add those --basetemp etc. options to the Additional Arguments section in the pycharm run config and it mostly works well in either debug or run mode.

One frustration that I have is,when running in debug, pytest swallows some of the stdout and stderror which kinda restrict the real time feedback (on warning etc) until the end of the run. I am sure there are clever way to stop this but I haven't dug into it too much.

Happy to have a discussion on improving the tests, they are definitely quite cumbersome! It is always challenging to make sure that we are testing all of the connected operations that pyemu is used for whilst staying reasonably efficient, both on GHA, and locally. This remote vs local efficiency is often a bit of a trade off too!

[jtwhite79]

I'm probably to blame for the convoluted nature if the testing. Early on I was struggling to write tests that could find all the binaries and dependencies needed across various OSs, which is probably a big reason for the complexity of the testing.

when I make a new test, I just copy the initialization and pathing parts from an existing test to make sure all of that pathing mess is maintained...for me the hardest part about all this is that relative/abs path from where the test is running to where the binaries and dependencies are (and making sure everything is done so that it is OS agnostic)

And when I'm running tests locally, I uncomment the test i want and try to run them from the terminal. And it helps to have VMs available for specific OSs...

[wkitlasten]

Running the failing tests in debug mode can help find the underlying cause. I recently found the Windohs versions of some exes were missing from my bin directory. "Perfect spheres are pointless."

…

On Tue, 11 Jun 2024 at 11:58 AM, laat0003 ***@***.***> wrote: Thanks for that @briochh <https://github.com/briochh>. I'm sure this is probably the reason. I'm in the process of moving house at the moment so won't have the time to look into it but if turns out not to be the problem then at least I'll have more of an idea why. — Reply to this email directly, view it on GitHub <#505 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADSJXRBFMNSJG4SOOCTBGK3ZGY4SZAVCNFSM6AAAAABHP4ZYTWVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TOMZSGYYTE> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>