Test and timing suite

[stevevanhooser]

Hi Eivind and Vijay -

In preparation for a 0.9.3.5 release before the end of the quarter, I wanted to ask if you have automated any of the benchmarking tables that you have produced in places such as https://github.com/emeyers/Brain-Observatory-Toolbox/wiki/Live-Script-Performance-Overview ?

If not, I thought it might be nice to add a bot.test.* namespace that runs those files to a) verify there are no errors and that b) measures the time in order to make the benchmark table. As an example, I'm imagining that one file might be bot.test.suite(PLATFORM), say bot.test.suite('Matlab Online') runs the full Matlab Online suite, and bot.test.suite('local') might run the suite on a local machine, etc.

What do you think? Or is this already in there somewhere that I've missed?

@ehennestad @vijayiyer05

Best
Steve

[vijayiyer05]

Hi @stevevanhooser, there are some unit tests coded up at bot.internal.test. But these haven't been used lately & at least some may pre-date recent refactoring, so several may not run/pass as-is.

Given the stage of project (key need to add new functionalities) & given that the rich set of examples provide rather good coverage, my thinking would be to steer clear of effort on formal unit testing at this time.

You may recall that I recently mentioned a 'smoke testing tool' hopefully coming soon. That would serve as a bridge between live script examples and formal unit testing, basically to hide latter from average developers. Let me check into status this week.

Another emerging best practice is the use of the build tool introduced in R2022b. A good & lightweight approach may be to add a buildfile (to the recommended location) which runs the examples as you describe. Note the environment dependence may need to be determined/inferred internally versus through an argument, as this workflow is meant for CI versus interactive use (to my current understanding).

[stevevanhooser]

Yes, I was envisioning something more like a set of smoke tests, that would also measure the timing table. (That is, we'd run a clean-up that removes the cache, runs the optophysiology example and measure the timing, run it a second time and measure the timing, etc.) That way, right before deploying each version we would be able to compute the timing table for the release notes with a single command (without having to manually run each example) and also verify that there aren't any small errors before deploying.

I wasn't thinking of granular unit tests..that would be a lot of work.

[ehennestad]

I have created a private repository containing the functions you outline above, i.e starting from clean caches, timing each of live scripts and then saving the results which can later be used to create the tabular overviews.

My thinking is similar to what Vijay mentions above, that these functions should be hidden from the average developer so I made a separate repo for it.

It should be possible to create a Github action on this private repo which listens for tags/releases on the main repo and runs these sanity checks.

This could be relevant for major/minor/patch releases, but maybe not necessary for the "sub-patch" of bumping it up to 0.9.3.5