Plotting tests fail because of different image sizes

[lsoucasse]

When comparing reference plots with CI plots, the image sizes are slightly different.

I tried to resize the image before using the compare_image function of matplotlib in tests/integration/test_dummy_integration.py, but the resizing creates small shifts in axes and labels which make the comparison not relevant.

[timlichtenberg]

It seems these reference plots are quite sensitive to minor changes in various things. Do we think it is a long-lasting solution to directly compare reference plots to one another or should we systematically resort to comparing only output numbers?

[lsoucasse]

It seems these reference plots are quite sensitive to minor changes in various things. Do we think it is a long-lasting solution to directly compare reference plots to one another or should we systematically resort to comparing only output numbers?

I quite like the idea to test the plots but I don't think it should be systematic for all tests as it involves managing many files.

[stefsmeets]

It seems these reference plots are quite sensitive to minor changes in various things. Do we think it is a long-lasting solution to directly compare reference plots to one another or should we systematically resort to comparing only output numbers?

It's a good question. Comparing numbers is great, and testing the plots does not replace this. However, a visual comparison of the data can often help quickly make sense of the data.

In addition, we want to regression test the plots (a large part of the output from PROTEUS is in the plots after all) as well as improve code coverage (see #268).

About this being a long-lasting solution: the pattern itself is used by matplotlib for their internal testing. I have used the pattern in several projects, and I have not encountered issues, the plots are typically reproducible cross-platform and cross-version.

We should look into the difference however. The major difference between other implementations of this pattern is that they directly work with the fig object, stripping any styling before doing the image compare. This reduces issues with e.g. font rendering (usually the biggest source of differences). Whereas in proteus we let the integration test write all the files first (including styling).

[stefsmeets]

The current code only supports saving the file and returning the figure (e.g. for use in a notebook). We should split out the actual plotting bit from the io (e.g. saving vs. displaying). This will also help solve the issue with the tests.