[ENH] Mocks rollout and implementation of testing behaviors #664
Conversation
|
Build Failed 💥 |
AlexVCaron
changed the title
|
Is it possible to have a comparison of timing before and after / with and without MOC? The code looks good, @arnaudbore do you think we can merge my two PR (python3.10 and the fix) next week and then Alex could go forward with this PR. And as soon as it is working we update the test data, and do a release candidate? I tested the 4 main flow once, I did not try Freewater. I think I will re-test with a RC1. |
|
Also, @AlexVCaron can you describe what is in the new archive? What is different? |
TLDR : files are practically the same, but the new ones are the right datatype so comparison of outputs to expectations works for both mocked and unmocked cases. Three outputs files awaited by the tests have slightly changed to align with unmocked tests, namely :
It worked when mocking, since we only check if processing errors were done by the script on those files, but didn't when running the full algorithm on them. The difference obtained between the output and the one awaited was tiny and after investigations, I think it was because of a type conversion error on my end. |
AlexVCaron
force-pushed
the
feat/rollout_mocks
branch
from
ea25380
to
3431d8a
Compare
|
Build Failed 💥 |
|
I started thinking about mocks again. Where do we go from here ? There is a few scripts that would benefit from mocking, but we need to address how and when we use the through Jenkins. I also created a sheet to follow improvements that come from using mocking here. I'll update it when new mocks come into action. |
AlexVCaron
force-pushed
the
feat/rollout_mocks
branch
from
3431d8a
to
8e07642
Compare
|
Build Failed 💥 |
|
Build Failed 💥 |
|
Build Failed 💥 |
|
We should talk about this at our retreat preparation! |
|
Hello @AlexVCaron, Thank you for updating !
Comment last updated at 2024-01-15 21:32:07 UTC |
AlexVCaron
force-pushed
the
feat/rollout_mocks
branch
from
28f5aab
to
1ffcee3
Compare
|
Build Failed 💥 |
|
Build Failed 💥 |
|
Build Failed 💥 |
AlexVCaron
force-pushed
the
feat/rollout_mocks
branch
from
7452db5
to
a958383
Compare
AlexVCaron
force-pushed
the
feat/rollout_mocks
branch
from
3961572
to
ee602cc
Compare
AlexVCaron
force-pushed
the
feat/rollout_mocks
branch
2 times, most recently
from
2ffd734
to
f8e5990
Compare
AlexVCaron
force-pushed
the
feat/rollout_mocks
branch
2 times, most recently
from
2218c73
to
d2d072b
Compare
|
Build passed ! Good Job 🍻 ! |
…s modules for pytest, loaded using conftest.py in the script tests directory.
…s modules for pytest, loaded using conftest.py in the script tests directory.
…y merging lib and scripts mocks together
AlexVCaron
force-pushed
the
feat/rollout_mocks
branch
from
d2d072b
to
be122ae
Compare
|
Build passed ! Good Job 🍻 ! |
Quick description
Running tests is long, and is bound to take even longer. To reduce this time, we decided to divide tests in two categories : library testing and scripts testing.
Library testing is fast, done on bite-size pieces of data when possible, and so will run always.
Script testing is slow, done on real-life heavy data from the real-world, and uses extensive parts of the library repetitively. Their reach can be limited with the use of mocks, to prevent repetitive testing and execution of heavy dependencies on the heavy data.
A mock is a replacement (or a patch) that obfuscates a portion of code that can be skipped or modified for the profit of the test case. They are dynamic, meaning that they get instantiated and take effect only at execution, when prescribed. They can be used to modify the effects of a piece of code (change the return value of a function or the attributes and methods of an object) and provide interfaces to inspect code execution (number of times called, with which arguments).
Current implementation :
All bindings into
pytestis done by way of a custom plugin, located inscilpy/tests/plugin.py. It handles 3 things :Mocks can be instantiated in any parts of Scilpy, may it be the scripts or the library. However, they must all be placed inside their associated test paths under the library directory :
<package_location>/tests/fixtures/mocks.py.mock_creatorfixture to create the mock(s). Refer toscilpy/reconst/tests/fixtures/mocks.pyfor an example.This makes it possible to discover all applicable mocks in pytest infrastructure (they are not applied yet !). Once done, the mock is applied to a test instance using the
mock_collectorfixture.Running tests with mocks is simple, add
--mocks <list of mocks or all>to your pytest command. To list available mocks, add--mocks all --fixturesFor more information, see
scilpy/tests/plugin.py, the module contains a full documentation. Also note that this PR enables all mocks in Jenkins by default.Type of change
Check the relevant options.
Provide data, screenshots, command line to test (if relevant)
To list fixtures and check mocks are available :
pytest --mocks all --fixtures2 tests are mocked as of now :
test_NODDI_mapsandtest_execute_angle_aware_bilateral_filtering.Run them with mocks with :
pytest scripts/tests/test_NODDI_maps.py --mocks allpytest scripts/tests/test_execute_angle_aware_bilateral_filtering.py --mocks allChecklist