Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Allow running subsets of test cases #18

Merged
merged 5 commits into from
Aug 18, 2023
Merged

Conversation

phoevos
Copy link
Contributor

@phoevos phoevos commented Aug 14, 2023

  • Allow running only a subset of the provided tests using pytest's filtering feature with the -k command-line option, which facilitates (de)selecting the test cases to run filtering on their name
  • Introduce a dedicated tox testenv for running all tests apart from the ones that assume MLFlow integration
  • Update README with instructions on using the newly added functionality

Closes #6

Allow running only a subset of the provided tests using pytest's
filtering feature with the '-k' command-line option, which facilitates
(de)selecting the test cases to run filtering on their name.

Signed-off-by: Phoevos Kalemkeris <[email protected]>
Add a comment in the top-level tox.ini with instructions on how to use
the newly added filtering functionality of the test suite.

Signed-off-by: Phoevos Kalemkeris <[email protected]>
Introduce a tox testenv for running all tests apart from the ones that
assume MLFlow integration. This relies on the newly added filtering
functionality of the test suite.

Signed-off-by: Phoevos Kalemkeris <[email protected]>
Update READMEs with instructions for filtering test cases using pytest
directly or tox.

Signed-off-by: Phoevos Kalemkeris <[email protected]>
@phoevos phoevos force-pushed the kf-4152-pytest-filter-tests branch from 246c5a2 to c284717 Compare August 14, 2023 16:34
Add a `filter` option to pytest to use over the builtin `override_ini`,
greatly simplifying invocation.

Signed-off-by: Phoevos Kalemkeris <[email protected]>
@phoevos phoevos force-pushed the kf-4152-pytest-filter-tests branch from 177ec43 to b3476d7 Compare August 17, 2023 10:27
@i-chvets
Copy link

Checked out branch and tried to run some tests according to instructions. Tests failed:

$ pytest -k "kfp"
platform linux -- Python 3.10.12, pytest-7.4.0, pluggy-1.2.0 -- /home/ichvets/charmed-kubeflow-uats/venv/bin/python3
cachedir: .pytest_cache
rootdir: /home/ichvets/charmed-kubeflow-uats/tests
configfile: pytest.ini
collected 5 items / 4 deselected / 1 selected                                                                     

test_notebooks.py::test_notebook[kfp-integration] 
-------------------------------------------------- live log call --------------------------------------------------
INFO     test_notebooks:test_notebooks.py:37 Running kfp-integration.ipynb...
WARNING  traitlets:kernelspec.py:286 Kernelspec name python3 cannot be found!
ERROR    traitlets:manager.py:92 No such kernel named python3
Traceback (most recent call last):
  File "/home/ichvets/charmed-kubeflow-uats/venv/lib/python3.10/site-packages/jupyter_client/manager.py", line 85, in wrapper
    out = await method(self, *args, **kwargs)
  File "/home/ichvets/charmed-kubeflow-uats/venv/lib/python3.10/site-packages/jupyter_client/manager.py", line 397, in _async_start_kernel
    kernel_cmd, kw = await self._async_pre_start_kernel(**kw)
  File "/home/ichvets/charmed-kubeflow-uats/venv/lib/python3.10/site-packages/jupyter_client/manager.py", line 359, in _async_pre_start_kernel
    self.kernel_spec,
  File "/home/ichvets/charmed-kubeflow-uats/venv/lib/python3.10/site-packages/jupyter_client/manager.py", line 182, in kernel_spec
    self._kernel_spec = self.kernel_spec_manager.get_kernel_spec(self.kernel_name)
  File "/home/ichvets/charmed-kubeflow-uats/venv/lib/python3.10/site-packages/jupyter_client/kernelspec.py", line 287, in get_kernel_spec
    raise NoSuchKernel(kernel_name)
jupyter_client.kernelspec.NoSuchKernel: No such kernel named python3
FAILED                                                                                                      [100%]

@i-chvets
Copy link

The functionality that this PR introduces works correctly, i.e. proper tests are selected when -k option is used.

@phoevos
Copy link
Contributor Author

phoevos commented Aug 18, 2023

@i-chvets, I have seen this error before and it's due to running the test inside a virtual environment. That won't work out of the box, you'd have to set up some Jupyter/ipython related stuff, hence it's not in the instructions.

This is supposed to be executed from inside a notebook where you'd usually just pip install the requirements user-wide (That's what the driver automation does ATM anyway).

@i-chvets
Copy link

@i-chvets, I have seen this error before and it's due to running the test inside a virtual environment. That won't work out of the box, you'd have to set up some Jupyter/ipython related stuff, hence it's not in the instructions.

This is supposed to be executed from inside a notebook where you'd usually just pip install the requirements user-wide (That's what the driver automation does ATM anyway).

Understood. Approving.

Copy link

@i-chvets i-chvets left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved. ACK tests should not be executed in virtual env, but in the notebook.

@phoevos phoevos merged commit fd6392d into main Aug 18, 2023
@phoevos phoevos deleted the kf-4152-pytest-filter-tests branch August 18, 2023 12:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Enable running subsets of notebook tests
2 participants