Enable running subsets of notebook tests

[phoevos]

Currently, we rely on dynamic test discovery to determine the set of notebooks to be executed as part of the test suite. More specifically, we look inside the notebooks directory (and its nested directories) for any Python notebook files (i.e. .ipynb), which we then use as input for pytest's parametrize to generate a test for each notebook.

The above means that all tests found inside the notebooks directory will be executed when pytest runs.

Goal

Enable specifying subsets of the notebook tests to run.

Solution

pytest CLI offers the -k option, which can be used to filter tests by keyword expressions (substring matching on names).

Let's assume we have the following notebooks:

1. kfp-integration.ipynb
2. kfp-minio-integration.ipynb
3. minio-integration.ipynb

Then we have the following options:

• Running pytest executes all three
• Running pytest -k 'kfp' executes 1 and 2
• Running pytest -k 'kfp and not minio' executes 1

This is possible because we use the notebook name as an ID when parametrizing the suite, which adds it to the test case name.