How does the "testing" part discover modules?

[FRidh]

We're having a monorepo with several packages and are using pytest as test runner. The dependencies are provided by a Nix environment. To have all the Python-parts in VS Code pick everything I tried several things in the past, eventually landing with a setup that was functioning for everyone. Suddenly the "testing" part wasn't able to find the packages in the repo, at least for some, while using pytest directly in the terminal discovers everything fine.

Even on different machines with the exact repo (and thus Nix environment), VS Code build, and plugin versions it sometimes works, sometimes not.

The .vscode/settings.json is a symlink to a file in the repo containing

{
  "python.envFile": "${workspaceFolder}/.env",
  "terminal.integrated.env.linux": { 
    "PYTHONPATH": "${workspaceFolder}/packages/nash_api:${workspaceFolder}/packages/nash_background:${workspaceFolder}/packages/nash_cluster:${workspaceFolder}/packages/nash_shared:${workspaceFolder}/packages/nash_storage_api:${workspaceFolder}/packages/nash_storage_client"
  },
  "terminal.integrated.env.osx": { 
    "PYTHONPATH": "${workspaceFolder}/packages/nash_api:${workspaceFolder}/packages/nash_background:${workspaceFolder}/packages/nash_cluster:${workspaceFolder}/packages/nash_shared:${workspaceFolder}/packages/nash_storage_api:${workspaceFolder}/packages/nash_storage_client"
  },
  "python.defaultInterpreterPath": "${workspaceFolder}/nix/env/bin/python3",
  "python.testing.pytestEnabled": true
}

In the terminal, users run nix develop which ensures the symbolic link is created, and the Nix environment is available

            export PYTHONPATH=$(readlink -f $(find ./packages -maxdepth 1  -type d ) | tr '\n' ':'):$PYTHONPATH

            # Make the development environment available
            nix build .#devEnv --out-link nix/env

            # Add VS Code configuration
            mkdir -p .vscode
            if [ -e ".vscode/settings.json" ] || [ -L ".vscode/settings.json" ]; then
                rm ".vscode/settings.json"
            fi
            ln -s ../nix/vscode/settings.json .vscode/settings.json

As you can see, we export here PYTHONPATH which supposedly should be picked up by other tools.

There is also a .env file that has

PYTHONPATH=packages/nash_api:packages/nash_background:packages/nash_cluster:packages/nash_shared:packages/nash_storage_api:packages/nash_storage_client

In pytest.ini we configure that we need importlib import mode.

addopts = --ignore=nix/  --import-mode=importlib -m "not needs_datalake and not hypothesis"

So, now I am wondering. How does the testing part exactly find tests? Which environment variables or configuration does it consider?

2023-08-25 11:44:04.920 [info] Discover tests for workspace name: backend - uri: /home/freddy/code/nash/backend
2023-08-25 11:44:04.920 [info] Running discovery for pytest using the new test adapter.
2023-08-25 11:44:04.942 [info] > ./nix/env/bin/python3 -m pytest -p vscode_pytest --collect-only
2023-08-25 11:44:04.942 [info] cwd: .
2023-08-25 11:44:07.685 [info] Test server connected to a client.
2023-08-25 11:44:07.686 [info] Using result resolver for discovery
2023-08-25 11:44:07.686 [error] pytest test discovery error 
 _pytest.nodes.Collector.CollectError: ImportError while importing test module '/home/freddy/code/nash/backend/packages/nash_api/nash_api/__tests__/test_main.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
packages/nash_api/nash_api/__tests__/test_main.py:12: in <module>
    import nash_api
E   ModuleNotFoundError: No module named 'nash_api'
 Check Python Test Logs for more details.

AttributeError: 'Namespace' object has no attribute 'environment'
 Check Python Test Logs for more details.

_pytest.nodes.Collector.CollectError: ImportError while importing test module '/home/freddy/code/nash/backend/packages/nash_storage_client/nash_storage/features/era5/__tests__/test_era5_repository.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
packages/nash_storage_client/nash_storage/features/era5/__tests__/test_era5_repository.py:2: in <module>
    from nash_storage.features.era5.era5_repository import ERA5Repository, ERA5RepositoryMock
E   ModuleNotFoundError: No module named 'nash_storage'
 Check Python Test Logs for more details.

Note executing from the terminal

$ PYTHONPATH=/home/freddy/.vscode/extensions/ms-python.python-2023.14.0/pythonFiles:$PYTHONPATH ./nix/env/bin/python3 -m pytest -p vscode_pytest --collect-only

functions fine! Which makes me wonder, is whatever call that invokes this overwriting PYTHONPATH instead of prefixing it?

[FRidh]

I rolled back VS Code to

$ code --version
1.78.2
b3e4e68a0bc097f0ae7907b217c1119af9e03435
x64

and the Python extension to v2023.8.0 and all is good.

Running again

$ code --version
1.81.1
6c3e3dba23e8fadc360aed75ce363ba185c49794
x64

the issue occurs with Python extensions newer than v2023.8.0.

[FRidh]

NOT OK v2023.9.11521013
NOT OK v2023.9.11461009
OK v2023.9.11371007
OK v2023.9.11311005

v2023.9.11461009 has a tag, but unfortunately older ones (such as v2023.9.11371007) do not.

[FRidh]

Opting out of the new test adapter (that I earlier on opted into) resolves the issue

"python.experiments.optOutFrom": ["pythonTestAdapter"]

Alternatively

"telemetry.telemetryLevel": "off"
"python.experiments.optOutFrom": [ "All"]

Unfortunately all of these are User settings instead of Workspace settings.

~/.config/Code/User/settings.json

#21642

[eleanorjboyd]

Hi @FRidh, apologies for the delay I will review this and get back to you, sorry!