Multiple source roots

[rafalkrupinski]

I have a project with both hand-crafted and generated source files

+ gen/mypackage
|  + __init__.py
|  + ...other .py files
+ src/python/mypackage
|  + helper.py
+ tests
   + test_stuff.py

when building dist package, everything from gen and src/python lands together, so no fancy namespace packages.

I'm trying to make pytest to treat the project in the same manner:

[tool.pytest.ini_options]
testpaths = [
    "tests",
]
pythonpath = [
    "gen",
    'src/python',
]

but

ImportError while importing test module '${project}/tests/test_stuff.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/lib/python3.11/importlib/__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests/test_helper.py:7: in <module>
    from mypackage.helper import iter_pages, next_link_cursor_extractor
E   ModuleNotFoundError: No module named 'mypackage.helper'
=================================================================================================== short test summary info ====================================================================================================
ERROR tests/test_helper.py
!!!

[RonnyPfannschmidt]

as there is a gen/mypackage/__init__.py its not a implied namespace, thus src/python/mypackage is not part of the package search path of mypackage

as far as i know, either __init__ needs to go, or declare a namespace

[rafalkrupinski]

declare a namespace

can you elaborate?

[RonnyPfannschmidt]

implicit namespace packages only happen if no __init__.pyexists

if there is one, then other search paths for the package need to be manually added to that packages __path__

https://docs.python.org/3/library/pkgutil.html#pkgutil.extend_path helps with that

[The-Compiler]

Your example works fine:

$ mkdir -p gen src/python/mypackage tests

$ cat > src/python/mypackage/helper.py
def iter_pages(): pass
def next_link_cursor_extractor(): pass

$ cat > tests/test_stuff.py
from mypackage.helper import iter_pages, next_link_cursor_extractor

def test_something():
    assert iter_pages() is None

$ cat > pyproject.toml
[tool.pytest.ini_options]
testpaths = [
    "tests",
]
pythonpath = [
    "gen",
    'src/python',
]

$ pytest
============================= test session starts ==============================
platform linux -- Python 3.12.7, pytest-8.3.3, pluggy-1.5.0
rootdir: /home/florian/tmp/12908
configfile: pyproject.toml
testpaths: tests
[...]
collected 1 item                                                               

tests/test_stuff.py .                                                    [100%]

============================== 1 passed in 0.01s ===============================

But for starters, the traceback in your output does not match up with the example you show, so that's not what you're actually running. Please show a self-contained example that someone else can run easily, possibly as a small GitHub project or somesuch (make sure to run it as well, to make sure it actually exhibits the issue you're describing).

[RonnyPfannschmidt]

your reproducer approximation is missing the gen/mypackage/__init__.py file (which in turn tends to disable pep420 namespaces)

[rafalkrupinski]

Additionally

cat > gen/mypackage/__init__.py
def generated():
    pass

change test:

cat > tests/test_stuff.py
from mypackage import generated
from mypackage.helper import iter_pages

def test_something():
    assert generated() is None
    assert iter_pages() is None

[The-Compiler]

Ah, my bad, shouldn't have looked at this in a hurry in the middle of giving a pytest training 😅

[rafalkrupinski]

Nah, my bad, I didn't include a reproducible code. It's in the other comment now.

[rafalkrupinski]

issue reproduction

mkdir -p gen/mypackage src/python/mypackage tests

cat <<EOT > gen/mypackage/__init__.py
def generated():
    pass
EOT

cat <<EOT > src/python/mypackage/helper.py
def iter_pages():
  pass
EOT

cat <<EOT > tests/test_stuff.py
from mypackage import generated
from mypackage.helper import iter_pages

def test_something():
    assert generated() is None
    assert iter_pages() is None
EOT

cat <<EOT > pyproject.toml
[tool.pytest.ini_options]
testpaths = [
    "tests",
]
pythonpath = [
    "gen",
    'src/python',
]
EOT

[rafalkrupinski]

I'll change my project to copy the files into a single directory before testing.