Question: does meson-python support python namespaces

[matteoaletti]

I am trying to migrate a project with python namespaces from setuptools to meson.

I package it with setuptools with this option in the pyproject.toml

[tool.setuptools.packages.find]
namespaces = True

While trying meson as build backend, I was able to have a working wheel only without the namespace structure. When I tried with the namespace structure I could not import the package because it was not finding the namespace.

Does meson support them?

[rgommers]

Hi @matteoaletti. I'm not 100% sure, but I think that using namespace packages shouldn't be a problem. We don't have a specific test for it, but if you look at https://packaging.python.org/en/latest/guides/packaging-namespace-packages/, then you'll see that all there is to it (from a package build perspective) is having a top-level directory without an __init__.py in it. And since Meson lets you explicitly control which files to install and to where, I would not expect any interesting differences between a regular and a namespace package.

I don't quite know what [tool.setuptools.packages.find] does - probably some remapping of install directories? Do you know, and can you share your code? If not, could you please try to share a minimal/dummy package that is representative of your problem?

[agriyakhetarpal]

I don't quite know what [tool.setuptools.packages.find] does - probably some remapping of install directories?

I can take a stab at answering this. With [tool.setuptools.packages.find], setuptools shall find namespace packages by default; with PEP517/518 the behaviour for pyproject.toml-based projects (setuptools.build_meta) is that it can emit a "Package would be ignored" warning but still includes it in the list of paths that get copied and installed subsequently. This is because include_package_data=True is enabled by default for projects with pyproject.toml-based declarative metadata, but this is disabled and set to False for setup.py-based builds or those with the setuptools.build_meta.__legacy__ build-backend (I am not aware of the reasons behind this).

find = {namespaces = false} should disable the lookup of namespace packages, though this is with [tool.setuptools.packages], not through [tool.setuptools.packages.find] (both of which just refer to the same thing, I guess, because of how TOML tables are written).

I usually end up using glob-based matching to counteract this, which is more explicit, neat, and works for both usual and namespace packages – and I would imagine Meson does, or should, behave in a similar way:

[tool.setuptools.packages.find]
where = ["myproject", "myproject.*"]  # for flat layouts

which is why it is (almost) always better to use src layouts

[rgommers]

Thanks for the context @agriyakhetarpal.

I'll convert this to a Discussion topic, since it's a usage question rather than a bug report. @matteoaletti if your question is answered please accept the answer in the UI. And if there's more to discuss, by all means do so.

[dnicolodi]

Thanks @benoit9126 for reporting this issue. It seems to be a problem of the editable wheel loader. I'll check what we can do to fix it, but it may be that the system we use to implement editable wheel is inherently not compatible with namespace packages.

[benoit9126]

Thanks @dnicolodi for your help. It would be really nice to be able to develop packages using meson-python with dependencies in the same namespace.

[benoit9126]

@dnicolodi I think I have found a possible mistake in the editable loader.

In the function find_spec of the editable loader, there is this code:

def find_spec(fullname: str, tree: Node) -> Optional[importlib.machinery.ModuleSpec]:
    namespace = False
    parts = fullname.split('.')

    # look for a package
    package = tree.get(tuple(parts))
    if isinstance(package, Node):
        for loader, suffix in LOADERS:
            src = package.get('__init__' + suffix)
            if isinstance(src, str):
                return build_module_spec(loader, fullname, src, package)
        else:
            namespace = True

    # look for a module
    for loader, suffix in LOADERS:
        src = tree.get((*parts[:-1], parts[-1] + suffix))
        if isinstance(src, str):
            return build_module_spec(loader, fullname, src, None)

    # namespace
    if namespace:
        spec = importlib.machinery.ModuleSpec(fullname, None, is_package=True)
        assert isinstance(spec.submodule_search_locations, list)  # make mypy happy
        spec.submodule_search_locations.append(os.path.join(__file__, fullname))
        return spec

    return None

If you look carefully at the line spec.submodule_search_locations.append(os.path.join(__file__, fullname)) in the if namespace: section, you will realize that the path concatenation give a weird result (in my opinion).

__file__ contains the name of the current Python file. In my example _roseau_bug_editable_load_file.py and fullname contains the name of the package to import (which is a namespace in this section). In my example, it is roseau. So the concatenation of both gives something like:
'/<something>/.venv/lib/python3.12/site-packages/_roseau_bug_editable_loader.py/roseau' which is not a valid path.

I would replace this line with

spec.submodule_search_locations.append(os.path.join(os.path.dirname(__file__), fullname))

or even

spec.submodule_search_locations.append(os.path.join(os.path.dirname(__file__), fullname.replace(".",os.path.sep)))

to ensure valid path.

In my example, the first patch works.

[dnicolodi]

Thanks @benoit9126 for looking into this. I didn't really find enough time to analyze the details. I have vague memories that the weird path is needed to support some other test case, but I may be misremembering. I should have places a comment there... Can you verify that the test suite still passes with your change?

[benoit9126]

@dnicolodi I will try to prepare a PR with this change. Problems may appear with the CI.