Python path probably not set correctly when installing nest-simulator/3.3

[jhnnsnk]

I installed nest-simulator/3.3 on JURECA DC with the loaded modules:

Stages/2022 StdEnv/2022 CMake GCC GSL jemalloc Boost ParaStationMPI Python SciPy-Stack

As a plan file, I used the the one for version 3.0 as an example. The installation worked fine, but the install check did not run. If the install check is commented out, the module is created properly and can be loaded. Starting the SLI version of NEST with nest works, but the Python module is not available through the PYTHONPATH. I have verified that the Python module exists in lib64/python3.9/site-packages and that it works, but apparently the path is not set correctly such that it is not found.
Do you @ackurth have an idea what could have gone wrong?

[terhorstd]

Not sure about the exact background here, but maybe a symlink lib -> lib64 could help. The description of site states:

For the tail part, it uses the empty string and then lib/site-packages (on Windows) or lib/pythonX.Y/site-packages (on Unix and macOS).

Seems it won't look into lib64. If the symlink works, try adding it in some post-install part of the planfile.

[jhnnsnk]

It looks like this automatic expansion does not work, not even with the symlink.

The PYTHONPATH set by Builder is just:

<path to Builder>/install/nest-simulator/3.3/default

If I add the tail to \$INSTALLDIR in default.module the Python module is found as expected. The PYTHONPATH then reads:

<path to Builder>/install/nest-simulator/3.3/default/lib64/python3.9/site-packages

Is there a good variable to use for getting the pythonX.Y automatically right?
I noticed that such an extended path is also used for other software, e.g., elephant.

By the way, during the installation, I get this error and I am not sure if this is expected:

Copying nest_simulator.egg-info to /p/project/cjinb33/senk1/software/Builder/install/nest-simulator/3.3/default/lib64/python3.9/site-packages/nest_simulator-3.3-py3.9.egg-info
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/p/software/jurecadc/stages/2022/software/Python/3.9.6-GCCcore-11.2.0/lib/python3.9/site-packages/pkg_resources/__init__.py", line 466, in get_distribution
    dist = get_provider(dist)
  File "/p/software/jurecadc/stages/2022/software/Python/3.9.6-GCCcore-11.2.0/lib/python3.9/site-packages/pkg_resources/__init__.py", line 342, in get_provider
    return working_set.find(moduleOrReq) or require(str(moduleOrReq))[0]
  File "/p/software/jurecadc/stages/2022/software/Python/3.9.6-GCCcore-11.2.0/lib/python3.9/site-packages/pkg_resources/__init__.py", line 886, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/p/software/jurecadc/stages/2022/software/Python/3.9.6-GCCcore-11.2.0/lib/python3.9/site-packages/pkg_resources/__init__.py", line 772, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'nest-simulator' distribution was not found and is required by the application
Defaulting to user installation because normal site-packages is not writeable

The installcheck still does not run with build_install_test ().

[terhorstd]

The exception is expected if not installed with an activated venv or within conda. That is also part of the …site-packages problem. NEST tries to figure out the correct python install path and finds itself in a system install environment. It tries to write to the system Python's package dir and fails, of course. There have been some changes in these Python install mechanics in recent NEST versions and likely this just hits a case that was not considered well enough.

In general Python package installs are difficult to handle with a non-pip package manager and many systems have worked around kinks and quirks. Builder does not have a magic bullet either.

I think @ackurth had a trick to do it, I also vaguely remember something in the NEST issues or pull-requests… need to search for it… nest-simulator:#1959 has some of the path calculation in it, maybe this hint's a the right place already.

[akorgor]

Is there a good variable to use for getting the pythonX.Y automatically right?

I suggest in Pull Request #55 to use the TCL command glob in the .module file like this:

prepend-path PYTHONPATH [glob -directory \$INSTALLDIR/ -type d lib*/python*/site-packages/]

[terhorstd]

Thanks @akorgor, that looks very promising!

The glob should be pretty unambiguous, since INSTALLDIR is specific to each installation and should contain only a single specifically versioned site-packages sub-dir for the NEST+Python combination.

@jhnnsnk, I've added you as a reviewer to #55, hope that's ok. Please reassign, if you know someone more suitable.