[BUG] make install in python venv leads to error message

[jinhzhou]

Describe the bug
Failed to install project when running make install in a python venv. Attempted to bypass the problem but ended up with failing tests.

To Reproduce
Steps to reproduce the behavior:

1. clone the repo into home dir using https in vscode
2. creating a new venv by running `python3 -m venv venv'
3. activating venv
4. running mkdir build
5. cd build
6. running cmake -DPython=ON -DPYTHON_INCLUDE_DIR=/home/user/oxDNA/venv/include/python3.12 -DPYTHON_EXECUTABLE=/home/user/oxDNA/venv/bin/python3.12 ..
7. running make -j4
8. running make install
9. Terminal outputs: ERROR: Can not perform a '--user' install. User site-packages are not visible in this virtualenv.

Attempts to fix the bug
Did the following:

1. Go to the pyvenv.cfg file in the Virtual environment folder
2. Set the include-system-site-packages to true and save the change
3. Reactivate the virtual environment.

advised in: microsoft/vscode-python#14327

Then reran make install while in the oxDNA/build directory

Result:
Error message disappeared. But attempts to verify install by running tests (make test) lead to the following:

In oat test:
error message displayed in terminal when testing duplex_finder

Tests for level 'run':
14/14

Tests for 'quick':
12/12

oxpy tests:
0/2

final output in the terminal:
CRITICAL: Not all tests have passed successfully
[100%] Built target test_oxpy
[100%] Built target test

Expected behavior
Installation completes and all relevant tests pass

Desktop (please complete the following information):

• OS: Garuda Linux (arch based distro)
• oxDNA Version [e.g. 3.0.0]
• c compiler is GNU 14.1.1
• CXX compiler is GNU 14.1.1
• cmake version: 3.30.0

Additional Information
Running make install after attempting to fix the bug had some curious output in the terminal.

Summary:
oat and oxpy were found in the user/ .local/lib/python3.12/site-packages (likely a result from attempting to fix the bug from previous attempts). Oat and oxpy is detected to be outside the venv set up within the oxDNA directory. Was advised to Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location from the terminal. Final line output is: Successfully installed oxDNA_analysis_tools-2.3.5 oxpy-3.6.2.dev31+g00504924.d20240716

[lorenzo-rovigatti]

Hi! Can you try using the -DOxpySystemInstall=On flag when calling cmake? (see here for some additional details).

[jinhzhou]

Hello! Thanks so much for replying!

Correct me if I'm wrong, doesn't -DOxpySystemInstall=On install Oxpy on my entire system instead of in the venv?

[lorenzo-rovigatti]

That should be the case if you compile the code with the system's python. With conda (and, as far as I understand, virtualenvs) it should install the packages in your $PATH (see the docs for details).

[ErikPoppleton]

For reasons I don’t understand, pip installs oxpy and oat in $HOME/.local/ which isn’t on your $PATH by default. There’s probably a warning about this in the log print out about this. -DOxpySystemInstall=1 will install it your current environment’s package directory which is system-level if you’re using a system-level Python, but, at least for Conda (never used venv), that will make it install in the current environment.

[jinhzhou]

Ah! Thanks for much for clarifying.

I shall redo the steps listed in the original post while using the DOxpySystemInstall=On flag when calling cmake tomorrow! I will update this post accordingly. Do you suggest I remove the files in the site-packages directory? Also curiously, I am able to run the example CIR_ASSEMBLY without any warnings, or errors.

[ErikPoppleton]

I would remove the copy in .local because if you later add it to your PATH and forget it’s there, it can cause problems with pip if you try to update oxpy (see the last point in this section).

The Python installation happens after compilation of oxDNA is complete. The CIRC_ASSEMBLY example doesn’t require any of the Python features so will run just fine. Only things which use oxpy to manage simulations or oat for analysis will fail if they’re not installed.

[jinhzhou]

Hi! I removed all files inside the /.local/lib directory and performed a clean install while using the DOxpySystemInstall=On flag. No new files were added to /.local/lib while many packages were added to the lib dir of the venv in the oxDNA directory.

Running make test results in some errors.

Notable ones are:

ModuleNotFoundError: No module named 'pkg_resources'
FileNotFoundError: [Errno 2] No such file or directory: 'angles.txt' for duplex_angle_plotter
ModuleNotFoundError: No module named 'distutils'

[lorenzo-rovigatti]

First of all, thanks for trying. Regardless of this particular issue, we need to strengthen the installation of the python modules (which is a nightmare because the python ecosystem sucks big time) and this feedback is very useful.
Unfortunately, I can't fully reproduce the issue. This is the series of commands I used:

cd /tmp
python3 -m venv venv
pip3 uninstall oxpy oxDNA_analysis_tools
source venv/bin/activate
cp ~/git/oxDNA /tmp/venv
cd /tmp/venv/oxDNA
mkdir pybuild
cd pybuild
cmake .. -DPython=On -DOxpySystemInstall=On -DPYTHON_INCLUDE_DIR=/tmp/venv/include/python3.11 -DPYTHON_EXECUTABLE=/tmp/venv/bin/python3
make install -j12

Trying to test the different components of the installation, I run make test_oat, which was fine apart from duplex_finder, which returned a lot of Error in finding common intersection point and a final --> AN ERROR OCCURRED <--. @ErikPoppleton maybe we can have a look?
As for make test_oxpy, it does fail, but oxpy works in itself (e.g. examples/OXPY/run.py runs just fine), so it has to be something else.

[lorenzo-rovigatti]

UPDATE: the problem with oat is due to a problem in oxpy. It looks like nucleotide positions read by python are just [x, x, x] instead of [x, y, z]. I don't know what's going on but there is definitely an issue with using virtualenv to install oxpy. Can someone else give it a try?

[jinhzhou]

Hi! Thanks for the replies!

It seems that you are installing everything within the venv directory when you are running:
cp ~/git/oxDNA /tmp/venv
cd /tmp/venv/oxDNA
mkdir pybuild....

While I installed everything in the oxDNA directory (cloned through https by vscode) containing a venv directory. Could that be causing any issues?

I will try to install it using the commands you listed and update accordingly.

I also encountered the same Error in finding common intersection point error in duplex_finder when I followed the commands listed in the original post!

Oxpy does seem to work fine even with the "non-normal" installation! In that I am able to run CIR_ASSEMBLY along with others in the examples directory. I am also able to use oat.UTILS.RyeReader and oat.UTILS.oxview to produce an image from data outputted by the examples!

[ErikPoppleton]

I am unfortunately on holiday at the moment and don’t have my computer with me to test. However, the only part of oat that gets positions from oxpy is the duplex finder. So assuming that is the only problem, everything else should work. That is a very weird bug though.

[lorenzo-rovigatti]

It seems that you are installing everything within the venv directory when you are running: cp ~/git/oxDNA /tmp/venv cd /tmp/venv/oxDNA mkdir pybuild....

This was done just for convenience. You can put it somewhere else and nothing is going to change.

While I installed everything in the oxDNA directory (cloned through https by vscode) containing a venv directory. Could that be causing any issues?

I don't think so!

I will try to install it using the commands you listed and update accordingly.

Great, let me know how it goes.

I also encountered the same Error in finding common intersection point error in duplex_finder when I followed the commands listed in the original post!

Oxpy does seem to work fine even with the "non-normal" installation! In that I am able to run CIR_ASSEMBLY along with others in the examples directory. I am also able to use oat.UTILS.RyeReader and oat.UTILS.oxview to produce an image from data outputted by the examples!

As Erik said, the CIRC_ASSEMBLY does not use oxpy, but the oxDNA executable. There are two different things, although they use the same underlying library.