Automatic installation wants root, and then fails due to pillow dependency

[tsbertalan]

I created a 3.8 virtualenv (at ~/.virtualenvs/PROJECT) for for dojopy, and then ran dojopy.install() there. It asked for root, and proceeded to make another virtualenv under mine, at ~/.virtualenvs/PROJECT/lib/python3.8/site-packages/auto_generated_dojo_env/bin/activate.

The install() process finished, but there was an error in the output above that ending with

File "/tmp/pip-install-zq3kyj_q/pillow/setup.py", line 804, in build_extensions
        raise RequiredDependencyException(f)
    __main__.RequiredDependencyException: jpeg

I though maybe I could fix this by activating that inner virtualenv directly after dectivating mine, and then installing Pillow>=8.0.0 directly. However, the python3 in that bin folder is a symlink to the nonexistent /root/.pyenv/versions/3.8.2/bin/python3, so that environment was broken if I tried to activate it in a fresh terminal (which python points to /usr/bin/python, and ipython fails with a Permission denied.

I created the above PROJECT venv directly with python3 -m venv .... Julia 1.7.2 is installed already in /usr/local/bin/julia, I believe with apt.

Looking a little deeper, decided that it would make more sense to ru n install_dojopy.bash as my user, but then put some sudos on the apt commands used. So I moved the dojopy clone to a new ~/Dropbox/Projects/Dojo WORKING_DIR, and did this diff (see fork):

diff --git a/dojopy/__init__.py b/dojopy/__init__.py
index 43dc227..957dfcc 100644
--- a/dojopy/__init__.py
+++ b/dojopy/__init__.py
@@ -15,7 +15,7 @@ def install(ENV_DIR="", *, confirm=False):
     WORKING_DIR = dirname(dirname(abspath(__file__)))
     FILE_DIR = dirname(abspath(__file__))
     p = subprocess.Popen(
-        ["sudo", "bash", join(FILE_DIR, "install_dojopy.bash"), WORKING_DIR, ENV_DIR],
+        ["bash", join(FILE_DIR, "install_dojopy.bash"), WORKING_DIR, ENV_DIR],
         stdout=subprocess.PIPE,
         stderr=subprocess.STDOUT,
         text=True)
diff --git a/dojopy/install_dojopy.bash b/dojopy/install_dojopy.bash
index 1a04391..ac2126c 100644
--- a/dojopy/install_dojopy.bash
+++ b/dojopy/install_dojopy.bash
@@ -1,6 +1,6 @@
 echo "█████ STEP 0/7 -define arguments"
 # directory where we cloned the dojopy package
-WORKING_DIR=${1:-$HOME/Documents/dojopip}
+WORKING_DIR="/home/tsbertalan/Dropbox/Projects/Dojo/"
 # directory where python environment will be installed
 ENV_DIR=${2:-$WORKING_DIR/auto_generated_dojo_env}
 # directory of the cloned dojopy package
@@ -24,11 +24,11 @@ echo "PY_PATH" $PY_PATH
 
 echo "█████ STEP 1/7 - pyenv installation"
 echo "█████ stage 1 - update and install dependencies"
-apt update -y
+sudo apt update -y
 
 
 echo "█████ stage 2 - install all of pyenv’s dependencies"
-apt install -y make build-essential libssl-dev zlib1g-dev \
+sudo apt install -y make build-essential libssl-dev zlib1g-dev libjpeg-dev \
 > libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev\
 > libncursesw5-dev xz-utils tk-dev libffi-dev liblzma-dev python-openssl\
 > git

I then pip-uninstalled dojopy and reran pip install -e ./dojopy in my env. After this, bash dojopy/dojopy/install_dojopy.bash seemed to work, but if I deactivate, source auto_generated_dojo_env/bin/activate, and run the sample pendulum 1-step script, I get

Traceback (most recent call last):
  File "test_dojopy.py", line 2, in <module>
    from julia import Base
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 618, in _load_backward_compatible
  File "/home/tsbertalan/Dropbox/Projects/Dojo/auto_generated_dojo_env/lib/python3.8/site-packages/julia/core.py", line 248, in load_module
    elif self.julia.isafunction(juliapath):
  File "/home/tsbertalan/Dropbox/Projects/Dojo/auto_generated_dojo_env/lib/python3.8/site-packages/julia/core.py", line 239, in julia
    self.__class__.julia = julia = Julia()
  File "/home/tsbertalan/Dropbox/Projects/Dojo/auto_generated_dojo_env/lib/python3.8/site-packages/julia/core.py", line 483, in __init__
    raise UnsupportedPythonError(jlinfo)
julia.core.UnsupportedPythonError: It seems your Julia and PyJulia setup are not supported.

Julia executable:
    julia
Python interpreter and libpython used by PyCall.jl:
    None
    None
Python interpreter used to import PyJulia and its libpython.
    /home/tsbertalan/Dropbox/Projects/Dojo/auto_generated_dojo_env/bin/python
    /home/tsbertalan/.pyenv/versions/3.8.2/lib/libpython3.8.so.1.0

In Julia >= 0.7, above two paths to `libpython` have to match exactly
in order for PyJulia to work out-of-the-box.  To configure PyCall.jl to use
Python interpreter "/home/tsbertalan/Dropbox/Projects/Dojo/auto_generated_dojo_env/bin/python",
run the following code in the Python REPL:

    >>> import julia
    >>> julia.install()

For more information, see:

    https://pyjulia.readthedocs.io/en/latest/troubleshooting.html

But there is a bird in the office now, so I'll need to go home and continue this troubleshooting tomorrow.

[tsbertalan]

I guess the bird left sometime last night. It looks like

from julia import Julia, install
install()
jl = Julia()

helps. (It looks like this should have been run by install_dojopy.bash via setup_pyjulia.py; not sure why that part didn't fire. I added set -e to my install_dojopy.bash to find out next time, and hardcoded all argument paths). Now the error from the test script is just

Traceback (most recent call last):
  File "test_dojopy.py", line 3, in <module>
    from julia import Dojo as dojo
  File "/home/tsbertalan/Dropbox/Projects/Dojo/auto_generated_dojo_env/lib/python3.8/site-packages/julia/core.py", line 260, in load_module
    raise ImportError("{} not found".format(juliapath))
ImportError: Dojo not found

So, I did the Dojo installation instructions in a Julia REPL

Dojo can be added via the Julia package manager (type ]):

pkg> add Dojo

I added this in setup_julia.py:

def install(*, confirm=False):
    ######################
    ### EXISTING CODE HERE
    ######################
    julia = _find_julia()
    subprocess.check_output(
        julia,
        '-e',
        'Pkg.add("Dojo")',
    )

This results in several very impressive unicode progress bars being displayed in the terminal for a few minutes.

After this, my auto_generated_dojo_env mysteriously didn't work (~/.pyenv/versions was somehow empty), but, when I blew away auto_generated_dojo_env and again re-ran bash install_dojopy.bash and re-sourced auto_generated_dojo_env/bin/activate, I at least got a reasonable error that nu wasn't defined.

It seems I can fix this by commenting out the Base.rand stuff and instead doing import numpy as np; u1 = np.random.uniform(size=(1,)). However, if I run the resulting script (see e40e622 on my fork for all modified scripts) (whether in terminal or copy-pasting into ipython), it finishes (slowly) without errors, but I am unable to open the MeshCat server that it claims is running at http://127.0.0.1:8700.