Problem with install.py and traintrack

[fialhocoelho]

After run the install.py script the following version of pytorch was installed:

1.8.1+cu102 (instead of version 1.8.0)

When we try to run traintrack from the Pipelines/TrackML_Example/ directory using pytorch 1.8.1+cu102, the following error returns:

from pytorch3d import _C
ImportError: libcudart.so.10.1: cannot open shared object file: No such file or directory

pytorch3d doesn’t work “officially” with pytorch 1.8.x (Work with PyTorch 1.4, 1.5.0, 1.5.1, 1.6.0, 1.7.0, or 1.7.1)

When we try to run traintrack from the Pipelines/TrackML_Example/ directory using pytorch 1.7.1+cu102, the following error returns:

Running stage, with args, and model library: LightningModules
INFO:root:Building model...
INFO:faiss.loader:Loading faiss.
INFO:root:Model found
INFO:root:Model built
Traceback (most recent call last):
  File "/home/jfialho/.conda/envs/exatrkx3/bin/traintrack", line 8, in <module>
    sys.exit(main())
  File "/home/jfialho/.conda/envs/exatrkx3/lib/python3.8/site-packages/traintrack/command_line_pipe.py", line 70, in main
    run_pipeline.start(run_args)
  File "/home/jfialho/.conda/envs/exatrkx3/lib/python3.8/site-packages/traintrack/run_pipeline.py", line 131, in start
    run_stage(**config)
  File "/home/jfialho/.conda/envs/exatrkx3/lib/python3.8/site-packages/traintrack/utils/data_utils.py", line 51, in wrapped
    return dFxn(*cp, **dp)
  File "/home/jfialho/.conda/envs/exatrkx3/lib/python3.8/site-packages/traintrack/run_pipeline.py", line 64, in run_stage
    model = model_class(model_config)
  File "LightningModules/Embedding/Models/layerless_embedding.py", line 21, in __init__
    super().__init__(hparams)
  File "LightningModules/Embedding/embedding_base.py", line 40, in __init__
    self.hparams = hparams
  File "/home/jfialho/.conda/envs/exatrkx3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 826, in __setattr__
    object.__setattr__(self, name, value)
AttributeError: can't set attribute

Packages in environment:

# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main  
absl-py                   0.12.0                   pypi_0    pypi
aiohttp                   3.7.4.post0              pypi_0    pypi
ase                       3.21.1                   pypi_0    pypi
async-timeout             3.0.1                    pypi_0    pypi
attrs                     21.2.0                   pypi_0    pypi
blas                      1.0                         mkl  
ca-certificates           2021.4.13            h06a4308_1  
cachetools                4.2.2                    pypi_0    pypi
certifi                   2020.12.5        py38h06a4308_0  
chardet                   4.0.0                    pypi_0    pypi
cudatoolkit               10.2.89              hfd86e86_1  
cupy-cuda102              9.0.0                    pypi_0    pypi
cycler                    0.10.0                   pypi_0    pypi
decorator                 4.4.2                    pypi_0    pypi
exatrkx-pipeline          0.3.0                     dev_0    <develop>
faiss-gpu                 1.7.0                    pypi_0    pypi
fastrlock                 0.6                      pypi_0    pypi
freetype                  2.10.4               h5ab3b9f_0  
fsspec                    2021.4.0                 pypi_0    pypi
future                    0.18.2                   pypi_0    pypi
fvcore                    0.1.5.post20210511          pypi_0    pypi
google-auth               1.30.0                   pypi_0    pypi
google-auth-oauthlib      0.4.4                    pypi_0    pypi
googledrivedownloader     0.4                      pypi_0    pypi
grpcio                    1.37.1                   pypi_0    pypi
h5py                      3.2.1                    pypi_0    pypi
idna                      2.10                     pypi_0    pypi
intel-openmp              2021.2.0           h06a4308_610  
iopath                    0.1.8                    pypi_0    pypi
isodate                   0.6.0                    pypi_0    pypi
jinja2                    3.0.0                    pypi_0    pypi
joblib                    1.0.1                    pypi_0    pypi
jpeg                      9b                   h024ee3a_2  
kiwisolver                1.3.1                    pypi_0    pypi
lcms2                     2.12                 h3be6417_0  
ld_impl_linux-64          2.33.1               h53a641e_7  
libffi                    3.3                  he6710b0_2  
libgcc-ng                 9.1.0                hdf63c60_0  
libpng                    1.6.37               hbc83047_0  
libstdcxx-ng              9.1.0                hdf63c60_0  
libtiff                   4.1.0                h2733197_1  
libuv                     1.40.0               h7b6447c_0  
llvmlite                  0.36.0                   pypi_0    pypi
lz4-c                     1.9.3                h2531618_0  
markdown                  3.3.4                    pypi_0    pypi
markupsafe                2.0.0                    pypi_0    pypi
matplotlib                3.4.2                    pypi_0    pypi
memory-profiler           0.58.0                   pypi_0    pypi
mkl                       2021.2.0           h06a4308_296  
mkl-service               2.3.0            py38h27cfd23_1  
mkl_fft                   1.3.0            py38h42c9631_2  
mkl_random                1.2.1            py38ha9443f7_2  
more-itertools            8.7.0                    pypi_0    pypi
multidict                 5.1.0                    pypi_0    pypi
ncurses                   6.2                  he6710b0_1  
networkx                  2.5.1                    pypi_0    pypi
ninja                     1.10.2               hff7bd54_1  
numba                     0.53.1                   pypi_0    pypi
numpy                     1.20.1           py38h93e21f0_0  
numpy-base                1.20.1           py38h7d8b39e_0  
oauthlib                  3.1.0                    pypi_0    pypi
olefile                   0.46                       py_0  
openssl                   1.1.1k               h27cfd23_0  
packaging                 20.9                     pypi_0    pypi
pandas                    1.2.4                    pypi_0    pypi
pillow                    8.2.0            py38he98fc37_0  
pip                       21.0.1           py38h06a4308_0  
portalocker               2.3.0                    pypi_0    pypi
protobuf                  3.17.0                   pypi_0    pypi
psutil                    5.8.0                    pypi_0    pypi
pyasn1                    0.4.8                    pypi_0    pypi
pyasn1-modules            0.2.8                    pypi_0    pypi
pydeprecate               0.3.0                    pypi_0    pypi
pyparsing                 2.4.7                    pypi_0    pypi
python                    3.8.8                hdb3f193_5  
python-dateutil           2.8.1                    pypi_0    pypi
python-louvain            0.15                     pypi_0    pypi
pytorch                   1.7.1           py3.8_cuda10.2.89_cudnn7.6.5_0    pytorch
pytorch-lightning         1.3.1                    pypi_0    pypi
pytorch3d                 0.4.0                    pypi_0    pypi
pytz                      2021.1                   pypi_0    pypi
pyyaml                    5.4.1                    pypi_0    pypi
rdflib                    5.0.0                    pypi_0    pypi
readline                  8.1                  h27cfd23_0  
requests                  2.25.1                   pypi_0    pypi
requests-oauthlib         1.3.0                    pypi_0    pypi
rsa                       4.7.2                    pypi_0    pypi
scikit-learn              0.24.2                   pypi_0    pypi
scipy                     1.6.3                    pypi_0    pypi
setuptools                52.0.0           py38h06a4308_0  
simple-slurm              0.1.7                    pypi_0    pypi
six                       1.15.0           py38h06a4308_0  
sklearn                   0.0                      pypi_0    pypi
sqlite                    3.35.4               hdfb4753_0  
tabulate                  0.8.9                    pypi_0    pypi
tensorboard               2.4.1                    pypi_0    pypi
tensorboard-plugin-wit    1.8.0                    pypi_0    pypi
termcolor                 1.1.0                    pypi_0    pypi
threadpoolctl             2.1.0                    pypi_0    pypi
tk                        8.6.10               hbc83047_0  
torch-cluster             1.5.9                    pypi_0    pypi
torch-geometric           1.7.0                    pypi_0    pypi
torch-scatter             2.0.6                    pypi_0    pypi
torch-sparse              0.6.9                    pypi_0    pypi
torch-spline-conv         1.2.1                    pypi_0    pypi
torchaudio                0.7.2                      py38    pytorch
torchmetrics              0.3.2                    pypi_0    pypi
torchvision               0.8.2                py38_cu102    pytorch
tqdm                      4.60.0                   pypi_0    pypi
trackml                   3                        pypi_0    pypi
traintrack                0.1.3                    pypi_0    pypi
typing_extensions         3.7.4.3            pyha847dfd_0  
urllib3                   1.26.4                   pypi_0    pypi
werkzeug                  2.0.0                    pypi_0    pypi
wheel                     0.36.2             pyhd3eb1b0_0  
xz                        5.2.5                h7b6447c_0  
yacs                      0.1.8                    pypi_0    pypi
yarl                      1.6.3                    pypi_0    pypi
zlib                      1.2.11               h7b6447c_3  
zstd                      1.4.9                haebb681_0  

CUDA driver and toolkit version:

$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Wed_Oct_23_19:24:38_PDT_2019
Cuda compilation tools, release 10.2, V10.2.89

[murnanedaniel]

Ah, I'm seeing now that PytorchLightning just updated their codebase last week so that this is no longer the standard way to handle hyperparameters (e.g. see https://discuss.pytorch.org/t/pytorch-lightning-module-cant-set-attribute-error/121125/2).

I will update this library shortly. In the meantime, you can patch it by changing

self.hparams = hparams

to

self.save_hyperparameters(hparams)

in the offending model file(s).