Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

train.py not running #62

Open
2601705041 opened this issue Nov 29, 2023 · 2 comments
Open

train.py not running #62

2601705041 opened this issue Nov 29, 2023 · 2 comments

Comments

@2601705041
Copy link

2601705041 commented Nov 29, 2023

my pytorch is 1.10.0 use this:
conda install pytorch==1.10.0 torchvision==0.11.0 torchaudio==0.10.0 cudatoolkit=11.3 -c pytorch -c conda-forge
because i can only get cpu version use:
conda install pytorch==1.9.1 torchvision==0.10.1 torchaudio==0.9.1 cudatoolkit=11.3 -c pytorch -c conda-forge
i don't konw why

python train.py
Global seed set to 42
Working dir: /home/j222/vid2avatar-main/outputs/Video/parkinglot
GPU available: True, used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
Error executing job with overrides: []
Traceback (most recent call last):
File "train.py", line 33, in main
model = V2AModel(opt)
File "/home/j222/vid2avatar-main/code/v2a_model.py", line 30, in init
self.model = V2A(opt.model, self.betas_path, self.gender, num_training_frames)
File "/home/j222/vid2avatar-main/code/lib/model/v2a.py", line 38, in init
self.deformer = SMPLDeformer(betas=betas, gender=self.gender)
File "/home/j222/vid2avatar-main/code/lib/model/deformer.py", line 12, in init
self.smpl = SMPLServer(gender=gender)
File "/home/j222/vid2avatar-main/code/lib/model/smpl.py", line 44, in init
output = self.forward(*torch.split(self.param_canonical, [1, 3, 72, 10], dim=1), absolute=True)
File "/home/j222/vid2avatar-main/code/lib/model/smpl.py", line 77, in forward
v_template=self.v_template)
File "/home/j222/vid2avatar-main/code/lib/smpl/body_models.py", line 342, in forward
self.lbs_weights, dtype=self.dtype,pose_blend=self.pose_blend)
File "/home/j222/vid2avatar-main/code/lib/smpl/lbs.py", line 180, in lbs
v_shaped = v_template + blend_shapes(betas, shapedirs)
File "/home/j222/vid2avatar-main/code/lib/smpl/lbs.py", line 272, in blend_shapes
blend_shape = torch.einsum('bl,mkl->bmk', [betas, shape_disps])
File "/home/j222/anaconda/envs/v2a/lib/python3.7/site-packages/torch/functional.py", line 325, in einsum
return einsum(equation, *_operands)
File "/home/j222/anaconda/envs/v2a/lib/python3.7/site-packages/torch/functional.py", line 327, in einsum
return _VF.einsum(equation, operands) # type: ignore[attr-defined]
RuntimeError: CUDA error: CUBLAS_STATUS_INVALID_VALUE when calling cublasSgemmStridedBatched( handle, opa, opb, m, n, k, &alpha, a, lda, stridea, b, ldb, strideb, &beta, c, ldc, stridec, num_batches)

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.

@2601705041
Copy link
Author

and here is something might wrong with hydra-core
print(hydra.utils.to_absolute_path('..')) ------> /home/j222/vid2avatar-main/code/..
i think it should be /home/j222/vid2avatar-main

@2601705041
Copy link
Author

well now i get a new environment (cuda 11.1) use this:
pip install torch==1.9.1+cu111 torchvision==0.10.1+cu111 torchaudio==0.9.1 -f https://download.pytorch.org/whl/torch_stable.html

python train.py
Traceback (most recent call last):
File "train.py", line 1, in
from v2a_model import V2AModel
File "/home/j222/vid2avatar-main/code/v2a_model.py", line 3, in
from lib.model.v2a import V2A
File "/home/j222/vid2avatar-main/code/lib/model/v2a.py", line 16, in
import kaolin
File "/home/j222/vid2avatar-main/kaolin/kaolin/init.py", line 1, in
from . import io
File "/home/j222/vid2avatar-main/kaolin/kaolin/io/init.py", line 5, in
from . import render
File "/home/j222/vid2avatar-main/kaolin/kaolin/io/render.py", line 23, in
from ..render.camera import generate_perspective_projection
File "/home/j222/vid2avatar-main/kaolin/kaolin/render/init.py", line 2, in
from . import mesh
File "/home/j222/vid2avatar-main/kaolin/kaolin/render/mesh/init.py", line 1, in
from .utils import *
File "/home/j222/vid2avatar-main/kaolin/kaolin/render/mesh/utils.py", line 22, in
from ... import ops
File "/home/j222/vid2avatar-main/kaolin/kaolin/ops/init.py", line 1, in
from . import batch
File "/home/j222/vid2avatar-main/kaolin/kaolin/ops/batch.py", line 17, in
from kaolin import _C
ImportError: /home/j222/vid2avatar-main/kaolin/kaolin/_C.so: undefined symbol: _ZNK2at10TensorBase8data_ptrIdEEPT_v

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant