Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PGL 数据并行加载 #474

Open
wuyunli opened this issue Oct 8, 2022 · 0 comments
Open

PGL 数据并行加载 #474

wuyunli opened this issue Oct 8, 2022 · 0 comments

Comments

@wuyunli
Copy link

wuyunli commented Oct 8, 2022

百度团队你好,
我使用 pgl 遇到一些困难,希望你们赐教, 具体地, 使用 pgl 处理数据的方法封装数据

from pgl.utils.data import Dataset
使用 spawn 并行加载 数据时
from paddle.io import DataLoader
报错如下

Traceback (most recent call last):
File "/home/wjq/crtnet/crtnet/sign/pctrain.py", line 170, in
dist.spawn(train(args, model, trn_loader, tst_loader, val_loader))
File "/home/wjq/crtnet/crtnet/sign/pctrain.py", line 72, in train
for i, batch_data in enumerate(trn_loader):
File "/home/wjq/anaconda3/lib/python3.9/site-packages/paddle/fluid/reader.py", line 566, in iter
return _DataLoaderIterMultiProcess(self)
File "/home/wjq/anaconda3/lib/python3.9/site-packages/paddle/fluid/dataloader/dataloader_iter.py", line 383, in init
self._init_thread()
File "/home/wjq/anaconda3/lib/python3.9/site-packages/paddle/fluid/dataloader/dataloader_iter.py", line 431, in _init_thread
self._var_names = [v.name for v in self._feed_list]
TypeError: 'int' object is not iterable
Exception ignored in: <function _DataLoaderIterMultiProcess.del at 0x7f8a38268160>
Traceback (most recent call last):
File "/home/wjq/anaconda3/lib/python3.9/site-packages/paddle/fluid/dataloader/dataloader_iter.py", line 712, in del
self._try_shutdown_all()
File "/home/wjq/anaconda3/lib/python3.9/site-packages/paddle/fluid/dataloader/dataloader_iter.py", line 503, in _try_shutdown_all
if not self._shutdown:
AttributeError: '_DataLoaderIterMultiProcess' object has no attribute '_shutdown'

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant