Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

昇腾310P3测试demo运行后报错并卡住 #10573

Open
ly303550688 opened this issue Oct 13, 2024 · 4 comments
Open

昇腾310P3测试demo运行后报错并卡住 #10573

ly303550688 opened this issue Oct 13, 2024 · 4 comments
Assignees
@MuShangCC

Comments

@ly303550688
Copy link

  • 版本、预测库信息：
       1）Paddle Lite 版本：最新版本编译
       2）Host 环境：容器，镜像registry.baidubce.com/device/paddle-npu:cann80RC1-ubuntu20-aarch64-gcc84-py39
       3）运行设备环境：鲲鹏920+310P3
  • 预测信息
       1）预测 API：官方测试demo
       2）预测选项信息：arm+ascend
    [ERROR] TBE(1094707,python3):2024-10-13-23:45:48.186.394 [../../../../../../latest/python/site-packages/tbe/common/repository_manager/utils/repository_manager_log.py:30][log] [../../../../../../latest/python/site-packages/tbe/common/repository_manager/route.py:63][repository_manager] Subprocess[task_distribute] raise error[]
    [ERROR] TBE(1094711,python3):2024-10-13-23:45:48.186.430 [../../../../../../latest/python/site-packages/tbe/common/repository_manager/utils/repository_manager_log.py:30][log] [../../../../../../latest/python/site-packages/tbe/common/repository_manager/route.py:63][repository_manager] Subprocess[task_distribute] raise error[]
    [ERROR] TBE(1094706,python3):2024-10-13-23:45:48.186.412 [../../../../../../latest/python/site-packages/tbe/common/repository_manager/utils/repository_manager_log.py:30][log] [../../../../../../latest/python/site-packages/tbe/common/repository_manager/route.py:63][repository_manager] Subprocess[task_distribute] raise error[]
    [ERROR] TBE(1094710,python3):2024-10-13-23:45:48.186.430 [../../../../../../latest/python/site-packages/tbe/common/repository_manager/utils/repository_manager_log.py:30][log] [../../../../../../latest/python/site-packages/tbe/common/repository_manager/route.py:63][repository_manager] Subprocess[task_distribute] raise error[]
    [ERROR] TBE(1094705,python3):2024-10-13-23:45:48.186.424 [../../../../../../latest/python/site-packages/tbe/common/repository_manager/utils/repository_manager_log.py:30][log] [../../../../../../latest/python/site-packages/tbe/common/repository_manager/route.py:63][repository_manager] Subprocess[task_distribute] raise error[]
    [ERROR] TBE(1094704,python3):2024-10-13-23:45:48.186.437 [../../../../../../latest/python/site-packages/tbe/common/repository_manager/utils/repository_manager_log.py:30][log] [../../../../../../latest/python/site-packages/tbe/common/repository_manager/route.py:63][repository_manager] Subprocess[task_distribute] raise error[]
    [ERROR] TBE(1094708,python3):2024-10-13-23:45:48.186.412 [../../../../../../latest/python/site-packages/tbe/common/repository_manager/utils/repository_manager_log.py:30][log] [../../../../../../latest/python/site-packages/tbe/common/repository_manager/route.py:63][repository_manager] Subprocess[task_distribute] raise error[]
    [ERROR] TBE(1094709,python3):2024-10-13-23:45:48.186.432 [../../../../../../latest/python/site-packages/tbe/common/repository_manager/utils/repository_manager_log.py:30][log] [../../../../../../latest/python/site-packages/tbe/common/repository_manager/route.py:63][repository_manager] Subprocess[task_distribute] raise error[]
    Process ForkServerPoolWorker-3:
    Process ForkServerPoolWorker-9:
    Process ForkServerPoolWorker-6:
    Process ForkServerPoolWorker-7:
    Process ForkServerPoolWorker-2:
    Process ForkServerPoolWorker-4:
    Process ForkServerPoolWorker-5:
    Process ForkServerPoolWorker-8:
    Traceback (most recent call last):
    Traceback (most recent call last):
    Traceback (most recent call last):
    Traceback (most recent call last):
    File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/pool.py", line 131, in worker
    put((job, i, result))
    Traceback (most recent call last):
    File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/pool.py", line 131, in worker
    put((job, i, result))
    File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/queues.py", line 377, in put
    self._writer.send_bytes(obj)
    File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/pool.py", line 131, in worker
    put((job, i, result))
    File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
    Traceback (most recent call last):
    Traceback (most recent call last):
    File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/queues.py", line 377, in put
    self._writer.send_bytes(obj)
    File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/pool.py", line 131, in worker
    put((job, i, result))
    File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/queues.py", line 377, in put
    self._writer.send_bytes(obj)
    File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 411, in _send_bytes
    self._send(header + buf)
    File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/pool.py", line 131, in worker
    put((job, i, result))
    File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
    File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/queues.py", line 377, in put
    self._writer.send_bytes(obj)
    File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
    File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/pool.py", line 131, in worker
    put((job, i, result))
    File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/pool.py", line 131, in worker
    put((job, i, result))
    File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 368, in _send
    n = write(self._handle, buf)
    File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/queues.py", line 377, in put
    self._writer.send_bytes(obj)
    File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 411, in _send_bytes
    self._send(header + buf)
    File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
    File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 411, in _send_bytes
    self._send(header + buf)
    File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/queues.py", line 377, in put
    self._writer.send_bytes(obj)
    File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/queues.py", line 377, in put
    self._writer.send_bytes(obj)
    File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
    BrokenPipeError: [Errno 32] Broken pipe
    File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 368, in _send
    n = write(self._handle, buf)
    File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 411, in _send_bytes
    self._send(header + buf)
    Traceback (most recent call last):
    File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 368, in _send
    n = write(self._handle, buf)
    File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
    File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
    File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 411, in _send_bytes
    self._send(header + buf)
    File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 411, in _send_bytes
    self._send(header + buf)

During handling of the above exception, another exception occurred:

File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 368, in _send
n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe
BrokenPipeError: [Errno 32] Broken pipe
File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 411, in _send_bytes
self._send(header + buf)
File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 368, in _send
n = write(self._handle, buf)
File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/pool.py", line 131, in worker
put((job, i, result))
File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 368, in _send
n = write(self._handle, buf)
Traceback (most recent call last):

During handling of the above exception, another exception occurred:

BrokenPipeError: [Errno 32] Broken pipe

During handling of the above exception, another exception occurred:

File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 368, in _send
n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe
File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/queues.py", line 377, in put
self._writer.send_bytes(obj)
Traceback (most recent call last):
BrokenPipeError: [Errno 32] Broken pipe

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()

During handling of the above exception, another exception occurred:

BrokenPipeError: [Errno 32] Broken pipe
File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 200, in send_bytes
self._send_bytes(m[offset:offset + size])

During handling of the above exception, another exception occurred:

File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 411, in _send_bytes
self._send(header + buf)
Traceback (most recent call last):
File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
Traceback (most recent call last):
File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 368, in _send
n = write(self._handle, buf)
File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/pool.py", line 136, in worker
put((job, i, (False, wrapped)))
File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
Traceback (most recent call last):
File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/queues.py", line 377, in put
self._writer.send_bytes(obj)
File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/pool.py", line 136, in worker
put((job, i, (False, wrapped)))
File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 200, in send_bytes
self._send_bytes(m[offset:offset + size])
BrokenPipeError: [Errno 32] Broken pipe
File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/pool.py", line 136, in worker
put((job, i, (False, wrapped)))
File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/pool.py", line 136, in worker
put((job, i, (False, wrapped)))
File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/queues.py", line 377, in put
self._writer.send_bytes(obj)
File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 411, in _send_bytes
self._send(header + buf)
File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()

During handling of the above exception, another exception occurred:

File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/pool.py", line 136, in worker
put((job, i, (False, wrapped)))
File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/queues.py", line 377, in put
self._writer.send_bytes(obj)
File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/queues.py", line 377, in put
self._writer.send_bytes(obj)
File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 200, in send_bytes
self._send_bytes(m[offset:offset + size])
File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 368, in _send
n = write(self._handle, buf)
File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
Traceback (most recent call last):
File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/queues.py", line 377, in put
self._writer.send_bytes(obj)
File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 200, in send_bytes
self._send_bytes(m[offset:offset + size])
File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/pool.py", line 136, in worker
put((job, i, (False, wrapped)))
File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 200, in send_bytes
self._send_bytes(m[offset:offset + size])
File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 411, in _send_bytes
self._send(header + buf)
BrokenPipeError: [Errno 32] Broken pipe
File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/pool.py", line 136, in worker
put((job, i, (False, wrapped)))
File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 200, in send_bytes
self._send_bytes(m[offset:offset + size])
File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 411, in _send_bytes
self._send(header + buf)
File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/queues.py", line 377, in put
self._writer.send_bytes(obj)
File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 411, in _send_bytes
self._send(header + buf)
File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 368, in _send
n = write(self._handle, buf)
File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/queues.py", line 377, in put
self._writer.send_bytes(obj)
File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 411, in _send_bytes
self._send(header + buf)
File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 368, in _send
n = write(self._handle, buf)
File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 200, in send_bytes
self._send_bytes(m[offset:offset + size])
File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 368, in _send
n = write(self._handle, buf)
File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
BrokenPipeError: [Errno 32] Broken pipe
File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 200, in send_bytes
self._send_bytes(m[offset:offset + size])
File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 368, in _send
n = write(self._handle, buf)
File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 411, in _send_bytes
self._send(header + buf)
BrokenPipeError: [Errno 32] Broken pipe
BrokenPipeError: [Errno 32] Broken pipe
File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/pool.py", line 136, in worker
put((job, i, (False, wrapped)))
File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 411, in _send_bytes
self._send(header + buf)
File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 368, in _send
n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe
File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/queues.py", line 377, in put
self._writer.send_bytes(obj)
File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 368, in _send
n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe
File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 200, in send_bytes
self._send_bytes(m[offset:offset + size])
BrokenPipeError: [Errno 32] Broken pipe
File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 411, in _send_bytes
self._send(header + buf)
File "/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/connection.py", line 368, in _send
n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe
/root/miniconda3/envs/paddle/lib/python3.9/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 41 leaked semaphore objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '
@ly303550688
Copy link
Author

#10131
类似的问题

@ly303550688
Copy link
Author

./run.sh mobilenet_v1_fp32_224 imagenet_224.txt test linux arm64 huawei_ascend_npu HUAWEI_ASCEND_NPU_SELECTED_DEVICE_IDS=0 cache
使用缓存方式，第二次没问题了，第一次会卡死

@MuShangCC
Copy link
Collaborator

现在使用缓存方式之后是没有报错吗

./run.sh mobilenet_v1_fp32_224 imagenet_224.txt test linux arm64 huawei_ascend_npu HUAWEI_ASCEND_NPU_SELECTED_DEVICE_IDS=0 cache
使用缓存方式，第二次没问题了，第一次会卡死

@ly303550688
Copy link
Author

使用缓存第一次会卡死，生成缓存之后再重启程序就没问题

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@MuShangCC MuShangCC

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@ly303550688 @MuShangCC