验证安装样例运行后出现Fatal process OOM in insufficient memory to create an Isolate #3807

Amoukk · 2020-04-20T00:45:31Z

Amoukk
Apr 20, 2020

问题概述

启动服务器的时候出现了Fatal process OOM in insufficient memory to create an Isolate 内存8G 启动前的时候内存占用也不超60% 但是最后就会弹出OOM

 OS: Windows64
 python version: 3.7.0

具体错误信息

E:\bc>nnictl create --config nni\examples\trials\mnist-tfv1\config_windows.yml
INFO: expand searchSpacePath: search_space.json to E:\bc\nni\examples\trials\mnist-tfv1\search_space.json
INFO: expand codeDir: . to E:\bc\nni\examples\trials\mnist-tfv1\.
INFO: Starting restful server...
ERROR: Restful server start failed!
INFO:  Stdout:
-----------------------------------------------------------------------
                Experiment start time 2020-04-20 08:15:40
-----------------------------------------------------------------------

<--- Last few GCs --->


<--- JS stacktrace --->





INFO:  Stderr:
-----------------------------------------------------------------------
                Experiment start time 2020-04-20 08:15:40
-----------------------------------------------------------------------

#
# Fatal process OOM in insufficient memory to create an Isolate
#


Traceback (most recent call last):
  File "d:\python\lib\site-packages\psutil\_pswindows.py", line 679, in wrapper
    return fun(self, *args, **kwargs)
  File "d:\python\lib\site-packages\psutil\_pswindows.py", line 933, in create_time
    user, system, created = cext.proc_times(self.pid)
ProcessLookupError: [Errno 3] No such process (originated from GetExitCodeProcess != STILL_ACTIVE)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "d:\python\lib\site-packages\psutil\__init__.py", line 373, in _init
    self.create_time()
  File "d:\python\lib\site-packages\psutil\__init__.py", line 723, in create_time
    self._create_time = self._proc.create_time()
  File "d:\python\lib\site-packages\psutil\_pswindows.py", line 681, in wrapper
    raise convert_oserror(err, pid=self.pid, name=self._name)
psutil.NoSuchProcess: psutil.NoSuchProcess process no longer exists (pid=16184)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "d:\python\lib\site-packages\nni_cmd\launcher.py", line 445, in launch_experiment
    kill_command(rest_process.pid)
  File "d:\python\lib\site-packages\nni_cmd\command_utils.py", line 37, in kill_command
    process = psutil.Process(pid=pid)
  File "d:\python\lib\site-packages\psutil\__init__.py", line 346, in __init__
    self._init(pid)
  File "d:\python\lib\site-packages\psutil\__init__.py", line 386, in _init
    raise NoSuchProcess(pid, None, msg)
psutil.NoSuchProcess: psutil.NoSuchProcess no process found with pid 16184

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "d:\python\lib\site-packages\nni_cmd\launcher.py", line 508, in create_experiment
    launch_experiment(args, experiment_config, 'new', config_file_name)
  File "d:\python\lib\site-packages\nni_cmd\launcher.py", line 447, in launch_experiment
    raise Exception(ERROR_INFO % 'Rest server stopped!')
Exception: ERROR: Rest server stopped!

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "d:\python\lib\site-packages\psutil\_pswindows.py", line 679, in wrapper
    return fun(self, *args, **kwargs)
  File "d:\python\lib\site-packages\psutil\_pswindows.py", line 933, in create_time
    user, system, created = cext.proc_times(self.pid)
ProcessLookupError: [Errno 3] No such process (originated from GetExitCodeProcess != STILL_ACTIVE)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "d:\python\lib\site-packages\psutil\__init__.py", line 373, in _init
    self.create_time()
  File "d:\python\lib\site-packages\psutil\__init__.py", line 723, in create_time
    self._create_time = self._proc.create_time()
  File "d:\python\lib\site-packages\psutil\_pswindows.py", line 681, in wrapper
    raise convert_oserror(err, pid=self.pid, name=self._name)
psutil.NoSuchProcess: psutil.NoSuchProcess process no longer exists (pid=16184)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "d:\python\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "d:\python\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "D:\python\Scripts\nnictl.exe\__main__.py", line 7, in <module>
  File "d:\python\lib\site-packages\nni_cmd\nnictl.py", line 223, in parse_args
    args.func(args)
  File "d:\python\lib\site-packages\nni_cmd\launcher.py", line 513, in create_experiment
    kill_command(restServerPid)
  File "d:\python\lib\site-packages\nni_cmd\command_utils.py", line 37, in kill_command
    process = psutil.Process(pid=pid)
  File "d:\python\lib\site-packages\psutil\__init__.py", line 346, in __init__
    self._init(pid)
  File "d:\python\lib\site-packages\psutil\__init__.py", line 386, in _init
    raise NoSuchProcess(pid, None, msg)
psutil.NoSuchProcess: psutil.NoSuchProcess no process found with pid 16184

scarlett2018 · 2020-04-20T02:42:56Z

scarlett2018
Apr 20, 2020
Maintainer

问题概述

启动服务器的时候出现了Fatal process OOM in insufficient memory to create an Isolate 内存8G 启动前的时候内存占用也不超60% 但是最后就会弹出OOM

 OS: Windows64
 python version: 3.7.0

具体错误信息

E:\bc>nnictl create --config nni\examples\trials\mnist-tfv1\config_windows.yml
INFO: expand searchSpacePath: search_space.json to E:\bc\nni\examples\trials\mnist-tfv1\search_space.json
INFO: expand codeDir: . to E:\bc\nni\examples\trials\mnist-tfv1\.
INFO: Starting restful server...
ERROR: Restful server start failed!
INFO:  Stdout:
-----------------------------------------------------------------------
                Experiment start time 2020-04-20 08:15:40
-----------------------------------------------------------------------

<--- Last few GCs --->


<--- JS stacktrace --->





INFO:  Stderr:
-----------------------------------------------------------------------
                Experiment start time 2020-04-20 08:15:40
-----------------------------------------------------------------------

#
# Fatal process OOM in insufficient memory to create an Isolate
#


Traceback (most recent call last):
  File "d:\python\lib\site-packages\psutil\_pswindows.py", line 679, in wrapper
    return fun(self, *args, **kwargs)
  File "d:\python\lib\site-packages\psutil\_pswindows.py", line 933, in create_time
    user, system, created = cext.proc_times(self.pid)
ProcessLookupError: [Errno 3] No such process (originated from GetExitCodeProcess != STILL_ACTIVE)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "d:\python\lib\site-packages\psutil\__init__.py", line 373, in _init
    self.create_time()
  File "d:\python\lib\site-packages\psutil\__init__.py", line 723, in create_time
    self._create_time = self._proc.create_time()
  File "d:\python\lib\site-packages\psutil\_pswindows.py", line 681, in wrapper
    raise convert_oserror(err, pid=self.pid, name=self._name)
psutil.NoSuchProcess: psutil.NoSuchProcess process no longer exists (pid=16184)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "d:\python\lib\site-packages\nni_cmd\launcher.py", line 445, in launch_experiment
    kill_command(rest_process.pid)
  File "d:\python\lib\site-packages\nni_cmd\command_utils.py", line 37, in kill_command
    process = psutil.Process(pid=pid)
  File "d:\python\lib\site-packages\psutil\__init__.py", line 346, in __init__
    self._init(pid)
  File "d:\python\lib\site-packages\psutil\__init__.py", line 386, in _init
    raise NoSuchProcess(pid, None, msg)
psutil.NoSuchProcess: psutil.NoSuchProcess no process found with pid 16184

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "d:\python\lib\site-packages\nni_cmd\launcher.py", line 508, in create_experiment
    launch_experiment(args, experiment_config, 'new', config_file_name)
  File "d:\python\lib\site-packages\nni_cmd\launcher.py", line 447, in launch_experiment
    raise Exception(ERROR_INFO % 'Rest server stopped!')
Exception: ERROR: Rest server stopped!

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "d:\python\lib\site-packages\psutil\_pswindows.py", line 679, in wrapper
    return fun(self, *args, **kwargs)
  File "d:\python\lib\site-packages\psutil\_pswindows.py", line 933, in create_time
    user, system, created = cext.proc_times(self.pid)
ProcessLookupError: [Errno 3] No such process (originated from GetExitCodeProcess != STILL_ACTIVE)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "d:\python\lib\site-packages\psutil\__init__.py", line 373, in _init
    self.create_time()
  File "d:\python\lib\site-packages\psutil\__init__.py", line 723, in create_time
    self._create_time = self._proc.create_time()
  File "d:\python\lib\site-packages\psutil\_pswindows.py", line 681, in wrapper
    raise convert_oserror(err, pid=self.pid, name=self._name)
psutil.NoSuchProcess: psutil.NoSuchProcess process no longer exists (pid=16184)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "d:\python\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "d:\python\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "D:\python\Scripts\nnictl.exe\__main__.py", line 7, in <module>
  File "d:\python\lib\site-packages\nni_cmd\nnictl.py", line 223, in parse_args
    args.func(args)
  File "d:\python\lib\site-packages\nni_cmd\launcher.py", line 513, in create_experiment
    kill_command(restServerPid)
  File "d:\python\lib\site-packages\nni_cmd\command_utils.py", line 37, in kill_command
    process = psutil.Process(pid=pid)
  File "d:\python\lib\site-packages\psutil\__init__.py", line 346, in __init__
    self._init(pid)
  File "d:\python\lib\site-packages\psutil\__init__.py", line 386, in _init
    raise NoSuchProcess(pid, None, msg)
psutil.NoSuchProcess: psutil.NoSuchProcess no process found with pid 16184

Hi @Amou6845 谢谢关注NNI并试用examples，请问你的其他NNI环境信息是什么：

nni Environment:
nni version:
nni mode(local|pai|remote):
OS: Windows64
python version: 3.7.0
is conda or virtualenv used?:
is running in docker?:

Are you able to reproduce? if so, how to reproduce?

0 replies

Amoukk · 2020-04-20T02:56:00Z

Amoukk
Apr 20, 2020
Author

Hi @Amou6845 谢谢关注NNI并试用examples，请问你的其他NNI环境信息是什么：

nni Environment:
nni version:
nni mode(local|pai|remote):
OS: Windows64
python version: 3.7.0
is conda or virtualenv used?:
is running in docker?:

Are you able to reproduce? if so, how to reproduce?

nni Environment:
nni version:v1.5
nni mode(local|pai|remote):local
OS: Windows64
python version: 3.7.0
is conda or virtualenv used?:no
is running in docker?:no

目前还没有找到能够解决的办法还请指导一下如何解决

0 replies

liuzhe-lz · 2020-04-28T08:41:36Z

liuzhe-lz
Apr 28, 2020
Maintainer

请先跟踪一下内存占用情况，确认占用大量内存的是哪个进程。

0 replies

Amoukk · 2020-04-28T09:16:39Z

Amoukk
Apr 28, 2020
Author

请先跟踪一下内存占用情况，确认占用大量内存的是哪个进程。

已经尝试过关闭占用较大的进程让内存保持在占用率50%左右但是启动服务器仍然出现OOM情况启动过程也没有内存占用率的波动

0 replies

liuzhe-lz · 2020-04-28T09:18:08Z

liuzhe-lz
Apr 28, 2020
Maintainer

也就是说启动过程中内存并没有占满但是node.js报OOM了？

0 replies

Amoukk · 2020-04-28T09:21:25Z

Amoukk
Apr 28, 2020
Author

也就是说启动过程中内存并没有占满但是node.js报OOM了？

是的在任务管理器中并没有看见内存占用率变化

0 replies

liuzhe-lz · 2020-04-30T08:38:11Z

liuzhe-lz
Apr 30, 2020
Maintainer

这个看起来比较像运行环境的问题，我们在自己的机器上没有能够复现。
如果方便的话请常试一下在WSL里面运行Linux版NNI。

0 replies

SkyloveQiu · 2020-10-24T19:10:26Z

SkyloveQiu
Oct 24, 2020

我也有同样的问题，请问有人解决了吗。

0 replies

scarlett2018 · 2021-06-10T01:43:40Z

scarlett2018
Jun 10, 2021
Maintainer

@Amoukk @SkyloveQiu - are you still seeing the problem? thanks.

0 replies

SkyloveQiu · 2021-06-10T08:05:52Z

SkyloveQiu
Jun 10, 2021

@scarlett2018 We decided to use WSL for that, the problem was resolved by using wsl

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

验证安装样例运行后出现Fatal process OOM in insufficient memory to create an Isolate #3807

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 10 comments

{{title}}

问题概述

具体错误信息

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

验证安装样例运行后出现Fatal process OOM in insufficient memory to create an Isolate #3807

Amoukk Apr 20, 2020

问题概述

具体错误信息

Replies: 10 comments

scarlett2018 Apr 20, 2020 Maintainer

问题概述

具体错误信息

Amoukk Apr 20, 2020 Author

liuzhe-lz Apr 28, 2020 Maintainer

Amoukk Apr 28, 2020 Author

liuzhe-lz Apr 28, 2020 Maintainer

Amoukk Apr 28, 2020 Author

liuzhe-lz Apr 30, 2020 Maintainer

SkyloveQiu Oct 24, 2020

scarlett2018 Jun 10, 2021 Maintainer

SkyloveQiu Jun 10, 2021

Amoukk
Apr 20, 2020

scarlett2018
Apr 20, 2020
Maintainer

Amoukk
Apr 20, 2020
Author

liuzhe-lz
Apr 28, 2020
Maintainer

Amoukk
Apr 28, 2020
Author

liuzhe-lz
Apr 28, 2020
Maintainer

Amoukk
Apr 28, 2020
Author

liuzhe-lz
Apr 30, 2020
Maintainer

SkyloveQiu
Oct 24, 2020

scarlett2018
Jun 10, 2021
Maintainer

SkyloveQiu
Jun 10, 2021