New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

running with `crash_safe=true` is much slower #59

Open

Co1lin opened this issue Oct 15, 2022 · 1 comment

Assignees

Labels

Collaborator

Co1lin commented Oct 15, 2022

No description provided.

Member

ganler commented Oct 17, 2022

Mark reasons for process isolation

PyTorch JIT leak => OOM; Running JIT trace for many times leads to OOM pytorch/pytorch#86537
Better oracle: crash & timeout;

crash_safe=True is now compatible with:

onnx-tvm
onnx-ort
onnx-trt
tflite

imcompatible with:

xla: reinitialization of cuda context; Failed setting context: CUDA_ERROR_NOT_INITIALIZED: initialization error at fork tensorflow/tensorflow#57877
pytorch-jit (experimental): performance and randomized haulting;

required by:

pytorch-jit: to route the leak => OOM issue;

Now investigating pytorch compatibility.

ganler self-assigned this

ganler added the enhancement label

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment