Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Int8_kv_cache run error for whisper #993

Closed
2 of 4 tasks
Eddie-Wang1120 opened this issue Jan 28, 2024 · 1 comment
Closed
2 of 4 tasks

Int8_kv_cache run error for whisper #993

Eddie-Wang1120 opened this issue Jan 28, 2024 · 1 comment
Assignees
@Tracin
Labels
bug Something isn't working stale triaged Issue has been triaged by maintainers

Comments

@Eddie-Wang1120
Copy link
Contributor

Eddie-Wang1120 commented Jan 28, 2024 •
edited
Loading

System Info

  • intel i5 13500
  • nvidia 4060ti 16G
  • Tensorrt-LLM commitID b57221b
  • container nvidia-docker run --entrypoint /bin/bash -it nvidia/cuda:12.1.0-devel-ubuntu22.04

Who can help?

No response

Information

  • The official example scripts
  • My own modified scripts

Tasks

  • An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
  • My own task or dataset (give details below)

Reproduction

steps in README about int8_kv_cache in #992

Expected behavior

run successfully and get right results

actual behavior

  File "/home/TensorRT-LLM/examples/whisper/run.py", line 349, in <module>
    results, total_duration = decode_dataset(
  File "/home/TensorRT-LLM/examples/whisper/run.py", line 332, in decode_dataset
    predictions = model.process_batch(features, text_prefix, num_beams)
  File "/home/TensorRT-LLM/examples/whisper/run.py", line 250, in process_batch
    output_ids = self.decoder.generate(decoder_input_ids,
  File "/home/TensorRT-LLM/examples/whisper/run.py", line 204, in generate
    output_ids = self.decoder_generation_session.decode(
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/runtime/generation.py", line 755, in wrapper
    ret = func(self, *args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/runtime/generation.py", line 2891, in decode
    return self.decode_regular(
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/runtime/generation.py", line 2548, in decode_regular
    should_stop, next_step_tensors, tasks, context_lengths, host_context_lengths, attention_mask, logits, encoder_input_lengths = self.handle_per_step(
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/runtime/generation.py", line 2404, in handle_per_step
    should_stop = self.dynamic_decoder.forward(
RuntimeError: [TensorRT-LLM][ERROR] CUDA runtime error in ::cudaStreamSynchronize(dynamic_decode_layer_->getStream()): unknown error (/home/jenkins/agent/workspace/LLM/main/L0_MergeRequest/tensorrt_llm/cpp/tensorrt_llm/thop/dynamicDecodeOp.cpp:203)
1       0x7fe146754eee void tensorrt_llm::common::check<cudaError>(cudaError, char const*, char const*, int) + 94
2       0x7fe146773c94 torch_ext::FtDynamicDecode<__half>::forward(at::Tensor&, int, int, int, int, unsigned long, int, at::Tensor, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, at::Tensor&, at::Tensor&, at::Tensor&, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, bool) + 1940
3       0x7fe14674fd6f torch_ext::DynamicDecodeOp::forward(at::Tensor, long, long, long, long, long, long, at::Tensor, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, at::Tensor, at::Tensor, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, bool) + 2271
4       0x7fe14676f6de /usr/local/lib/python3.10/dist-packages/tensorrt_llm/libs/libth_common.so(+0x926de) [0x7fe14676f6de]
5       0x7fe14677036f std::_Function_handler<void (std::vector<c10::IValue, std::allocator<c10::IValue> >&), torch::class_<torch_ext::DynamicDecodeOp>::defineMethod<torch::detail::WrapMethod<at::Tensor (torch_ext::DynamicDecodeOp::*)(at::Tensor, long, long, long, long, long, long, at::Tensor, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, at::Tensor, at::Tensor, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, bool)> >(std::string, torch::detail::WrapMethod<at::Tensor (torch_ext::DynamicDecodeOp::*)(at::Tensor, long, long, long, long, long, long, at::Tensor, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, at::Tensor, at::Tensor, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, bool)>, std::string, std::initializer_list<torch::arg>)::{lambda(std::vector<c10::IValue, std::allocator<c10::IValue> >&)#1}>::_M_invoke(std::_Any_data const&, std::vector<c10::IValue, std::allocator<c10::IValue> >&) + 31
6       0x7fe31cd6e73e /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_python.so(+0x93f73e) [0x7fe31cd6e73e]
7       0x7fe31ce50f53 /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_python.so(+0xa21f53) [0x7fe31ce50f53]
8       0x7fe31ce0e62a /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_python.so(+0x9df62a) [0x7fe31ce0e62a]
9       0x7fe31ce0e858 /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_python.so(+0x9df858) [0x7fe31ce0e858]
10      0x7fe31c81dbb4 /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_python.so(+0x3eebb4) [0x7fe31c81dbb4]
11      0x555deaedf10e python3(+0x15a10e) [0x555deaedf10e]
12      0x555deaed5a7b _PyObject_MakeTpCall + 603
13      0x555deaeedc20 python3(+0x168c20) [0x555deaeedc20]
14      0x555deb00572b python3(+0x28072b) [0x555deb00572b]
15      0x555deaeee42b PyObject_Call + 187
16      0x555deaeca5d7 _PyEval_EvalFrameDefault + 10791
17      0x555deaeed93e python3(+0x16893e) [0x555deaeed93e]
18      0x555deaeca5d7 _PyEval_EvalFrameDefault + 10791
19      0x555deaeed93e python3(+0x16893e) [0x555deaeed93e]
20      0x555deaeca5d7 _PyEval_EvalFrameDefault + 10791
21      0x555deaedf9fc _PyFunction_Vectorcall + 124
22      0x555deaeee492 PyObject_Call + 290
23      0x555deaeca5d7 _PyEval_EvalFrameDefault + 10791
24      0x555deaeed7f1 python3(+0x1687f1) [0x555deaeed7f1]
25      0x555deaec953c _PyEval_EvalFrameDefault + 6540
26      0x555deaeed7f1 python3(+0x1687f1) [0x555deaeed7f1]
27      0x555deaec953c _PyEval_EvalFrameDefault + 6540
28      0x555deaedf9fc _PyFunction_Vectorcall + 124
29      0x555deaec845c _PyEval_EvalFrameDefault + 2220
30      0x555deaedf9fc _PyFunction_Vectorcall + 124
31      0x555deaec953c _PyEval_EvalFrameDefault + 6540
32      0x555deaec49c6 python3(+0x13f9c6) [0x555deaec49c6]
33      0x555deafba256 PyEval_EvalCode + 134
34      0x555deafe5108 python3(+0x260108) [0x555deafe5108]
35      0x555deafde9cb python3(+0x2599cb) [0x555deafde9cb]
36      0x555deafe4e55 python3(+0x25fe55) [0x555deafe4e55]
37      0x555deafe4338 _PyRun_SimpleFileObject + 424
38      0x555deafe3f83 _PyRun_AnyFileObject + 67
39      0x555deafd6a5e Py_RunMain + 702
40      0x555deafad02d Py_BytesMain + 45
41      0x7fe359bddd90 /usr/lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fe359bddd90]
42      0x7fe359bdde40 __libc_start_main + 128
43      0x555deafacf25 _start + 37
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df69db140'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df6a452b0'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df6a9ec90'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df6afa780'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df6b4ee40'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df6baf3f0'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df6c09240'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df6c62e20'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df6cbb820'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df6d25690'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df6d7f920'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df6dd5b00'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df6e358e0'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df6e8d730'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df6ee7e90'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df6f422c0'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df6f9bcc0'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df6ffee00'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df70594d0'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df70b2f70'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df710d0c0'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df7167740'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df71c1b20'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df721d7a0'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df7277db0'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df72da800'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df7334d30'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df738e310'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df73e7110'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df7441520'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df749ae00'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df74f73a0'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df754f680'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df75b1fe0'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df760c7f0'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df76668b0'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df76c0460'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df771abe0'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df7774db0'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df77cb640'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df782ae40'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df78ebfa0'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df794ad10'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df79a4790'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df79fef80'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df7a5abc0'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df7ab3520'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df7b0f1d0'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df7b69460'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df7bca490'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df7bc5b30'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df7c7e4c0'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df7cda720'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df7d341d0'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df7d88e50'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df7de6e60'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df7e40fc0'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df7ea3ea0'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df7efddb0'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df7f5a190'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df7fae500'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df800ce00'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df806b0a0'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df80c5100'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df812d0b0'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [defaultAllocator.cpp::deallocate::62] Error Code 1: Cuda Runtime (unknown error)
[01/28/2024-14:51:46] [TRT] [E] 1: [defaultAllocator.cpp::deallocate::62] Error Code 1: Cuda Runtime (unknown error)
[01/28/2024-14:51:46] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaStream::47] Error Code 1: Cuda Runtime (unknown error)[01/28/2024-14:51:46] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (unknown error)
[01/28/2024-14:51:46] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (unknown error)
[01/28/2024-14:51:46] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (unknown error)
[01/28/2024-14:51:46] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (unknown error)
[01/28/2024-14:51:46] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (unknown error)
[01/28/2024-14:51:46] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (unknown error)
[01/28/2024-14:51:46] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (unknown error)
[01/28/2024-14:51:46] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (unknown error)
[01/28/2024-14:51:46] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (unknown error)
[01/28/2024-14:51:46] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (unknown error)
[01/28/2024-14:51:46] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (unknown error)
[01/28/2024-14:51:46] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (unknown error)
[01/28/2024-14:51:46] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (unknown error)
.......

additional notes

It seems like a problem occurs in opreator or driver beacuase it not catched in python.
@Eddie-Wang1120 Eddie-Wang1120 added the bug Something isn't working label Jan 28, 2024
@Eddie-Wang1120 Eddie-Wang1120 mentioned this issue Jan 28, 2024
Closed
@Tracin Tracin added the triaged Issue has been triaged by maintainers label Jan 30, 2024
@nv-guomingz
Copy link
Collaborator

Hi @Eddie-Wang1120 would u please try our latest code base to see if the issue still exists?

And do u still have further issue or question now? If not, we'll close it soon.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Tracin Tracin

Labels
bug Something isn't working stale triaged Issue has been triaged by maintainers
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@Tracin @Eddie-Wang1120 @nv-guomingz