Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

Int8_kv_cache run error for whisper #993

Closed
2 of 4 tasks
Eddie-Wang1120 opened this issue Jan 28, 2024 · 1 comment
Closed
2 of 4 tasks

Int8_kv_cache run error for whisper #993

Eddie-Wang1120 opened this issue Jan 28, 2024 · 1 comment
Assignees
@Tracin
Labels
bug Something isn't working stale triaged Issue has been triaged by maintainers

Comments

@Eddie-Wang1120
Copy link
Contributor

Eddie-Wang1120 commented Jan 28, 2024 •
edited
Loading

System Info

  • intel i5 13500
  • nvidia 4060ti 16G
  • Tensorrt-LLM commitID b57221b
  • container nvidia-docker run --entrypoint /bin/bash -it nvidia/cuda:12.1.0-devel-ubuntu22.04

Who can help?

No response

Information

  • The official example scripts
  • My own modified scripts

Tasks

  • An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
  • My own task or dataset (give details below)

Reproduction

steps in README about int8_kv_cache in #992

Expected behavior

run successfully and get right results

actual behavior

  File "/home/TensorRT-LLM/examples/whisper/run.py", line 349, in <module>
    results, total_duration = decode_dataset(
  File "/home/TensorRT-LLM/examples/whisper/run.py", line 332, in decode_dataset
    predictions = model.process_batch(features, text_prefix, num_beams)
  File "/home/TensorRT-LLM/examples/whisper/run.py", line 250, in process_batch
    output_ids = self.decoder.generate(decoder_input_ids,
  File "/home/TensorRT-LLM/examples/whisper/run.py", line 204, in generate
    output_ids = self.decoder_generation_session.decode(
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/runtime/generation.py", line 755, in wrapper
    ret = func(self, *args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/runtime/generation.py", line 2891, in decode
    return self.decode_regular(
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/runtime/generation.py", line 2548, in decode_regular
    should_stop, next_step_tensors, tasks, context_lengths, host_context_lengths, attention_mask, logits, encoder_input_lengths = self.handle_per_step(
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/runtime/generation.py", line 2404, in handle_per_step
    should_stop = self.dynamic_decoder.forward(
RuntimeError: [TensorRT-LLM][ERROR] CUDA runtime error in ::cudaStreamSynchronize(dynamic_decode_layer_->getStream()): unknown error (/home/jenkins/agent/workspace/LLM/main/L0_MergeRequest/tensorrt_llm/cpp/tensorrt_llm/thop/dynamicDecodeOp.cpp:203)
1       0x7fe146754eee void tensorrt_llm::common::check<cudaError>(cudaError, char const*, char const*, int) + 94
2       0x7fe146773c94 torch_ext::FtDynamicDecode<__half>::forward(at::Tensor&, int, int, int, int, unsigned long, int, at::Tensor, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, at::Tensor&, at::Tensor&, at::Tensor&, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, bool) + 1940
3       0x7fe14674fd6f torch_ext::DynamicDecodeOp::forward(at::Tensor, long, long, long, long, long, long, at::Tensor, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, at::Tensor, at::Tensor, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, bool) + 2271
4       0x7fe14676f6de /usr/local/lib/python3.10/dist-packages/tensorrt_llm/libs/libth_common.so(+0x926de) [0x7fe14676f6de]
5       0x7fe14677036f std::_Function_handler<void (std::vector<c10::IValue, std::allocator<c10::IValue> >&), torch::class_<torch_ext::DynamicDecodeOp>::defineMethod<torch::detail::WrapMethod<at::Tensor (torch_ext::DynamicDecodeOp::*)(at::Tensor, long, long, long, long, long, long, at::Tensor, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, at::Tensor, at::Tensor, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, bool)> >(std::string, torch::detail::WrapMethod<at::Tensor (torch_ext::DynamicDecodeOp::*)(at::Tensor, long, long, long, long, long, long, at::Tensor, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, at::Tensor, at::Tensor, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, c10::optional<at::Tensor>, bool)>, std::string, std::initializer_list<torch::arg>)::{lambda(std::vector<c10::IValue, std::allocator<c10::IValue> >&)#1}>::_M_invoke(std::_Any_data const&, std::vector<c10::IValue, std::allocator<c10::IValue> >&) + 31
6       0x7fe31cd6e73e /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_python.so(+0x93f73e) [0x7fe31cd6e73e]
7       0x7fe31ce50f53 /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_python.so(+0xa21f53) [0x7fe31ce50f53]
8       0x7fe31ce0e62a /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_python.so(+0x9df62a) [0x7fe31ce0e62a]
9       0x7fe31ce0e858 /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_python.so(+0x9df858) [0x7fe31ce0e858]
10      0x7fe31c81dbb4 /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_python.so(+0x3eebb4) [0x7fe31c81dbb4]
11      0x555deaedf10e python3(+0x15a10e) [0x555deaedf10e]
12      0x555deaed5a7b _PyObject_MakeTpCall + 603
13      0x555deaeedc20 python3(+0x168c20) [0x555deaeedc20]
14      0x555deb00572b python3(+0x28072b) [0x555deb00572b]
15      0x555deaeee42b PyObject_Call + 187
16      0x555deaeca5d7 _PyEval_EvalFrameDefault + 10791
17      0x555deaeed93e python3(+0x16893e) [0x555deaeed93e]
18      0x555deaeca5d7 _PyEval_EvalFrameDefault + 10791
19      0x555deaeed93e python3(+0x16893e) [0x555deaeed93e]
20      0x555deaeca5d7 _PyEval_EvalFrameDefault + 10791
21      0x555deaedf9fc _PyFunction_Vectorcall + 124
22      0x555deaeee492 PyObject_Call + 290
23      0x555deaeca5d7 _PyEval_EvalFrameDefault + 10791
24      0x555deaeed7f1 python3(+0x1687f1) [0x555deaeed7f1]
25      0x555deaec953c _PyEval_EvalFrameDefault + 6540
26      0x555deaeed7f1 python3(+0x1687f1) [0x555deaeed7f1]
27      0x555deaec953c _PyEval_EvalFrameDefault + 6540
28      0x555deaedf9fc _PyFunction_Vectorcall + 124
29      0x555deaec845c _PyEval_EvalFrameDefault + 2220
30      0x555deaedf9fc _PyFunction_Vectorcall + 124
31      0x555deaec953c _PyEval_EvalFrameDefault + 6540
32      0x555deaec49c6 python3(+0x13f9c6) [0x555deaec49c6]
33      0x555deafba256 PyEval_EvalCode + 134
34      0x555deafe5108 python3(+0x260108) [0x555deafe5108]
35      0x555deafde9cb python3(+0x2599cb) [0x555deafde9cb]
36      0x555deafe4e55 python3(+0x25fe55) [0x555deafe4e55]
37      0x555deafe4338 _PyRun_SimpleFileObject + 424
38      0x555deafe3f83 _PyRun_AnyFileObject + 67
39      0x555deafd6a5e Py_RunMain + 702
40      0x555deafad02d Py_BytesMain + 45
41      0x7fe359bddd90 /usr/lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fe359bddd90]
42      0x7fe359bdde40 __libc_start_main + 128
43      0x555deafacf25 _start + 37
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df69db140'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df6a452b0'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df6a9ec90'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df6afa780'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df6b4ee40'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df6baf3f0'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df6c09240'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df6c62e20'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df6cbb820'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df6d25690'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df6d7f920'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df6dd5b00'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df6e358e0'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df6e8d730'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df6ee7e90'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df6f422c0'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df6f9bcc0'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df6ffee00'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df70594d0'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df70b2f70'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df710d0c0'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df7167740'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df71c1b20'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df721d7a0'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df7277db0'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df72da800'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df7334d30'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df738e310'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df73e7110'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df7441520'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df749ae00'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df74f73a0'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df754f680'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df75b1fe0'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df760c7f0'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df76668b0'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df76c0460'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df771abe0'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df7774db0'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df77cb640'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df782ae40'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df78ebfa0'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df794ad10'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df79a4790'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df79fef80'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df7a5abc0'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df7ab3520'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df7b0f1d0'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df7b69460'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df7bca490'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df7bc5b30'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df7c7e4c0'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df7cda720'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df7d341d0'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df7d88e50'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df7de6e60'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df7e40fc0'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df7ea3ea0'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df7efddb0'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df7f5a190'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df7fae500'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df800ce00'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df806b0a0'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df80c5100'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [graphContext.h::~MyelinGraphContext::55] Error Code 1: Myelin ([impl.cpp:cuda_object_deallocate:345] Error 999 destroying stream '0x555df812d0b0'.)
[01/28/2024-14:51:46] [TRT] [E] 1: [defaultAllocator.cpp::deallocate::62] Error Code 1: Cuda Runtime (unknown error)
[01/28/2024-14:51:46] [TRT] [E] 1: [defaultAllocator.cpp::deallocate::62] Error Code 1: Cuda Runtime (unknown error)
[01/28/2024-14:51:46] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaStream::47] Error Code 1: Cuda Runtime (unknown error)[01/28/2024-14:51:46] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (unknown error)
[01/28/2024-14:51:46] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (unknown error)
[01/28/2024-14:51:46] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (unknown error)
[01/28/2024-14:51:46] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (unknown error)
[01/28/2024-14:51:46] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (unknown error)
[01/28/2024-14:51:46] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (unknown error)
[01/28/2024-14:51:46] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (unknown error)
[01/28/2024-14:51:46] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (unknown error)
[01/28/2024-14:51:46] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (unknown error)
[01/28/2024-14:51:46] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (unknown error)
[01/28/2024-14:51:46] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (unknown error)
[01/28/2024-14:51:46] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (unknown error)
[01/28/2024-14:51:46] [TRT] [E] 1: [cudaResources.cpp::~ScopedCudaEvent::24] Error Code 1: Cuda Runtime (unknown error)
.......

additional notes

It seems like a problem occurs in opreator or driver beacuase it not catched in python.
@Eddie-Wang1120 Eddie-Wang1120 added the bug Something isn't working label Jan 28, 2024
@Eddie-Wang1120 Eddie-Wang1120 mentioned this issue Jan 28, 2024
Closed
@Tracin Tracin added the triaged Issue has been triaged by maintainers label Jan 30, 2024
@nv-guomingz
Copy link
Collaborator

Hi @Eddie-Wang1120 would u please try our latest code base to see if the issue still exists?

And do u still have further issue or question now? If not, we'll close it soon.

# for free to join this conversation on GitHub. Already have an account? # to comment
Assignees

@Tracin Tracin

Labels
bug Something isn't working stale triaged Issue has been triaged by maintainers
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@Tracin @Eddie-Wang1120 @nv-guomingz