Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

sensevoice convert onnx to triton fail #2226

Open
didadida-r opened this issue Nov 22, 2024 · 0 comments
Open

sensevoice convert onnx to triton fail #2226

didadida-r opened this issue Nov 22, 2024 · 0 comments
Labels
bug Something isn't working

Comments

@didadida-r
Copy link

Notice: In order to resolve issues more efficiently, please raise issue following the template.
（注意：为了更加高效率解决您遇到的问题，请按照模板提问，补充细节）

🐛 Bug

To Reproduce

Steps to reproduce the behavior (always include the command you ran):

  1. Run cmd '....'
TensorRT.trtexec [TensorRT v8502] # /usr/src/tensorrt/bin/trtexec --onnx=./model_repo_sense_voice_small/encoder/1/model.onnx --saveEngine=model.plan --minShapes=speech:1x8x560,speech_lengths:1,language:1,textnorm:1 --optShapes=speech:4x128x560,speech_lengths:4,language:4,textnorm:4 --maxShapes=speech:16x512x560,speech_lengths:16,language:16,textnorm:16
  1. See error
[11/22/2024-07:40:48] [I] === Model Options ===
[11/22/2024-07:40:48] [I] Format: ONNX
[11/22/2024-07:40:48] [I] Model: ./model_repo_sense_voice_small/encoder/1/model.onnx
[11/22/2024-07:40:48] [I] Output:
[11/22/2024-07:40:48] [I] === Build Options ===
[11/22/2024-07:40:48] [I] Max batch: explicit batch
[11/22/2024-07:40:48] [I] Memory Pools: workspace: default, dlaSRAM: default, dlaLocalDRAM: default, dlaGlobalDRAM: default
[11/22/2024-07:40:48] [I] minTiming: 1
[11/22/2024-07:40:48] [I] avgTiming: 8
[11/22/2024-07:40:48] [I] Precision: FP32
[11/22/2024-07:40:48] [I] LayerPrecisions:
[11/22/2024-07:40:48] [I] Calibration:
[11/22/2024-07:40:48] [I] Refit: Disabled
[11/22/2024-07:40:48] [I] Sparsity: Disabled
[11/22/2024-07:40:48] [I] Safe mode: Disabled
[11/22/2024-07:40:48] [I] DirectIO mode: Disabled
[11/22/2024-07:40:48] [I] Restricted mode: Disabled
[11/22/2024-07:40:48] [I] Build only: Disabled
[11/22/2024-07:40:48] [I] Save engine: model.plan
[11/22/2024-07:40:48] [I] Load engine:
[11/22/2024-07:40:48] [I] Profiling verbosity: 0
[11/22/2024-07:40:48] [I] Tactic sources: Using default tactic sources
[11/22/2024-07:40:48] [I] timingCacheMode: local
[11/22/2024-07:40:48] [I] timingCacheFile:
[11/22/2024-07:40:48] [I] Heuristic: Disabled
[11/22/2024-07:40:48] [I] Preview Features: Use default preview flags.
[11/22/2024-07:40:48] [I] Input(s)s format: fp32:CHW
[11/22/2024-07:40:48] [I] Output(s)s format: fp32:CHW
[11/22/2024-07:40:48] [I] Input build shape: speech=1x8x560+4x128x560+16x512x560
[11/22/2024-07:40:48] [I] Input build shape: speech_lengths=1+4+16
[11/22/2024-07:40:48] [I] Input build shape: textnorm=1+4+16
[11/22/2024-07:40:48] [I] Input build shape: language=1+4+16
[11/22/2024-07:40:48] [I] Input calibration shapes: model
[11/22/2024-07:40:48] [I] === System Options ===
[11/22/2024-07:40:48] [I] Device: 0
[11/22/2024-07:40:48] [I] DLACore:
[11/22/2024-07:40:48] [I] Plugins:
[11/22/2024-07:40:48] [I] === Inference Options ===
[11/22/2024-07:40:48] [I] Batch: Explicit
[11/22/2024-07:40:48] [I] Input inference shape: language=4
[11/22/2024-07:40:48] [I] Input inference shape: textnorm=4
[11/22/2024-07:40:48] [I] Input inference shape: speech_lengths=4
[11/22/2024-07:40:48] [I] Input inference shape: speech=4x128x560
[11/22/2024-07:40:48] [I] Iterations: 10
[11/22/2024-07:40:48] [I] Duration: 3s (+ 200ms warm up)
[11/22/2024-07:40:48] [I] Sleep time: 0ms
[11/22/2024-07:40:48] [I] Idle time: 0ms
[11/22/2024-07:40:48] [I] Streams: 1
[11/22/2024-07:40:48] [I] ExposeDMA: Disabled
[11/22/2024-07:40:48] [I] Data transfers: Enabled
[11/22/2024-07:40:48] [I] Spin-wait: Disabled
[11/22/2024-07:40:48] [I] Multithreading: Disabled
[11/22/2024-07:40:48] [I] CUDA Graph: Disabled
[11/22/2024-07:40:48] [I] Separate profiling: Disabled
[11/22/2024-07:40:48] [I] Time Deserialize: Disabled
[11/22/2024-07:40:48] [I] Time Refit: Disabled
[11/22/2024-07:40:48] [I] NVTX verbosity: 0
[11/22/2024-07:40:48] [I] Persistent Cache Ratio: 0
[11/22/2024-07:40:48] [I] Inputs:
[11/22/2024-07:40:48] [I] === Reporting Options ===
[11/22/2024-07:40:48] [I] Verbose: Disabled
[11/22/2024-07:40:48] [I] Averages: 10 inferences
[11/22/2024-07:40:48] [I] Percentiles: 90,95,99
[11/22/2024-07:40:48] [I] Dump refittable layers:Disabled
[11/22/2024-07:40:48] [I] Dump output: Disabled
[11/22/2024-07:40:48] [I] Profile: Disabled
[11/22/2024-07:40:48] [I] Export timing to JSON file:
[11/22/2024-07:40:48] [I] Export output to JSON file:
[11/22/2024-07:40:48] [I] Export profile to JSON file:
[11/22/2024-07:40:48] [I]
[11/22/2024-07:40:55] [I] === Device Information ===
[11/22/2024-07:40:55] [I] Selected Device: NVIDIA GeForce RTX 2080 Ti
[11/22/2024-07:40:55] [I] Compute Capability: 7.5
[11/22/2024-07:40:55] [I] SMs: 68
[11/22/2024-07:40:55] [I] Compute Clock Rate: 1.545 GHz
[11/22/2024-07:40:55] [I] Device Global Memory: 11011 MiB
[11/22/2024-07:40:55] [I] Shared Memory per SM: 64 KiB
[11/22/2024-07:40:55] [I] Memory Bus Width: 352 bits (ECC disabled)
[11/22/2024-07:40:55] [I] Memory Clock Rate: 7 GHz
[11/22/2024-07:40:55] [I]
[11/22/2024-07:40:55] [I] TensorRT version: 8.5.2
[11/22/2024-07:40:55] [I] [TRT] [MemUsageChange] Init CUDA: CPU +14, GPU +0, now: CPU 27, GPU 157 (MiB)
[11/22/2024-07:40:58] [I] [TRT] [MemUsageChange] Init builder kernel library: CPU +342, GPU +76, now: CPU 423, GPU 233 (MiB)
[11/22/2024-07:40:58] [I] Start parsing network model
[11/22/2024-07:40:59] [I] [TRT] ----------------------------------------------------------------
[11/22/2024-07:40:59] [I] [TRT] Input filename:   ./model_repo_sense_voice_small/encoder/1/model.onnx
[11/22/2024-07:40:59] [I] [TRT] ONNX IR version:  0.0.8
[11/22/2024-07:40:59] [I] [TRT] Opset version:    18
[11/22/2024-07:40:59] [I] [TRT] Producer name:    pytorch
[11/22/2024-07:40:59] [I] [TRT] Producer version: 2.3.0
[11/22/2024-07:40:59] [I] [TRT] Domain:
[11/22/2024-07:40:59] [I] [TRT] Model version:    0
[11/22/2024-07:40:59] [I] [TRT] Doc string:
[11/22/2024-07:40:59] [I] [TRT] ----------------------------------------------------------------
[11/22/2024-07:41:00] [W] [TRT] parsers/onnx/onnx2trt_utils.cpp:375: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[11/22/2024-07:41:00] [I] [TRT] No importer registered for op: LayerNormalization. Attempting to import as plugin.
[11/22/2024-07:41:00] [I] [TRT] Searching for plugin: LayerNormalization, plugin_version: 1, plugin_namespace:
[11/22/2024-07:41:00] [E] [TRT] parsers/onnx/ModelImporter.cpp:726: While parsing node number 101 [LayerNormalization -> "/encoder/encoders0.0/norm1/LayerNormalization_output_0"]:
[11/22/2024-07:41:00] [E] [TRT] parsers/onnx/ModelImporter.cpp:727: --- Begin node ---
[11/22/2024-07:41:00] [E] [TRT] parsers/onnx/ModelImporter.cpp:728: input: "/encoder/encoders0.0/norm1/Cast_output_0"
input: "onnx::LayerNormalization_10442"
input: "onnx::LayerNormalization_10443"
output: "/encoder/encoders0.0/norm1/LayerNormalization_output_0"
name: "/encoder/encoders0.0/norm1/LayerNormalization"
op_type: "LayerNormalization"
attribute {
  name: "axis"
  i: -1
  type: INT
}
attribute {
  name: "epsilon"
  f: 1e-05
  type: FLOAT
}

[11/22/2024-07:41:00] [E] [TRT] parsers/onnx/ModelImporter.cpp:729: --- End node ---
[11/22/2024-07:41:00] [E] [TRT] parsers/onnx/ModelImporter.cpp:731: ERROR: parsers/onnx/builtin_op_importers.cpp:5429 In function importFallbackPluginImporter:
[8] Assertion failed: creator && "Plugin not found, are the plugin name, version, and namespace correct?"
[11/22/2024-07:41:00] [E] Failed to parse onnx file
[11/22/2024-07:41:00] [I] Finish parsing network model
[11/22/2024-07:41:00] [E] Parsing model failed
[11/22/2024-07:41:00] [E] Failed to create engine from model or file.
[11/22/2024-07:41:00] [E] Engine set up failed
&&&& FAILED TensorRT.trtexec [TensorRT v8502] # /usr/src/tensorrt/bin/trtexec --onnx=./model_repo_sense_voice_small/encoder/1/model.onnx --saveEngine=model.plan --minShapes=speech:1x8x560,speech_lengths:1,language:1,textnorm:1 --optShapes=speech:4x128x560,speech_lengths:4,language:4,textnorm:4 --maxShapes=speech:16x512x560,speech_lengths:16,language:16,textnorm:16
&&&& RUNNING TensorRT.trtexec [TensorRT v8502] # /usr/src/tensorrt/bin/trtexec --loadEngine=model.plan --noDataTransfers --verbose --shapes=speech:4x65x560,speech_lengths:4,language:4,textnorm:4
[11/22/2024-07:41:01] [I] === Model Options ===
[11/22/2024-07:41:01] [I] Format: *
[11/22/2024-07:41:01] [I] Model:
[11/22/2024-07:41:01] [I] Output:
[11/22/2024-07:41:01] [I] === Build Options ===
[11/22/2024-07:41:01] [I] Max batch: explicit batch
[11/22/2024-07:41:01] [I] Memory Pools: workspace: default, dlaSRAM: default, dlaLocalDRAM: default, dlaGlobalDRAM: default
[11/22/2024-07:41:01] [I] minTiming: 1
[11/22/2024-07:41:01] [I] avgTiming: 8
[11/22/2024-07:41:01] [I] Precision: FP32
[11/22/2024-07:41:01] [I] LayerPrecisions:
[11/22/2024-07:41:01] [I] Calibration:
[11/22/2024-07:41:01] [I] Refit: Disabled
[11/22/2024-07:41:01] [I] Sparsity: Disabled
[11/22/2024-07:41:01] [I] Safe mode: Disabled
[11/22/2024-07:41:01] [I] DirectIO mode: Disabled
[11/22/2024-07:41:01] [I] Restricted mode: Disabled
[11/22/2024-07:41:01] [I] Build only: Disabled
[11/22/2024-07:41:01] [I] Save engine:
[11/22/2024-07:41:01] [I] Load engine: model.plan
[11/22/2024-07:41:01] [I] Profiling verbosity: 0
[11/22/2024-07:41:01] [I] Tactic sources: Using default tactic sources
[11/22/2024-07:41:01] [I] timingCacheMode: local
[11/22/2024-07:41:01] [I] timingCacheFile:
[11/22/2024-07:41:01] [I] Heuristic: Disabled
[11/22/2024-07:41:01] [I] Preview Features: Use default preview flags.
[11/22/2024-07:41:01] [I] Input(s)s format: fp32:CHW
[11/22/2024-07:41:01] [I] Output(s)s format: fp32:CHW
[11/22/2024-07:41:01] [I] Input build shape: speech=4x65x560+4x65x560+4x65x560
[11/22/2024-07:41:01] [I] Input build shape: speech_lengths=4+4+4
[11/22/2024-07:41:01] [I] Input build shape: textnorm=4+4+4
[11/22/2024-07:41:01] [I] Input build shape: language=4+4+4
[11/22/2024-07:41:01] [I] Input calibration shapes: model
[11/22/2024-07:41:01] [I] === System Options ===
[11/22/2024-07:41:01] [I] Device: 0
[11/22/2024-07:41:01] [I] DLACore:
[11/22/2024-07:41:01] [I] Plugins:
[11/22/2024-07:41:01] [I] === Inference Options ===
[11/22/2024-07:41:01] [I] Batch: Explicit
[11/22/2024-07:41:01] [I] Input inference shape: language=4
[11/22/2024-07:41:01] [I] Input inference shape: textnorm=4
[11/22/2024-07:41:01] [I] Input inference shape: speech_lengths=4
[11/22/2024-07:41:01] [I] Input inference shape: speech=4x65x560
[11/22/2024-07:41:01] [I] Iterations: 10
[11/22/2024-07:41:01] [I] Duration: 3s (+ 200ms warm up)
[11/22/2024-07:41:01] [I] Sleep time: 0ms
[11/22/2024-07:41:01] [I] Idle time: 0ms
[11/22/2024-07:41:01] [I] Streams: 1
[11/22/2024-07:41:01] [I] ExposeDMA: Disabled
[11/22/2024-07:41:01] [I] Data transfers: Disabled
[11/22/2024-07:41:01] [I] Spin-wait: Disabled
[11/22/2024-07:41:01] [I] Multithreading: Disabled
[11/22/2024-07:41:01] [I] CUDA Graph: Disabled
[11/22/2024-07:41:01] [I] Separate profiling: Disabled
[11/22/2024-07:41:01] [I] Time Deserialize: Disabled
[11/22/2024-07:41:01] [I] Time Refit: Disabled
[11/22/2024-07:41:01] [I] NVTX verbosity: 0
[11/22/2024-07:41:01] [I] Persistent Cache Ratio: 0
[11/22/2024-07:41:01] [I] Inputs:
[11/22/2024-07:41:01] [I] === Reporting Options ===
[11/22/2024-07:41:01] [I] Verbose: Enabled
[11/22/2024-07:41:01] [I] Averages: 10 inferences
[11/22/2024-07:41:01] [I] Percentiles: 90,95,99
[11/22/2024-07:41:01] [I] Dump refittable layers:Disabled
[11/22/2024-07:41:01] [I] Dump output: Disabled
[11/22/2024-07:41:01] [I] Profile: Disabled
[11/22/2024-07:41:01] [I] Export timing to JSON file:
[11/22/2024-07:41:01] [I] Export output to JSON file:
[11/22/2024-07:41:01] [I] Export profile to JSON file:
[11/22/2024-07:41:01] [I]
[11/22/2024-07:41:08] [I] === Device Information ===
[11/22/2024-07:41:08] [I] Selected Device: NVIDIA GeForce RTX 2080 Ti
[11/22/2024-07:41:08] [I] Compute Capability: 7.5
[11/22/2024-07:41:08] [I] SMs: 68
[11/22/2024-07:41:08] [I] Compute Clock Rate: 1.545 GHz
[11/22/2024-07:41:08] [I] Device Global Memory: 11011 MiB
[11/22/2024-07:41:08] [I] Shared Memory per SM: 64 KiB
[11/22/2024-07:41:08] [I] Memory Bus Width: 352 bits (ECC disabled)
[11/22/2024-07:41:08] [I] Memory Clock Rate: 7 GHz
[11/22/2024-07:41:08] [I]
[11/22/2024-07:41:08] [I] TensorRT version: 8.5.2
[11/22/2024-07:41:08] [V] [TRT] Registered plugin creator - ::BatchTilePlugin_TRT version 1
[11/22/2024-07:41:08] [V] [TRT] Registered plugin creator - ::BatchedNMS_TRT version 1
[11/22/2024-07:41:08] [V] [TRT] Registered plugin creator - ::BatchedNMSDynamic_TRT version 1
[11/22/2024-07:41:08] [V] [TRT] Registered plugin creator - ::Clip_TRT version 1
[11/22/2024-07:41:08] [V] [TRT] Registered plugin creator - ::CoordConvAC version 1
[11/22/2024-07:41:08] [V] [TRT] Registered plugin creator - ::CropAndResize version 1
[11/22/2024-07:41:08] [V] [TRT] Registered plugin creator - ::CropAndResizeDynamic version 1
[11/22/2024-07:41:08] [V] [TRT] Registered plugin creator - ::DecodeBbox3DPlugin version 1
[11/22/2024-07:41:08] [V] [TRT] Registered plugin creator - ::DetectionLayer_TRT version 1
[11/22/2024-07:41:08] [V] [TRT] Registered plugin creator - ::EfficientNMS_TRT version 1
[11/22/2024-07:41:08] [V] [TRT] Registered plugin creator - ::EfficientNMS_ONNX_TRT version 1
[11/22/2024-07:41:08] [V] [TRT] Registered plugin creator - ::EfficientNMS_Explicit_TF_TRT version 1
[11/22/2024-07:41:08] [V] [TRT] Registered plugin creator - ::EfficientNMS_Implicit_TF_TRT version 1
[11/22/2024-07:41:08] [V] [TRT] Registered plugin creator - ::FlattenConcat_TRT version 1
[11/22/2024-07:41:08] [V] [TRT] Registered plugin creator - ::fMHA_V2 version 1
[11/22/2024-07:41:08] [V] [TRT] Registered plugin creator - ::fMHCA version 1
[11/22/2024-07:41:08] [V] [TRT] Registered plugin creator - ::GenerateDetection_TRT version 1
[11/22/2024-07:41:08] [V] [TRT] Registered plugin creator - ::GridAnchor_TRT version 1
[11/22/2024-07:41:08] [V] [TRT] Registered plugin creator - ::GridAnchorRect_TRT version 1
[11/22/2024-07:41:08] [V] [TRT] Registered plugin creator - ::GroupNorm version 1
[11/22/2024-07:41:08] [V] [TRT] Registered plugin creator - ::InstanceNormalization_TRT version 1
[11/22/2024-07:41:08] [V] [TRT] Registered plugin creator - ::LayerNorm version 1
[11/22/2024-07:41:08] [V] [TRT] Registered plugin creator - ::LReLU_TRT version 1
[11/22/2024-07:41:08] [V] [TRT] Registered plugin creator - ::MultilevelCropAndResize_TRT version 1
[11/22/2024-07:41:08] [V] [TRT] Registered plugin creator - ::MultilevelProposeROI_TRT version 1
[11/22/2024-07:41:08] [V] [TRT] Registered plugin creator - ::MultiscaleDeformableAttnPlugin_TRT version 1
[11/22/2024-07:41:08] [V] [TRT] Registered plugin creator - ::NMS_TRT version 1
[11/22/2024-07:41:08] [V] [TRT] Registered plugin creator - ::NMSDynamic_TRT version 1
[11/22/2024-07:41:08] [V] [TRT] Registered plugin creator - ::Normalize_TRT version 1
[11/22/2024-07:41:08] [V] [TRT] Registered plugin creator - ::PillarScatterPlugin version 1
[11/22/2024-07:41:08] [V] [TRT] Registered plugin creator - ::PriorBox_TRT version 1
[11/22/2024-07:41:08] [V] [TRT] Registered plugin creator - ::ProposalLayer_TRT version 1
[11/22/2024-07:41:08] [V] [TRT] Registered plugin creator - ::Proposal version 1
[11/22/2024-07:41:08] [V] [TRT] Registered plugin creator - ::ProposalDynamic version 1
[11/22/2024-07:41:08] [V] [TRT] Registered plugin creator - ::PyramidROIAlign_TRT version 1
[11/22/2024-07:41:08] [V] [TRT] Registered plugin creator - ::Region_TRT version 1
[11/22/2024-07:41:08] [V] [TRT] Registered plugin creator - ::Reorg_TRT version 1
[11/22/2024-07:41:08] [V] [TRT] Registered plugin creator - ::ResizeNearest_TRT version 1
[11/22/2024-07:41:08] [V] [TRT] Registered plugin creator - ::RPROI_TRT version 1
[11/22/2024-07:41:08] [V] [TRT] Registered plugin creator - ::ROIAlign_TRT version 1
[11/22/2024-07:41:08] [V] [TRT] Registered plugin creator - ::ScatterND version 1
[11/22/2024-07:41:08] [V] [TRT] Registered plugin creator - ::SeqLen2Spatial version 1
[11/22/2024-07:41:08] [V] [TRT] Registered plugin creator - ::SpecialSlice_TRT version 1
[11/22/2024-07:41:08] [V] [TRT] Registered plugin creator - ::SplitGeLU version 1
[11/22/2024-07:41:08] [V] [TRT] Registered plugin creator - ::Split version 1
[11/22/2024-07:41:08] [V] [TRT] Registered plugin creator - ::VoxelGeneratorPlugin version 1
[11/22/2024-07:41:08] [E] Error opening engine file: model.plan
[11/22/2024-07:41:08] [E] Failed to create engine from model or file.
[11/22/2024-07:41:08] [E] Engine set up failed
&&&& FAILED TensorRT.trtexec [TensorRT v8502] # /usr/src/tensorrt/bin/trtexec --loadEngine=model.plan --noDataTransfers --verbose --shapes=speech:4x65x560,speech_lengths:4,language:4,textnorm:4

Code sample

Expected behavior

Environment

  • OS (e.g., Linux):
  • FunASR Version (e.g., 1.0.0):
  • ModelScope Version (e.g., 1.11.0):
  • PyTorch Version (e.g., 2.0.0):
  • How you installed funasr (pip, source):
  • Python version:
  • GPU (e.g., V100M32)
  • CUDA/cuDNN version (e.g., cuda11.7):
  • Docker version (e.g., funasr-runtime-sdk-cpu-0.4.1)
  • Any other relevant information:

Additional context
@didadida-r didadida-r added the bug Something isn't working label Nov 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@didadida-r