Failed convert Huggingface Model to Tensorrt

[dathudeptrai]

Hi, I ran this script. The model was able to convert to tensorrt but then when I ran inference It failed to apply tensorrt engine. Bellow is a bug I met.

2022-09-16 20:20:05.105238: I tensorflow/compiler/tf2tensorrt/common/utils.cc:104] Linked TensorRT version: 8.4.2
2022-09-16 20:20:05.105331: I tensorflow/compiler/tf2tensorrt/common/utils.cc:106] Loaded TensorRT version: 8.4.2
2022-09-16 20:20:12.817319: I tensorflow/compiler/tf2tensorrt/convert/convert_nodes.cc:1275] [TF-TRT] Sparse compute capability is enabled.
2022-09-16 20:20:13.873705: W tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:82] TF-TRT Warning: DefaultLogger (# 0 (VALUES TensorRTInputPH_0))
2022-09-16 20:20:13.873812: W tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:82] TF-TRT Warning: DefaultLogger (# 0 (VALUES TensorRTInputPH_1))
2022-09-16 20:20:38.453412: W tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:82] TF-TRT Warning: DefaultLogger Skipping tactic 0x0000000000000000 due to Myelin error: Incompatible effective shapes in operation between: StatefulPartitionedCall/tf_bert_for_pre_training/bert/embeddings/Tile'-(f32[2,2,768][]) and TensorRTInputPH_4'-(f32[1,128,768][98304,768,1]).
2022-09-16 20:20:38.491285: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:86] DefaultLogger 10: [optimizer.cpp::computeCosts::3626] Error Code 10: Internal Error (Could not find any implementation for node {ForeignNode[PartitionedCall/TRTEngineOp_000_000/tftrt_constant-0:CONSTANT...PartitionedCall/TRTEngineOp_000_000/StatefulPartitionedCall/tf_bert_for_pre_training/mlm___cls/predictions/BiasAdd-sum:ELEMENTWISE]}.)
2022-09-16 20:20:38.556379: W tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:1061] TF-TRT Warning: Engine creation for PartitionedCall/TRTEngineOp_000_000 failed. The native segment will be used instead. Reason: INTERNAL: Failed to build TensorRT engine
2022-09-16 20:20:38.556563: W tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:894] TF-TRT Warning: Engine retrieval for input shapes: [[3], [2], [1,1,1,128], [128], [1,128,768]] failed. Running native segment for PartitionedCall/TRTEngineOp_000_000

@DEKHTIARJonathan

[ncomly-nvidia]

@dathudeptrai can you please provide more info on your environment so we can reproduce this issue?

[dathudeptrai]

@ncomly-nvidia Hi, I fixed the issue. I think the repo lack of tutorial. For example, I have to set dynamic_shape_profile_strategy to Range+Optimal so that the model able to run inference on dynamic input shape. (https://www.tensorflow.org/api_docs/python/tf/experimental/tensorrt/Converter)