How can i quantize model when i use custom model on tensorrt-llm?whether i need to write c++ code or not?any examples?thank u for your time and help.

[DelongYang666]

No description provided.

[nv-guomingz]

Step 0. https://nvidia.github.io/TensorRT-LLM/architecture/add-model.html
Step 1. Implement your private op if needed(C++ refer to plugin implmentation, it's not a must-have option)
Step 2. Quantize the model with recipe like int4/int8 weights only or int8 smoothquant.

[DelongYang666]

Thank u！i saw the document [https://nvidia.github.io/TensorRT-LLM/architecture/add-model.html] 3 hours ago. And I will try.And, a vedio on bilibili told me that fp8 may performence well than other quantization way. But , this way depends on modelopt. The modelopt does not support my custom model to quantize. Any ways to implement fp8 ?

[nv-guomingz]

Thank u！i saw the document [https://nvidia.github.io/TensorRT-LLM/architecture/add-model.html] 3 hours ago. And I will try.And, a vedio on bilibili told me that fp8 may performence well than other quantization way. But , this way depends on modelopt. The modelopt does not support my custom model to quantize. Any ways to implement fp8 ?

@RalphMao do you have any comments on this question?