QAT : TRT 8 compatible workflow #804
Draft
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
SrivastavaKshitij
changed the title
| input = input_quantizer.get_output(0), | ||
| scale = scale_trt.get_output(0)) | ||
|
|
||
| if hasattr(module._input_quantizer,'quant_axis'): |
There was a problem hiding this comment.
This seems like it can be simplified by re-using the result from the if block at line 25
# for free
to join this conversation on GitHub.
Already have an account?
# to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Hi @jaybdub
I am introducing this new QAT workflow which is compatible with TensorRT 8.
TRT introduced IQuantize and IDequantize Layers which are to be manually placed in the network based on the guidelines mentioned in Q/DQ placement.
I have added support to quantize
nn.Conv2d,nn.MaxPool2dandnn.AdaptiveAvfPool2d- layers that are necessary to quantize Resnet(s). I have also added aQuantGenericTensorwhich can be used to add QDQ layer anywhere in the model based on Nvidia's guidelines.This PR also introduces the option to choose between per tensor quantization and per channel quantization. All quant layers are scriptable with
torch.jit.scriptMost of the files that I have modified / changed are under
contribfolders, so it doesn't affect the main torch2trt library.I will continue to add support for more layers but I believe this PR is big enough to land and then I can put up smaller PRs to add more functionalities.
Entire workflow is tested with Pytorch NGC Container
22.04-py3Thanks.