Calibration Fails with (Assertion mIndex >= 0 failed.symbol is not concrete)

[lukqw]

Description

I am currently trying to 8-bit integer quantize my tensorflow 2.3 SavedModel(s) via TensorRT.
Model in case is a custom yolov3-tiny relu model, which I have successfully split into two separate models with working inference.

When running the below code snippet for both the full yolov3-tiny relu model (yolo_relu) and the first part of yolov3-tiny relu (head_relu) model I am successful in generating a 8bit quantized version with TensorRT.

But, when running quantization for the second part of the yolov3-tiny relu model (tail_relu), I am receiving the error further down below. Please note that I am aware that the second part of my model (tail_relu) has a different input shape of (None,26,26,256) which I have accounted for with a separate calibration_input_tail method.

This is the code I am using:

from tensorflow.python.compiler.tensorrt import trt_convert as trt

import os
import cv2
import tensorflow as tf

#calibration function for second half of model which has input shape of 1,26,26,256
def calibration_input_tail():
    for x in range(500):
        yield [tf.random.normal((1, 26, 26, 256))]

def calibration_input():
    for x in range(500):
        yield [tf.random.normal((1, 416, 416, 3))]


def quantize_model(name):
    conv_params = trt.DEFAULT_TRT_CONVERSION_PARAMS._replace(
            precision_mode=trt.TrtPrecisionMode.INT8,
            max_workspace_size_bytes=4000000000,
            use_calibration=True)

    converter = trt.TrtGraphConverterV2(input_saved_model_dir=f'model_data/SavedModel/{name}_model', conversion_params=conv_params)


    if name == 'tail_relu':
        cali_input = calibration_input_tail
    else:
        cali_input = calibration_input

    converter.convert(calibration_input_fn=cali_input)


    converter.save(f'quantized_{name}_model')

#successful
#quantize_model("yolo_relu")
#successful
#quantize_model("head_relu")

#unsuccessful
quantize_model("tail_relu")

This is the error I am getting

2021-11-22 10:40:30.120420: E tensorflow/compiler/tf2tensorrt/utils/trt_logger.cc:42] DefaultLogger 2: [graph.cpp::getDefinition::336] Error Code 2: Internal Error (Assertion mIndex >= 0 failed.symbol is not concrete)
2021-11-22 10:40:30.257167: E tensorflow/compiler/tf2tensorrt/kernels/trt_engine_op.cc:1185] Calibration failed: Internal: Failed to build TensorRT engine
Traceback (most recent call last):
  File "quantize_trt.py", line 49, in <module>
    quantize_model("tail_relu")
  File "quantize_trt.py", line 37, in quantize_model
    converter.convert(calibration_input_fn=cali_input)
  File "/home/xavier/avdp-edge/lab/venv2/lib/python3.6/site-packages/tensorflow/python/compiler/tensorrt/trt_convert.py", line 1136, in convert
    self._converted_func(*map(ops.convert_to_tensor, inp))
  File "/home/xavier/avdp-edge/lab/venv2/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 1711, in __call__
    return self._call_impl(args, kwargs)
  File "/home/xavier/avdp-edge/lab/venv2/lib/python3.6/site-packages/tensorflow/python/eager/wrap_function.py", line 247, in _call_impl
    args, kwargs, cancellation_manager)
  File "/home/xavier/avdp-edge/lab/venv2/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 1729, in _call_impl
    return self._call_with_flat_signature(args, kwargs, cancellation_manager)
  File "/home/xavier/avdp-edge/lab/venv2/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 1778, in _call_with_flat_signature
    return self._call_flat(args, self.captured_inputs, cancellation_manager)
  File "/home/xavier/avdp-edge/lab/venv2/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 1961, in _call_flat
    ctx, args, cancellation_manager=cancellation_manager))
  File "/home/xavier/avdp-edge/lab/venv2/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 596, in call
    ctx=ctx)
  File "/home/xavier/avdp-edge/lab/venv2/lib/python3.6/site-packages/tensorflow/python/eager/execute.py", line 60, in quick_execute
    inputs, attrs, num_outputs)
tensorflow.python.framework.errors_impl.InternalError:  Failed to feed calibration data
         [[node TRTEngineOp_0_0 (defined at quantize_trt.py:37) ]] [Op:__inference_pruned_7004]

Function call stack:
pruned

Can someone help me out here? I understand it's a very niche problem, but in theory I am just trying to quantize a model, while supplying correctly shaped calibration data. Additionally this "second half" model is just a part of another model, which i have successfully quantized with the code above.

Environment

Jetson Xavier NX

TensorRT Version: 8.0.1
NVIDIA GPU:
NVIDIA Driver Version:
CUDA Version: 10.2
CUDNN Version: 8.2.1
Operating System: aarch64
Python Version (if applicable): python 3.6.9
Tensorflow Version (if applicable): 2.5.0

Relevant Files

I am including all three models yolo_relu_model, head_relu_model and tail_relu_model:

yolo drive

head drive

tail drive

Steps To Reproduce

After downloading the models you would have to call the script above.