Ragged Tensor does not work with BoxCOCOMetrics

[IMvision12]

I have a train_ds :

<_PrefetchDataset element_spec=(TensorSpec(shape=(4, 640, 640, 3), dtype=tf.float32, name=None), {'classes': RaggedTensorSpec(TensorShape([4, None]), tf.float32, 1, tf.int64), 'boxes': RaggedTensorSpec(TensorShape([4, None, 4]), tf.float32, 1, tf.int64)})>

Error : ValueError: Expected boxes to be a Tensor with a final dimension of 4. Instead, got boxes.shape=(20, None, None).

Gist : https://colab.research.google.com/drive/1x0kuVTtXvfBFDkfVcB-9mFwAvVVNTksB?usp=sharing

@ianstenbit @jbischof

[james77777778]

Hi @IMvision12

I think the bug was caused by eval_ds rather than train_ds.

Please refer to the following link for the working pipeline:
https://colab.research.google.com/drive/1yAXnov0S0k0Xs5ttHbmVFlhAs2J6Yblc?usp=sharing

I ensure the shape of the bounding boxes at

def dict_to_tuple(inputs):
    inputs["bounding_boxes"] = keras_cv.bounding_box.to_dense(
        inputs["bounding_boxes"]
    )
    inputs["bounding_boxes"] = keras_cv.bounding_box.to_ragged(
        inputs["bounding_boxes"]
    )
    return inputs["images"], inputs["bounding_boxes"]

The root cause should be the usage of keras_cv.layers.Resizing

Kindly ping @hugopi

[IMvision12]

hey @james77777778 I know this works:

def dict_to_tuple(inputs):
    inputs["bounding_boxes"] = keras_cv.bounding_box.to_dense(
        inputs["bounding_boxes"]
    )
    inputs["bounding_boxes"] = keras_cv.bounding_box.to_ragged(
        inputs["bounding_boxes"]
    )
    return inputs["images"], inputs["bounding_boxes"]

But my question is if we are converting boxes to dense first and then to ragged then whats the use, it should directly work with ragged input right?

we are converting ragged -> Dense -> ragged

[IMvision12]

Indeed the issue is with resizing layer coz:

This does not work (Normal Resize):

inference_resizing= keras_cv.layers.Resizing(
    640, 640, bounding_box_format="xyxy", pad_to_aspect_ratio=True
)
eval_ds = eval_ds.map(inference_resizing, num_parallel_calls=tf.data.AUTOTUNE)

def dict_to_tuple(inputs):
    return inputs["images"], inputs["bounding_boxes"]


train_ds = train_ds.map(dict_to_tuple, num_parallel_calls=tf.data.AUTOTUNE)
eval_ds = eval_ds.map(dict_to_tuple, num_parallel_calls=tf.data.AUTOTUNE)

train_ds = train_ds.prefetch(tf.data.AUTOTUNE)
eval_ds = eval_ds.prefetch(tf.data.AUTOTUNE)

This works (JitteredResize):

inference_resizing =keras_cv.layers.JitteredResize(
            target_size=(640, 640), scale_factor=(0.75, 1.3), bounding_box_format="xyxy"
        )
eval_ds = eval_ds.map(inference_resizing, num_parallel_calls=tf.data.AUTOTUNE)

def dict_to_tuple(inputs):
    return inputs["images"], inputs["bounding_boxes"]


train_ds = train_ds.map(dict_to_tuple, num_parallel_calls=tf.data.AUTOTUNE)
eval_ds = eval_ds.map(dict_to_tuple, num_parallel_calls=tf.data.AUTOTUNE)

train_ds = train_ds.prefetch(tf.data.AUTOTUNE)
eval_ds = eval_ds.prefetch(tf.data.AUTOTUNE)

[Inshu32]

Hi I have a similar problem, I want to normally resize my images by using keras_cv.layers.Resizing() function. Right now I am using JitteredResize function as it is used in the example implementation on Keras website. By going through the above discussion, doesnot normal resizing work with Ragged tensors ? My data is such that I donot want cropping which is happening in JitteredResize. @IMvision12 @james77777778 @LukeWood