Error during creation endpoints with config.pbtxt

[IlyaMescheryakov1402]

I have created endpoint like this:

clearml-serving --id "<>" model add --engine triton --endpoint 'conformer_joint' --model-id '<>' --preprocess 'preprocess_joint.py' --aux-config "./config.pbtxt"

config.pbtxt file:

name: "conformer_joint"
default_model_filename: "model.bin"
max_batch_size: 16
dynamic_batching {
    max_queue_delay_microseconds: 100
}
input: [
    {
        name: "encoder_outputs"
        data_type: TYPE_FP32
        dims: [
            1,
            640
        ]
    },
    {
        name: "decoder_outputs"
        data_type: TYPE_FP32
        dims: [
            640,
            1
        ]
    }
]
output: [
    {
        name: "outputs"
        data_type: TYPE_FP32
        dims: [
            129
        ]
    }
]

preprocess_joint.py file:

from typing import Any, Union, Optional, Callable

class Preprocess(object):
    def __init__(self):
        # set internal state, this will be called only once. (i.e. not per request)
        pass

    def preprocess(
            self,
            body: Union[bytes, dict],
            state: dict, 
            collect_custom_statistics_fn: Optional[Callable[[dict], None]]
        ) -> Any:
        return body["encoder_outputs"], body["decoder_outputs"]

    def postprocess(
            self,
            data: Any,
            state: dict, 
            collect_custom_statistics_fn: Optional[Callable[[dict], None]]
        ) -> dict:
        return {"data":data.tolist()}

triton container and inference container show no errors, and I can find this triton model with right config.pbtxt in folder /models/conformer_joint. But when I try to make a request to model like this:

import numpy as np
import requests
body={
    "encoder_outputs": [np.random.randn(1, 640).tolist()],
    "decoder_outputs": [np.random.randn(640, 1).tolist()]
}
response = requests.post(f"<>/conformer_joint", json=body)
response.json()

I am getting an error:

Error processing request: object of type 'NoneType' has no len()

Model endpoint in serving task:

conformer_joint {
  engine_type = "triton"
  serving_url = "conformer_joint"
  model_id = "<>"
  preprocess_artifact = "py_code_conformer_joint"
  auxiliary_cfg = """name: "conformer_joint"
default_model_filename: "model.bin"
max_batch_size: 16
dynamic_batching {
    max_queue_delay_microseconds: 100
}
input: [
    {
        name: "encoder_outputs"
        data_type: TYPE_FP32
        dims: [
            1,
            640
        ]
    },
    {
        name: "decoder_outputs"
        data_type: TYPE_FP32
        dims: [
            640,
            1
        ]
    }
]
output: [
    {
        name: "outputs"
        data_type: TYPE_FP32
        dims: [
            129
        ]
    }
]
"""
}

Error occurs in process function of TritonPreprocessRequest (https://github.com/allegroai/clearml-serving/blob/main/clearml_serving/serving/preprocess_service.py#L358C9-L358C81) because function use endpoint params like input_name, input_type and input_size. When we create endpoint like above, this parameters placed in auxiliary_cfg attribute.

Is there any chance to fix that error and create endpoint like above?

[jkhenning]

Hi @IlyaMescheryakov1402,

You have to provide input/output layers when registering the model, otherwise it will not be able to properly encode the request. This cannot be inside the pbtxt file, since we fail to parse it in too many cases. We pushed a fix for this issue now you will get an error if input/output is not provided in the CLI when registering the model.

[IlyaMescheryakov1402]

Hi @jkhenning

Thanks for the answer!

So, I need to convert input from config.pbtxt like this

input: [
        {
            name: "length"
            data_type: TYPE_INT64
            reshape: {
                shape: []
            }
            dims: [
                1
            ]
        },
        {
            name: "audio_signal"
            data_type: TYPE_FP16
            dims: [
                80,
                -1
            ]
        }
    ]

How to convert and set reshape part (and other parts besides name, shape and type) while providing input layers?

P.S. --aux-config input.reshape... doesn't work for me

[joachimhgg]

I solved this using --aux-config input.0.reshape

[IlyaMescheryakov1402]

@joachimhgg Thank you! It works!