Unable to load onnx models into Triton

[amirhmk]

Hi there,

I have been working on deploying our inference pipeline on clearml-serving using the docker-compose approach. I've hashed out most of the issues thus far thanks to the community, now I am facing another issue while loading onnx models.

I am getting the following error:

clearml-serving-triton   | mmdet  | UNAVAILABLE: Internal: **failed to stat file /models/mmdet/1/model.onnx**

I exec'd into the container to see what's inside /models, and under /models/mmdet/1 there was a model.bin but no model.onnx. I created the modeling using the OutputModel. I also tried doing it through the CLI:

clearml-serving --id $SERVING_ID model upload --name "mmdet_cli" --project $PROJECT_NAME --path /mmdet/model.onnx

but the same thing. I'm guessing when the folder structure is getting set up, this file gets renamed to a .bin extention. Should this be happening or am I doing something wrong?

When I download the file from the models section in the portal, it's an onnx file, exactly the one I uploaded. So not sure where this renaming is happening tbh...

[thepycoder]

Hi @amirhmk!

The model extentions are hardcoded based on the framework type of the model in ClearML.
You have 2 options:

• Override the default filename as in the example here, you'll see the config override in the CLI commands
• Or you can manually specify the model framework when you upload it, set framework argument to 'onnx' and it should properly rename it!

EDIT: I just saw that your framework in clearml is properly set. That's weird and probably a bug on our end. Please check if the first fix works for you, we'll investigate the second :)

Hope this helps! :)

[amirhmk]

Hi @thepycoder, option 1 worked, thanks! Adding default_model_filename: "model.bin" to config.pbtxt and passing it as a file was fine. Though regarding your second point, did onnx models use to have the .bin extention that this is setup like this?

Also as a small aside, I found out that if one passes a file (path) for --aux-config, it'll completely copy paste the passed in config file, and still add the input/output shapes/types/names to the bottom of the final config file - resulting in errors. I thought it would make sense to only pass the config file, and not have to even pass all the shapes one-by-one in the command. Wdyt?

[thepycoder]

option 1 worked, thanks!

Excellent!

Though regarding your second point, did onnx models use to have the .bin extention that this is setup like this?

Possibly yes, this might be a remnant of earlier ONNX days, I'll have to check.

I thought it would make sense to only pass the config file, and not have to even pass all the shapes one-by-one in the command. Wdyt?

Yeah, I think that makes sense, I'll discuss it with the devs :) Thanks a lot for the suggestion!

[ainoam]

I thought it would make sense to only pass the config file, and not have to even pass all the shapes one-by-one in the command. Wdyt?

@amirhmk Aren't these already optional CLI parameters? i.e. you don't have to specify them if their already specified in the config file.

[amirhmk]

@ainoam so most of those are, but the input(or output) shapes/types are mandatory.

[ainoam]

@amirhmk Can you give an example of your CLI invocation, and mark which parameters seem to be unnecessarily mandatory (i.e. also specified in your aux file)

[IlyaMescheryakov1402]

@ainoam #62