Set Triton version

[okyspace]

Hi, currently when I use clearml serving, when deploying serving triton service, it is always version 21.03. Is there a way that I can config or set to 21.05? I need some feature from 21.05.

[jkhenning]

Hi @okyspace,

That's a good point - we'll add this option 🙂

[okyspace]

@jkhenning May I add on. While allowing user to define triton version, perhaps you can somehow accept args to set triton e.g. version, control mode, polling intervals, log-verbose, port numbers, etc.. so that it become more flexible for users.
Hope this is useful to improve the service.

[jkhenning]

I agree :) - We'll post suggested configuration options here soon and make sure they fit your purpose.

[bmartinn]

Just updating here the full feature request:
Allow controlling the following from clearml-serving CLI

• Container image
• Container execution arguments
• Triton engine command line arguments

[okyspace]

@bmartinn Thanks for the add on. May I ask how to use these flags? They are not updated in the readme.
Attempted this but got error of unrecognised flag.
clearml-serving triton --project "serving" --name "serving example --update-frequency 0.2"

I am currently using 0.33 version which should contain the fixes right?

[bmartinn]

Hi @okyspace
My apologies, we are working on the documentation of this new module, I'm hoping the full docs will be released very soon :)
In the mean time clearml-serving --help, clearml-serving launch --help, and clearml-serving triton --help 😄

What exactly is the error you are getting ?
(btw: --update-frequency is not a valid command line option, is this a typo ?)

[okyspace]

I was trying to pass in --repository-poll-sec to triton using --engine-container-args "--repository-poll-sec 10". but this is not working. May i know how to do it?

[bmartinn]

@okyspace you are right, I missed it, let me see if we can add the triton arguments ;)

[qwaxys]

Closing this issue as inactive, feel free to open a new issue and link to this one (and probably issue #3 )

[amirhmk]

I was wonderrig if this was ever implemented? I'm using the docker-compose setup, so not sure if the CLI will work for my usecase (perhaps passing them as env variabels would work?)

For instance, I'd like to set some variabels like this:

('grpc.max_send_message_length', 512 * 1024 * 1024), ('grpc.max_receive_message_length', 512 * 1024 * 1024)]