model_yaml_config usage is not explained well enough

[Foundsheep]

📚 The doc issue

Expected :

The documentation about model_yaml_config sounds as if we could use it as below in config.properties and access it later.

• file name : config.properties
• content :

inference_address=https://127.0.0.1:8443
management_address=https://127.0.0.1:8444
metrics_address=https://127.0.0.1:8445
model_yaml_config={\
  "pippy": {\
    "rpc_timeout": <some value>\
  }\
}

and I can't access the model_yaml_config property through context.model_yaml_config and actually it throws an error.

---

Reality :

However, the way we could use the property is as below.

• command : torch-model-archiver --model-name <something> --serialized-file <some path> ... --config-file <yaml file path>

and this logic is very confusing when compared with what is written in the documentation

Suggest a potential alternative/fix

The logic seems like when my handler, having inherited BaseHandler, doesn't acutally assign self.model_yaml_config in its initialize method. Actually, it is assigned when Service is instantiated with .__init__ method

I suggest either of the two

1. Modify the documentation to use model_yaml_config property with torch-model-archiver --config-file <path> argument
2. Or modify the code to assign model_yaml_config through config.properties as it sounds in the current documentation.

[mreso]

Hi @Foundsheep
sorry that the documentation seems a bit confusing. There might be a misunderstanding here between models and model_yaml_config.

The models config would be the part which goes into the config.properties. It defines the model frontend parameter like number of worker, batch delay etc. As there is only one config.properties, there is only a songle models config (potentially having multiple models).

The model_yaml_config on the other hand is a yaml file that is mart of the .mar file or model folder and holds additional backend related model parameters like if you want to apply torch.compile to your model. Your handler can access that file through self.model_yaml_config if it was part of the torch-model-archiver command.

If you like to contribute you are welcome to create a doc PR to help clarifying this. Otherwise, I will try to squeeze it in at some point.