Add support for custom intervals #814
Conversation
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
model_analyzer/config/generate/brute_plus_binary_search_run_config_generator.py
Fixed
Show fixed
Hide fixed
model_analyzer/config/generate/quick_plus_concurrency_sweep_run_config_generator.py
Fixed
Show fixed
Hide fixed
| if self._cli_config.is_llm_model(): | ||
| # The possible inference loads are concurrency, request rate, periodic concurrency, or custom (request-intervals) | ||
| # - If custom is specified, it is used | ||
| # - For LLM models, periodic concurrency is used |
There was a problem hiding this comment.
Rework comment as LLM has gone away
model_analyzer/config/generate/perf_analyzer_config_generator.py
Outdated
Show resolved
Hide resolved
| @@ -153,3 +157,9 @@ def _set_concurrency(self, run_config: RunConfig, concurrency: int) -> RunConfig | |||
| perf_config.update_config({"concurrency-range": concurrency}) | |||
|
|
|||
| return run_config | |||
|
|
|||
There was a problem hiding this comment.
This method is duplicated (also in brute search). Maybe this should be a static method in ModelProfileSpec?
There was a problem hiding this comment.
Yes, this is still duplicated. I didn't clean it up yet. Both classes implement ConfigGeneratorInterface. You could create a base class with common code if you want.
| @@ -511,9 +511,12 @@ def _get_next_perf_analyzer_config( | |||
|
|
|||
| perf_analyzer_config.update_config_from_profile_config(model_name, self._config) | |||
|
|
|||
| concurrency = self._calculate_concurrency(dimension_values) | |||
| perf_config_params = {"batch-size": 1} | |||
There was a problem hiding this comment.
I feel like it'd be cleaner if the PerfAnalyzerConfig() constructor initialized batch-size: 1. Seems like we are always needed to add this in the code.
tgerdesnv
force-pushed
the
tgerdes-fix-custom-intervals
branch
from
1b24043 to
38abbe8
Compare
tgerdesnv
force-pushed
the
tgerdes-fix-custom-intervals
branch
from
fae1195 to
6b3a199
Compare
| "concurrency-range": default_concurrency, | ||
| } | ||
| default_perf_analyzer_config.update_config(perf_config_params) | ||
| if not "request-intervals" in model.perf_analyzer_flags(): |
There was a problem hiding this comment.
Doing a self-review: I'm wondering if this should be if not model.is_load_specified() just like line 515
Fix support for Perf Analyzer's request-intervals
Fixes #808