[Feature] Pluggable platform-specific scheduler #13161

yannicks1 · 2025-02-12T15:17:17Z

This PR enables different platforms to plugin their (hardware) specific scheduler class in vLLM version V0 and therefore addresses the scheduler part of this RFC #11162. A pluggable scheduler is needed to add support for the IBM Spyre accelerator #9652.
Note that this feature is under development for V1 but missing in the current V0.

Changes

added attribute scheduler_cls to class SchedulerConfig in vllm/config.py. Attribute can be a string containing the path to the scheduler class or the class itself.
added scheduler_cls to EngineArgs and add_cli_args() argparser
loading scheduler based on scheduler_cls in vllm/engine/llm_engine.py
added test to verify functionality: tests/plugins_tests/test_scheduler_plugins.py and appended it to the test pipeline (.buildkite/test-pipeline.yaml)

Remarks

since the scheduler_cls is initialized with "vllm.core.scheduler.Scheduler", the default behaviour does not change. However, the change allows to point to a (hardware) specific scheduler class in platforms.
I opted for initialization with "vllm.core.scheduler.Scheduler" and not "auto" as it is done for attribute worker_cls to have minimal changes (Otherwise I would have to overwrite "auto" for all platforms using the default scheduler).

Signed-off-by: Yannick Schnider <yannick.schnider1@ibm.com>

github-actions · 2025-02-12T15:17:33Z

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

comaniac · 2025-02-12T16:52:49Z

While we need more discussions about this feature in v1, I think it's ok for v0 to have it. Could you add a unit test with a dummy scheduler to 1) test the functionality, and 2) be an example?

Signed-off-by: Yannick Schnider <Yannick.Schnider1@ibm.com>

yannicks1 · 2025-02-14T15:05:21Z

Thanks for your feedback @comaniac ! I have added a test which validates the functionality.

tests/plugins/vllm_add_dummy_scheduler/vllm_add_dummy_scheduler/dummy_platform.py

youkaichao · 2025-02-15T08:06:16Z

vllm/config.py

@@ -1338,6 +1338,7 @@ class ParallelConfig:
    # will be determined based on the platform.
    worker_cls: str = "auto"
    sd_worker_cls: str = "auto"
+    scheduler_cls: str = "vllm.core.scheduler.Scheduler"


I think you can make the type either a string (qualname) or a type directly, similar to distributed_executor_backend that can directly be a class object. then you can easily test it, similar to https://github.com/vllm-project/vllm/blob/main/tests/engine/test_executor.py

youkaichao · 2025-02-15T08:07:17Z

vllm/engine/llm_engine.py

@@ -346,6 +346,8 @@ def get_tokenizer_for_seq(sequence: Sequence) -> AnyTokenizer:
        # Create the scheduler.
        # NOTE: the cache_config here have been updated with the numbers of
        # GPU and CPU blocks, which are profiled in the distributed executor.
+        Scheduler = resolve_obj_by_qualname(


if it is a string, use resolve_obj_by_qualname .

if not, it should be directly used as a class (and maybe assert it inherits from a base class?)

Signed-off-by: Yannick Schnider <Yannick.Schnider1@ibm.com>

…actoring allow scheduler_cls to be string or class directly, update testing

yannicks1 · 2025-02-17T19:52:49Z

Thanks for your feedback @youkaichao and @comaniac . I addressed it, please review again

youkaichao · 2025-02-18T03:43:06Z

tests/plugins_tests/test_scheduler_plugins.py

+            enforce_eager=True,  # reduce test time
+        )
+        vllm_config = engine_args.create_engine_config()
+        vllm_config.parallel_config.scheduler_cls = DummyScheduler


we should have a top-level argument for it, e.g. from --scheduler_cls "mod.name" or EngineArgs(scheduler_cls="mod.name")

youkaichao · 2025-02-18T03:43:39Z

vllm/config.py

@@ -1338,6 +1338,7 @@ class ParallelConfig:
    # will be determined based on the platform.
    worker_cls: str = "auto"
    sd_worker_cls: str = "auto"
+    scheduler_cls: Union[str, Type[object]] = "vllm.core.scheduler.Scheduler"


please add it in the SchedulerConfig ?

Signed-off-by: Yannick Schnider <Yannick.Schnider1@ibm.com>

…-argument move scheduler_cls to SchedulerConfig and add it to EngineArgs

mergify · 2025-02-18T14:37:16Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @yannicks1.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

yannicks1 · 2025-02-18T15:00:53Z

thanks for reviewing again @youkaichao , @comaniac . I have incorporated your feedback.

youkaichao

LGTM, thanks for the contribution!

Signed-off-by: Yannick Schnider <yannick.schnider1@ibm.com> Signed-off-by: Yannick Schnider <Yannick.Schnider1@ibm.com>

Signed-off-by: Yannick Schnider <yannick.schnider1@ibm.com> Signed-off-by: Yannick Schnider <Yannick.Schnider1@ibm.com> Signed-off-by: Linkun Chen <github@lkchen.net>

Signed-off-by: Yannick Schnider <yannick.schnider1@ibm.com> Signed-off-by: Yannick Schnider <Yannick.Schnider1@ibm.com> Signed-off-by: saeediy <saidakbarp@gmail.com>

enabling pluggable scheduler for different hardware

17a83e5

Signed-off-by: Yannick Schnider <yannick.schnider1@ibm.com>

yannicks1 marked this pull request as ready for review February 12, 2025 16:41

yannicks1 requested review from zhuohan123, youkaichao, alexm-redhat, comaniac and njhill as code owners February 12, 2025 16:41

sducouedic mentioned this pull request Feb 13, 2025

Ysc pluggable scheduler IBM/vllm-spyre#4

Merged

yannicks1 mentioned this pull request Feb 14, 2025

test to detect (platform specific) plugged-in scheduler yannicks1/vllm#1

Merged

mergify bot added the ci/build label Feb 14, 2025

test to detect (platform specific) plugged-in scheduler

835e97b

Signed-off-by: Yannick Schnider <Yannick.Schnider1@ibm.com>

yannicks1 force-pushed the ysc-pluggable-scheduler branch from e0beefe to 835e97b Compare February 14, 2025 14:58

comaniac reviewed Feb 14, 2025

View reviewed changes

tests/plugins/vllm_add_dummy_scheduler/vllm_add_dummy_scheduler/dummy_platform.py Outdated Show resolved Hide resolved

youkaichao reviewed Feb 15, 2025

View reviewed changes

wangxiyuan mentioned this pull request Feb 17, 2025

[RFC]: Hardware pluggable #11162

Open

1 task

yannicks1 added 2 commits February 17, 2025 18:20

allow scheduler_cls to be string or class directly, update testing

e5d8805

Signed-off-by: Yannick Schnider <Yannick.Schnider1@ibm.com>

Merge pull request #2 from yannicks1/ysc-pluggable-scheduler-test-ref…

bb9584a

…actoring allow scheduler_cls to be string or class directly, update testing

comaniac approved these changes Feb 17, 2025

View reviewed changes

comaniac added the ready ONLY add when PR is ready to merge/full CI is needed label Feb 17, 2025

youkaichao removed the ready ONLY add when PR is ready to merge/full CI is needed label Feb 18, 2025

youkaichao reviewed Feb 18, 2025

View reviewed changes

yannicks1 added 3 commits February 18, 2025 12:32

moving scheduler_cls from ParallelConfig to SchedulerConfig

0ef6ec0

Signed-off-by: Yannick Schnider <Yannick.Schnider1@ibm.com>

adding scheduler_cls to EngineArgs

37622e0

Signed-off-by: Yannick Schnider <Yannick.Schnider1@ibm.com>

adding scheduler-cls to arg parser

e487dcb

Signed-off-by: Yannick Schnider <Yannick.Schnider1@ibm.com>

Merge pull request #3 from yannicks1/ysc-pluggable-scheduler-toplevel…

4a05ba1

…-argument move scheduler_cls to SchedulerConfig and add it to EngineArgs

mergify bot added the needs-rebase label Feb 18, 2025

Merge branch 'main' into ysc-pluggable-scheduler

751a171

mergify bot removed the needs-rebase label Feb 18, 2025

youkaichao approved these changes Feb 19, 2025

View reviewed changes

youkaichao merged commit 4233302 into vllm-project:main Feb 19, 2025
18 checks passed

xjpang pushed a commit to xjpang/vllm that referenced this pull request Feb 20, 2025

[Feature] Pluggable platform-specific scheduler (vllm-project#13161)

97e7fc1

Signed-off-by: Yannick Schnider <yannick.schnider1@ibm.com> Signed-off-by: Yannick Schnider <Yannick.Schnider1@ibm.com>

kerthcet pushed a commit to kerthcet/vllm that referenced this pull request Feb 21, 2025

[Feature] Pluggable platform-specific scheduler (vllm-project#13161)

6cb2972

Signed-off-by: Yannick Schnider <yannick.schnider1@ibm.com> Signed-off-by: Yannick Schnider <Yannick.Schnider1@ibm.com>

Akshat-Tripathi pushed a commit to krai/vllm that referenced this pull request Mar 3, 2025

[Feature] Pluggable platform-specific scheduler (vllm-project#13161)

b5c9857

Signed-off-by: Yannick Schnider <yannick.schnider1@ibm.com> Signed-off-by: Yannick Schnider <Yannick.Schnider1@ibm.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature] Pluggable platform-specific scheduler #13161

[Feature] Pluggable platform-specific scheduler #13161

yannicks1 commented Feb 12, 2025 •

edited by github-actions bot

Loading

github-actions bot commented Feb 12, 2025

comaniac commented Feb 12, 2025

yannicks1 commented Feb 14, 2025

youkaichao Feb 15, 2025

youkaichao Feb 15, 2025

yannicks1 commented Feb 17, 2025

youkaichao Feb 18, 2025

youkaichao Feb 18, 2025

mergify bot commented Feb 18, 2025

yannicks1 commented Feb 18, 2025

youkaichao left a comment

[Feature] Pluggable platform-specific scheduler #13161

[Feature] Pluggable platform-specific scheduler #13161

Conversation

yannicks1 commented Feb 12, 2025 • edited by github-actions bot Loading

github-actions bot commented Feb 12, 2025

comaniac commented Feb 12, 2025

yannicks1 commented Feb 14, 2025

youkaichao Feb 15, 2025

Choose a reason for hiding this comment

youkaichao Feb 15, 2025

Choose a reason for hiding this comment

yannicks1 commented Feb 17, 2025

youkaichao Feb 18, 2025

Choose a reason for hiding this comment

youkaichao Feb 18, 2025

Choose a reason for hiding this comment

mergify bot commented Feb 18, 2025

yannicks1 commented Feb 18, 2025

youkaichao left a comment

Choose a reason for hiding this comment

yannicks1 commented Feb 12, 2025 •

edited by github-actions bot

Loading