[VLM] Report multi_modal_placeholders in output #10407

lk-chen · 2024-11-18T00:12:42Z

Report multi_modal_placeholders in output

This helps inspect VLM model performance, e.g. distinguish text tokens and multi-modal embeddings.

related: #8346

github-actions · 2024-11-18T00:12:57Z

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

Add ready label to the PR
Enable auto-merge.

🚀

DarkLight1337

Thanks for improving the developer experience!

DarkLight1337 · 2024-11-18T03:48:37Z

Do you get these test errors locally?

lk-chen · 2024-11-18T05:39:25Z

Do you get these test errors locally?

I cannot reproduce those errors due to lacking permission to llama models.
But by reading the errors, I made commit 8c5fdf8 and hope to pass the test

DarkLight1337 · 2024-11-18T05:41:42Z

Please fix the lint errors.

* confirm that `offline_inference_vision_language.py` and `benchmark_throughput.py` runs * FIXME: the placeholders in output, however, is empty - will fix in next commit Signed-off-by: Linkun Chen <lkchen+anyscale@github.com>

* add test for pixtral * fix a typo Signed-off-by: Linkun Chen <lkchen+anyscale@github.com>

Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu> Signed-off-by: Linkun Chen <lkchen+anyscale@github.com>

…ect#10383) Signed-off-by: youkaichao <youkaichao@gmail.com> Signed-off-by: Linkun Chen <lkchen+anyscale@github.com>

…odels (vllm-project#10374) Signed-off-by: Roger Wang <ywang@roblox.com> Signed-off-by: Linkun Chen <lkchen+anyscale@github.com>

Signed-off-by: Chendi Xue <chendi.xue@intel.com> Signed-off-by: Linkun Chen <lkchen+anyscale@github.com>

…ject#10394) Signed-off-by: Isotr0py <2037008807@qq.com> Signed-off-by: Linkun Chen <lkchen+anyscale@github.com>

Signed-off-by: youkaichao <youkaichao@gmail.com> Signed-off-by: Linkun Chen <lkchen+anyscale@github.com>

Signed-off-by: Kunshang Ji <kunshang.ji@intel.com> Signed-off-by: Linkun Chen <lkchen+anyscale@github.com>

…ject#10403) Signed-off-by: imkero <kerorek@outlook.com> Signed-off-by: Linkun Chen <lkchen+anyscale@github.com>

vllm-project#10392) Signed-off-by: wchen61 <wchen61@foxmail.com> Signed-off-by: Linkun Chen <lkchen+anyscale@github.com>

Signed-off-by: Cyrus Leung <tlleungac@connect.ust.hk> Signed-off-by: Linkun Chen <lkchen+anyscale@github.com>

Signed-off-by: Linkun Chen <lkchen+anyscale@github.com>

…m-project#10327) Signed-off-by: Isotr0py <2037008807@qq.com> Signed-off-by: Linkun Chen <lkchen+anyscale@github.com>

…vllm-project#10375) Signed-off-by: Hollow Man <hollowman@opensuse.org> Signed-off-by: Linkun Chen <lkchen+anyscale@github.com>

… optional argument also require it to be passed as kwargs, to avoid breaking existing code. Signed-off-by: Linkun Chen <lkchen+anyscale@github.com>

mypy is not smart enough to validate kwargs Signed-off-by: Linkun Chen <lkchen+anyscale@github.com>

lk-chen · 2024-11-18T06:09:32Z

Please fix the lint errors.

Done. Could you help me remove the accidentally-added reviewers? Those were added when I tried to fix DCO

tests/models/decoder_only/vision_language/test_pixtral.py

Signed-off-by: Linkun Chen <lkchen+anyscale@github.com>

comaniac

LGTM. The reviewers were added automatically due to code owners so it's fine.

Also cc @ywang96 @mgoin

Signed-off-by: Linkun Chen <lkchen+anyscale@github.com>

Signed-off-by: Linkun Chen <lkchen+anyscale@github.com> Signed-off-by: Maxime Fournioux <55544262+mfournioux@users.noreply.github.com>

Signed-off-by: Linkun Chen <lkchen+anyscale@github.com> Signed-off-by: rickyx <rickyx@anyscale.com>

Signed-off-by: Linkun Chen <lkchen+anyscale@github.com> Signed-off-by: Tyler Michael Smith <tyler@neuralmagic.com>

Signed-off-by: Linkun Chen <lkchen+anyscale@github.com>

lk-chen requested review from DarkLight1337 and ywang96 as code owners November 18, 2024 00:12

DarkLight1337 approved these changes Nov 18, 2024

View reviewed changes

DarkLight1337 added the ready ONLY add when PR is ready to merge/full CI is needed label Nov 18, 2024

Linkun Chen and others added 20 commits November 18, 2024 05:52

Patch multi_modal_placeholders to RequestOutput

678c291

* confirm that `offline_inference_vision_language.py` and `benchmark_throughput.py` runs * FIXME: the placeholders in output, however, is empty - will fix in next commit Signed-off-by: Linkun Chen <lkchen+anyscale@github.com>

pipe multi_modal_placeholders from intput to final output

a1cdcb3

* add test for pixtral * fix a typo Signed-off-by: Linkun Chen <lkchen+anyscale@github.com>

[V1] Add code owners for V1 (vllm-project#10397)

f60964a

Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu> Signed-off-by: Linkun Chen <lkchen+anyscale@github.com>

[2/N][torch.compile] make compilation cfg part of vllm cfg (vllm-proj…

578e482

…ect#10383) Signed-off-by: youkaichao <youkaichao@gmail.com> Signed-off-by: Linkun Chen <lkchen+anyscale@github.com>

[V1] Refactor model executable interface for all text-only language m…

7fa97cf

…odels (vllm-project#10374) Signed-off-by: Roger Wang <ywang@roblox.com> Signed-off-by: Linkun Chen <lkchen+anyscale@github.com>

[CI/Build] Fix IDC hpu [Device not found] issue (vllm-project#10384)

629f512

Signed-off-by: Chendi Xue <chendi.xue@intel.com> Signed-off-by: Linkun Chen <lkchen+anyscale@github.com>

[Bugfix][CPU] Fix CPU embedding runner with tensor parallel (vllm-pro…

7539ab8

…ject#10394) Signed-off-by: Isotr0py <2037008807@qq.com> Signed-off-by: Linkun Chen <lkchen+anyscale@github.com>

[platforms] refactor cpu code (vllm-project#10402)

bce660d

Signed-off-by: youkaichao <youkaichao@gmail.com> Signed-off-by: Linkun Chen <lkchen+anyscale@github.com>

[Hardware] [HPU]add mark_step for hpu (vllm-project#10239)

305708b

Signed-off-by: Kunshang Ji <kunshang.ji@intel.com> Signed-off-by: Linkun Chen <lkchen+anyscale@github.com>

[Bugfix] Fix mrope_position_delta in non-last prefill chunk (vllm-pro…

871a773

…ject#10403) Signed-off-by: imkero <kerorek@outlook.com> Signed-off-by: Linkun Chen <lkchen+anyscale@github.com>

[Misc] Enhance offline_inference to support user-configurable paramet… (

242bb53

vllm-project#10392) Signed-off-by: wchen61 <wchen61@foxmail.com> Signed-off-by: Linkun Chen <lkchen+anyscale@github.com>

Fix initialization

f5312d3

Signed-off-by: Cyrus Leung <tlleungac@connect.ust.hk> Signed-off-by: Linkun Chen <lkchen+anyscale@github.com>

Run isort

439e324

Signed-off-by: Linkun Chen <lkchen+anyscale@github.com>

isort

60815f2

Signed-off-by: Linkun Chen <lkchen+anyscale@github.com>

isort

ec46755

Signed-off-by: Linkun Chen <lkchen+anyscale@github.com>

[Misc] Add uninitialized params tracking for AutoWeightsLoader (vll…

ce3ae6f

…m-project#10327) Signed-off-by: Isotr0py <2037008807@qq.com> Signed-off-by: Linkun Chen <lkchen+anyscale@github.com>

[Bugfix] Ignore ray reinit error when current platform is ROCm or XPU (…

466b2cf

…vllm-project#10375) Signed-off-by: Hollow Man <hollowman@opensuse.org> Signed-off-by: Linkun Chen <lkchen+anyscale@github.com>

update RequestOutput.__init__() to take multi_modal_placeholders as…

3f092ce

… optional argument also require it to be passed as kwargs, to avoid breaking existing code. Signed-off-by: Linkun Chen <lkchen+anyscale@github.com>

update RequestOutput.__init__() to take multi_modal_placeholders as…

dd8427e

… optional argument also require it to be passed as kwargs, to avoid breaking existing code. Signed-off-by: Linkun Chen <lkchen+anyscale@github.com>

update RequestOutput.__init__() to take multi_modal_placeholders as…

76ac8b0

… optional argument also require it to be passed as kwargs, to avoid breaking existing code. Signed-off-by: Linkun Chen <lkchen+anyscale@github.com>

lk-chen force-pushed the main branch from ee8697e to 76ac8b0 Compare November 18, 2024 05:52

lk-chen requested a review from WoosukKwon as a code owner November 18, 2024 05:52

lk-chen requested review from robertgshaw2-redhat, njhill, comaniac and alexm-redhat as code owners November 18, 2024 05:52

Merge branch 'vllm-project:main' into main

904e925

mergify bot added the ci/build label Nov 18, 2024

lk-chen marked this pull request as draft November 18, 2024 05:55

disable mypy type check

c963a25

mypy is not smart enough to validate kwargs Signed-off-by: Linkun Chen <lkchen+anyscale@github.com>

lk-chen marked this pull request as ready for review November 18, 2024 06:00

disable mypy type check

470fbd3

mypy is not smart enough to validate kwargs Signed-off-by: Linkun Chen <lkchen+anyscale@github.com>

DarkLight1337 removed request for comaniac, njhill, WoosukKwon, alexm-redhat and robertgshaw2-redhat November 18, 2024 06:10

DarkLight1337 reviewed Nov 18, 2024

View reviewed changes

tests/models/decoder_only/vision_language/test_pixtral.py Outdated Show resolved Hide resolved

remove unnecessary debug code

550be23

Signed-off-by: Linkun Chen <lkchen+anyscale@github.com>

comaniac approved these changes Nov 18, 2024

View reviewed changes

DarkLight1337 merged commit c7dec92 into vllm-project:main Nov 18, 2024
50 checks passed

coolkp pushed a commit to coolkp/vllm that referenced this pull request Nov 20, 2024

[VLM] Report multi_modal_placeholders in output (vllm-project#10407)

dd84f40

Signed-off-by: Linkun Chen <lkchen+anyscale@github.com>

KuntaiDu pushed a commit to KuntaiDu/vllm that referenced this pull request Nov 20, 2024

[VLM] Report multi_modal_placeholders in output (vllm-project#10407)

add442e

Signed-off-by: Linkun Chen <lkchen+anyscale@github.com>

rickyyx pushed a commit to rickyyx/vllm that referenced this pull request Nov 20, 2024

[VLM] Report multi_modal_placeholders in output (vllm-project#10407)

40753ce

Signed-off-by: Linkun Chen <lkchen+anyscale@github.com> Signed-off-by: rickyx <rickyx@anyscale.com>

prashantgupta24 pushed a commit to opendatahub-io/vllm that referenced this pull request Dec 3, 2024

[VLM] Report multi_modal_placeholders in output (vllm-project#10407)

663fb57

Signed-off-by: Linkun Chen <lkchen+anyscale@github.com>

sleepwalker2017 pushed a commit to sleepwalker2017/vllm that referenced this pull request Dec 13, 2024

[VLM] Report multi_modal_placeholders in output (vllm-project#10407)

2813a5f

Signed-off-by: Linkun Chen <lkchen+anyscale@github.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[VLM] Report multi_modal_placeholders in output #10407

[VLM] Report multi_modal_placeholders in output #10407

lk-chen commented Nov 18, 2024 •

edited by github-actions bot

Loading

github-actions bot commented Nov 18, 2024

DarkLight1337 left a comment

DarkLight1337 commented Nov 18, 2024

lk-chen commented Nov 18, 2024

DarkLight1337 commented Nov 18, 2024

lk-chen commented Nov 18, 2024 •

edited

Loading

comaniac left a comment

[VLM] Report multi_modal_placeholders in output #10407

[VLM] Report multi_modal_placeholders in output #10407

Conversation

lk-chen commented Nov 18, 2024 • edited by github-actions bot Loading

github-actions bot commented Nov 18, 2024

DarkLight1337 left a comment

Choose a reason for hiding this comment

DarkLight1337 commented Nov 18, 2024

lk-chen commented Nov 18, 2024

DarkLight1337 commented Nov 18, 2024

lk-chen commented Nov 18, 2024 • edited Loading

comaniac left a comment

Choose a reason for hiding this comment

lk-chen commented Nov 18, 2024 •

edited by github-actions bot

Loading

lk-chen commented Nov 18, 2024 •

edited

Loading