Fix: Build error seen on Power Architecture #10421

mikejuliet13 · 2024-11-18T13:14:09Z

The issue in the vLLM library involves the introduction of the -mf16c flag and FP16 vector types, which are specific to x86 architecture. These changes cause build failures on Power architecture, though the library builds and functions correctly on x86. To resolve this, architecture-specific checks will be implemented. The -mf16c flag will not be applied on the Power architecture systems, while FP32Vec types will be used as a fallback for unsupported FP16 vector types on Power architecture. This fix ensures compatibility and successful builds across different architectures.

github-actions · 2024-11-18T13:14:29Z

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

Add ready label to the PR
Enable auto-merge.

🚀

sumitd2 · 2024-11-18T13:35:13Z

cc: @khluu

simon-mo · 2024-11-18T18:01:00Z

Please fix clang-format

mgoin · 2024-11-18T18:52:40Z

cmake/cpu_extension.cmake

+if (CMAKE_SYSTEM_PROCESSOR STREQUAL "ppc64le")
+    list(APPEND CXX_COMPILE_FLAGS
+    "-fopenmp"
+    "-DVLLM_CPU_EXTENSION")
+else()
 list(APPEND CXX_COMPILE_FLAGS
    "-fopenmp"
    "-mf16c"
    "-DVLLM_CPU_EXTENSION")
+endif()


Maybe you could invert this to always apply

list(APPEND CXX_COMPILE_FLAGS "-fopenmp" "-DVLLM_CPU_EXTENSION")

And then add a case just if the processor is x86 to add "-mf16c"

similar to this block -

vllm/cmake/cpu_extension.cmake

Line 57 in 284203f

if (AVX512_FOUND AND NOT AVX512_DISABLED)

Above flag was applied for all architectures. We know for Power its not applicable so added exclusive check to avoid it for Power.
Adding exclusive check for x86 may avoid it applying for other supported architectures.

x86 is the only architecture that supports this flag, so I don't see how it would hurt
Please see this incoming PR for ARM support for an illustration of what I'm asking for here
https://github.com/vllm-project/vllm/pull/9957/files#diff-46b07f808e558aff84dcc365e53a91400c4be399757ff40c2da4278a50e89fbdL19-R24

DarkLight1337 · 2024-11-19T03:32:53Z

Please merge from main to fix the failing tests.

Signed-off-by: Manjul Mohan <manjul.mohan@ibm.com>

Signed-off-by: B-201 <Joy25810@foxmail.com> Signed-off-by: Manjul Mohan <manjul.mohan@ibm.com>

…0414) Signed-off-by: Isotr0py <2037008807@qq.com> Signed-off-by: Manjul Mohan <manjul.mohan@ibm.com>

Signed-off-by: B-201 <Joy25810@foxmail.com> Signed-off-by: Manjul Mohan <manjul.mohan@ibm.com>

…#10406) Signed-off-by: youkaichao <youkaichao@gmail.com> Signed-off-by: Manjul Mohan <manjul.mohan@ibm.com>

Signed-off-by: ismael-dm <ismaeldm99@gmail.com> Signed-off-by: Manjul Mohan <manjul.mohan@ibm.com>

Signed-off-by: Andrew Nesbitt <andrewnez@gmail.com> Signed-off-by: Manjul Mohan <manjul.mohan@ibm.com>

Signed-off-by: mgoin <michael@neuralmagic.com> Signed-off-by: Manjul Mohan <manjul.mohan@ibm.com>

Signed-off-by: yan ma <yan.ma@intel.com> Signed-off-by: Manjul Mohan <manjul.mohan@ibm.com>

…t#9014) Signed-off-by: Angus Wang <wangjadehao@gmail.com> Signed-off-by: Manjul Mohan <manjul.mohan@ibm.com>

Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com> Signed-off-by: Manjul Mohan <manjul.mohan@ibm.com>

Signed-off-by: youkaichao <youkaichao@gmail.com> Signed-off-by: Manjul Mohan <manjul.mohan@ibm.com>

…ct#9929) Signed-off-by: rickyx <rickyx@anyscale.com> Signed-off-by: Manjul Mohan <manjul.mohan@ibm.com>

We have too many dependencies and all patch updates can be a little noisy. This is to have dependabot ignore all patch version updates. Signed-off-by: Manjul Mohan <manjul.mohan@ibm.com>

Signed-off-by: Jee Jee Li <pandaleefree@gmail.com> Signed-off-by: Manjul Mohan <manjul.mohan@ibm.com>

…form` (vllm-project#10358) Signed-off-by: Mengqing Cao <cmq0113@163.com> Signed-off-by: Manjul Mohan <manjul.mohan@ibm.com>

…rst (vllm-project#10433) Signed-off-by: Manjul Mohan <manjul.mohan@ibm.com>

…oject#10430) Signed-off-by: Travis Johnson <tsjohnso@us.ibm.com> Signed-off-by: Manjul Mohan <manjul.mohan@ibm.com>

Signed-off-by: Jee Jee Li <pandaleefree@gmail.com> Signed-off-by: Manjul Mohan <manjul.mohan@ibm.com>

Signed-off-by: Manjul Mohan <manjul.mohan@ibm.com>

Signed-off-by: youkaichao <youkaichao@gmail.com> Signed-off-by: Manjul Mohan <manjul.mohan@ibm.com>

Signed-off-by: Manjul Mohan <manjul.mohan@ibm.com>

mgoin

I guess we can land for now and fix the CPU cmake in the ARM support PRs

Signed-off-by: Manjul Mohan <manjul.mohan@ibm.com> Signed-off-by: B-201 <Joy25810@foxmail.com> Signed-off-by: Isotr0py <2037008807@qq.com> Signed-off-by: youkaichao <youkaichao@gmail.com> Signed-off-by: ismael-dm <ismaeldm99@gmail.com> Signed-off-by: Andrew Nesbitt <andrewnez@gmail.com> Signed-off-by: mgoin <michael@neuralmagic.com> Signed-off-by: yan ma <yan.ma@intel.com> Signed-off-by: Angus Wang <wangjadehao@gmail.com> Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com> Signed-off-by: rickyx <rickyx@anyscale.com> Signed-off-by: Jee Jee Li <pandaleefree@gmail.com> Signed-off-by: Mengqing Cao <cmq0113@163.com> Signed-off-by: Travis Johnson <tsjohnso@us.ibm.com> Co-authored-by: Manjul Mohan manjul.mohan@ibm.com <manjulmohan@ltcd97-lp2.aus.stglabs.ibm.com> Co-authored-by: B-201 <Joy25810@foxmail.com> Co-authored-by: Isotr0py <2037008807@qq.com> Co-authored-by: youkaichao <youkaichao@gmail.com> Co-authored-by: ismael-dm <ismaeldm99@gmail.com> Co-authored-by: Andrew Nesbitt <andrewnez@gmail.com> Co-authored-by: Michael Goin <michael@neuralmagic.com> Co-authored-by: Yan Ma <yan.ma@intel.com> Co-authored-by: Angus Wang <wangjadehao@gmail.com> Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com> Co-authored-by: Ricky Xu <rickyx@anyscale.com> Co-authored-by: Kevin H. Luu <kevin@anyscale.com> Co-authored-by: Jee Jee Li <pandaleefree@gmail.com> Co-authored-by: Mengqing Cao <cmq0113@163.com> Co-authored-by: Travis Johnson <tsjohnso@us.ibm.com> Co-authored-by: Russell Bryant <rbryant@redhat.com>

Signed-off-by: Manjul Mohan <manjul.mohan@ibm.com> Signed-off-by: B-201 <Joy25810@foxmail.com> Signed-off-by: Isotr0py <2037008807@qq.com> Signed-off-by: youkaichao <youkaichao@gmail.com> Signed-off-by: ismael-dm <ismaeldm99@gmail.com> Signed-off-by: Andrew Nesbitt <andrewnez@gmail.com> Signed-off-by: mgoin <michael@neuralmagic.com> Signed-off-by: yan ma <yan.ma@intel.com> Signed-off-by: Angus Wang <wangjadehao@gmail.com> Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com> Signed-off-by: rickyx <rickyx@anyscale.com> Signed-off-by: Jee Jee Li <pandaleefree@gmail.com> Signed-off-by: Mengqing Cao <cmq0113@163.com> Signed-off-by: Travis Johnson <tsjohnso@us.ibm.com> Co-authored-by: Manjul Mohan manjul.mohan@ibm.com <manjulmohan@ltcd97-lp2.aus.stglabs.ibm.com> Co-authored-by: B-201 <Joy25810@foxmail.com> Co-authored-by: Isotr0py <2037008807@qq.com> Co-authored-by: youkaichao <youkaichao@gmail.com> Co-authored-by: ismael-dm <ismaeldm99@gmail.com> Co-authored-by: Andrew Nesbitt <andrewnez@gmail.com> Co-authored-by: Michael Goin <michael@neuralmagic.com> Co-authored-by: Yan Ma <yan.ma@intel.com> Co-authored-by: Angus Wang <wangjadehao@gmail.com> Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com> Co-authored-by: Ricky Xu <rickyx@anyscale.com> Co-authored-by: Kevin H. Luu <kevin@anyscale.com> Co-authored-by: Jee Jee Li <pandaleefree@gmail.com> Co-authored-by: Mengqing Cao <cmq0113@163.com> Co-authored-by: Travis Johnson <tsjohnso@us.ibm.com> Co-authored-by: Russell Bryant <rbryant@redhat.com> Signed-off-by: rickyx <rickyx@anyscale.com>

Signed-off-by: Manjul Mohan <manjul.mohan@ibm.com> Signed-off-by: B-201 <Joy25810@foxmail.com> Signed-off-by: Isotr0py <2037008807@qq.com> Signed-off-by: youkaichao <youkaichao@gmail.com> Signed-off-by: ismael-dm <ismaeldm99@gmail.com> Signed-off-by: Andrew Nesbitt <andrewnez@gmail.com> Signed-off-by: mgoin <michael@neuralmagic.com> Signed-off-by: yan ma <yan.ma@intel.com> Signed-off-by: Angus Wang <wangjadehao@gmail.com> Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com> Signed-off-by: rickyx <rickyx@anyscale.com> Signed-off-by: Jee Jee Li <pandaleefree@gmail.com> Signed-off-by: Mengqing Cao <cmq0113@163.com> Signed-off-by: Travis Johnson <tsjohnso@us.ibm.com> Co-authored-by: Manjul Mohan manjul.mohan@ibm.com <manjulmohan@ltcd97-lp2.aus.stglabs.ibm.com> Co-authored-by: B-201 <Joy25810@foxmail.com> Co-authored-by: Isotr0py <2037008807@qq.com> Co-authored-by: youkaichao <youkaichao@gmail.com> Co-authored-by: ismael-dm <ismaeldm99@gmail.com> Co-authored-by: Andrew Nesbitt <andrewnez@gmail.com> Co-authored-by: Michael Goin <michael@neuralmagic.com> Co-authored-by: Yan Ma <yan.ma@intel.com> Co-authored-by: Angus Wang <wangjadehao@gmail.com> Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com> Co-authored-by: Ricky Xu <rickyx@anyscale.com> Co-authored-by: Kevin H. Luu <kevin@anyscale.com> Co-authored-by: Jee Jee Li <pandaleefree@gmail.com> Co-authored-by: Mengqing Cao <cmq0113@163.com> Co-authored-by: Travis Johnson <tsjohnso@us.ibm.com> Co-authored-by: Russell Bryant <rbryant@redhat.com> Signed-off-by: Tyler Michael Smith <tyler@neuralmagic.com>

Signed-off-by: Manjul Mohan <manjul.mohan@ibm.com> Signed-off-by: B-201 <Joy25810@foxmail.com> Signed-off-by: Isotr0py <2037008807@qq.com> Signed-off-by: youkaichao <youkaichao@gmail.com> Signed-off-by: ismael-dm <ismaeldm99@gmail.com> Signed-off-by: Andrew Nesbitt <andrewnez@gmail.com> Signed-off-by: mgoin <michael@neuralmagic.com> Signed-off-by: yan ma <yan.ma@intel.com> Signed-off-by: Angus Wang <wangjadehao@gmail.com> Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com> Signed-off-by: rickyx <rickyx@anyscale.com> Signed-off-by: Jee Jee Li <pandaleefree@gmail.com> Signed-off-by: Mengqing Cao <cmq0113@163.com> Signed-off-by: Travis Johnson <tsjohnso@us.ibm.com> Co-authored-by: Manjul Mohan manjul.mohan@ibm.com <manjulmohan@ltcd97-lp2.aus.stglabs.ibm.com> Co-authored-by: B-201 <Joy25810@foxmail.com> Co-authored-by: Isotr0py <2037008807@qq.com> Co-authored-by: youkaichao <youkaichao@gmail.com> Co-authored-by: ismael-dm <ismaeldm99@gmail.com> Co-authored-by: Andrew Nesbitt <andrewnez@gmail.com> Co-authored-by: Michael Goin <michael@neuralmagic.com> Co-authored-by: Yan Ma <yan.ma@intel.com> Co-authored-by: Angus Wang <wangjadehao@gmail.com> Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com> Co-authored-by: Ricky Xu <rickyx@anyscale.com> Co-authored-by: Kevin H. Luu <kevin@anyscale.com> Co-authored-by: Jee Jee Li <pandaleefree@gmail.com> Co-authored-by: Mengqing Cao <cmq0113@163.com> Co-authored-by: Travis Johnson <tsjohnso@us.ibm.com> Co-authored-by: Russell Bryant <rbryant@redhat.com>

mergify bot added the ci/build label Nov 18, 2024

DarkLight1337 added the ready ONLY add when PR is ready to merge/full CI is needed label Nov 18, 2024

simon-mo approved these changes Nov 18, 2024

View reviewed changes

mgoin reviewed Nov 18, 2024

View reviewed changes

mikejuliet13 force-pushed the vllm-power-issue branch from 4d0d6cd to 52e3d52 Compare November 19, 2024 12:13

Manjul Mohan manjul.mohan@ibm.com and others added 21 commits November 19, 2024 07:26

Fix: Build error seen on Power Architecture

0ce09ab

Signed-off-by: Manjul Mohan <manjul.mohan@ibm.com>

[Model][LoRA]LoRA support added for glm-4v (vllm-project#10418)

201a6da

Signed-off-by: B-201 <Joy25810@foxmail.com> Signed-off-by: Manjul Mohan <manjul.mohan@ibm.com>

[Model] Remove transformers attention porting in VITs (vllm-project#1…

9120161

…0414) Signed-off-by: Isotr0py <2037008807@qq.com> Signed-off-by: Manjul Mohan <manjul.mohan@ibm.com>

[Doc] Update doc for LoRA support in GLM-4V (vllm-project#10425)

d7b14ce

Signed-off-by: B-201 <Joy25810@foxmail.com> Signed-off-by: Manjul Mohan <manjul.mohan@ibm.com>

[5/N][torch.compile] torch.jit.script --> torch.compile (vllm-project…

06de800

…#10406) Signed-off-by: youkaichao <youkaichao@gmail.com> Signed-off-by: Manjul Mohan <manjul.mohan@ibm.com>

[Doc] Add documentation for Structured Outputs (vllm-project#9943)

d3a6317

Signed-off-by: ismael-dm <ismaeldm99@gmail.com> Signed-off-by: Manjul Mohan <manjul.mohan@ibm.com>

Fix open_collective value in FUNDING.yml (vllm-project#10426)

b2a1685

Signed-off-by: Andrew Nesbitt <andrewnez@gmail.com> Signed-off-by: Manjul Mohan <manjul.mohan@ibm.com>

[Model][Bugfix] Support TP for PixtralHF ViT (vllm-project#10405)

4c610a5

Signed-off-by: mgoin <michael@neuralmagic.com> Signed-off-by: Manjul Mohan <manjul.mohan@ibm.com>

[Hardware][XPU] AWQ/GPTQ support for xpu backend (vllm-project#10107)

96d56ba

Signed-off-by: yan ma <yan.ma@intel.com> Signed-off-by: Manjul Mohan <manjul.mohan@ibm.com>

[Kernel] Explicitly specify other value in tl.load calls (vllm-projec…

21da67d

…t#9014) Signed-off-by: Angus Wang <wangjadehao@gmail.com> Signed-off-by: Manjul Mohan <manjul.mohan@ibm.com>

[Kernel] Initial Machete W4A8 support + Refactors (vllm-project#9855)

2b855b1

Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com> Signed-off-by: Manjul Mohan <manjul.mohan@ibm.com>

[3/N][torch.compile] consolidate custom op logging (vllm-project#10399)

887d326

Signed-off-by: youkaichao <youkaichao@gmail.com> Signed-off-by: Manjul Mohan <manjul.mohan@ibm.com>

[ci][bugfix] fix kernel tests (vllm-project#10431)

db5dddb

Signed-off-by: youkaichao <youkaichao@gmail.com> Signed-off-by: Manjul Mohan <manjul.mohan@ibm.com>

[misc] partial prefix & random input generation benchmark (vllm-proje…

53e3a96

…ct#9929) Signed-off-by: rickyx <rickyx@anyscale.com> Signed-off-by: Manjul Mohan <manjul.mohan@ibm.com>

[ci/build] Have dependabot ignore all patch update (vllm-project#10436)

ec45058

We have too many dependencies and all patch updates can be a little noisy. This is to have dependabot ignore all patch version updates. Signed-off-by: Manjul Mohan <manjul.mohan@ibm.com>

[Bugfix]Fix Phi-3 BNB online quantization (vllm-project#10417)

cce69dc

Signed-off-by: Jee Jee Li <pandaleefree@gmail.com> Signed-off-by: Manjul Mohan <manjul.mohan@ibm.com>

[Platform][Refactor] Extract func get_default_attn_backend to `Plat…

2ce7cd4

…form` (vllm-project#10358) Signed-off-by: Mengqing Cao <cmq0113@163.com> Signed-off-by: Manjul Mohan <manjul.mohan@ibm.com>

Add openai.beta.chat.completions.parse example to structured_outputs.…

6372003

…rst (vllm-project#10433) Signed-off-by: Manjul Mohan <manjul.mohan@ibm.com>

[Bugfix] Guard for negative counter metrics to prevent crash (vllm-pr…

c0482f6

…oject#10430) Signed-off-by: Travis Johnson <tsjohnso@us.ibm.com> Signed-off-by: Manjul Mohan <manjul.mohan@ibm.com>

[Misc] Avoid misleading warning messages (vllm-project#10438)

3fcfe67

Signed-off-by: Jee Jee Li <pandaleefree@gmail.com> Signed-off-by: Manjul Mohan <manjul.mohan@ibm.com>

[Doc] Add the start of an arch overview page (vllm-project#10368)

392acf9

Signed-off-by: Manjul Mohan <manjul.mohan@ibm.com>

youkaichao and others added 2 commits November 19, 2024 07:26

[misc][plugin] improve plugin loading (vllm-project#10443)

30deada

Signed-off-by: youkaichao <youkaichao@gmail.com> Signed-off-by: Manjul Mohan <manjul.mohan@ibm.com>

Fix for clang-format (3.11)

cd96cde

Signed-off-by: Manjul Mohan <manjul.mohan@ibm.com>

mikejuliet13 force-pushed the vllm-power-issue branch from a07e591 to cd96cde Compare November 19, 2024 12:26

mikejuliet13 requested review from robertgshaw2-redhat, tlrmchlsmth, WoosukKwon, zhuohan123, youkaichao, alexm-redhat, comaniac and njhill as code owners November 19, 2024 12:26

Merge branch 'vllm-project:main' into vllm-power-issue

6b053cb

DarkLight1337 enabled auto-merge (squash) November 19, 2024 15:16

mgoin approved these changes Nov 19, 2024

View reviewed changes

DarkLight1337 disabled auto-merge November 19, 2024 15:18

DarkLight1337 enabled auto-merge (squash) November 19, 2024 15:18

youkaichao disabled auto-merge November 19, 2024 17:34

youkaichao merged commit 1ea291a into vllm-project:main Nov 19, 2024
70 of 72 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix: Build error seen on Power Architecture #10421

Fix: Build error seen on Power Architecture #10421

mikejuliet13 commented Nov 18, 2024 •

edited by github-actions bot

Loading

github-actions bot commented Nov 18, 2024

sumitd2 commented Nov 18, 2024

simon-mo commented Nov 18, 2024

mgoin Nov 18, 2024

mkumatag Nov 19, 2024

mikejuliet13 Nov 19, 2024

mgoin Nov 19, 2024

DarkLight1337 commented Nov 19, 2024

mgoin left a comment

Fix: Build error seen on Power Architecture #10421

Fix: Build error seen on Power Architecture #10421

Conversation

mikejuliet13 commented Nov 18, 2024 • edited by github-actions bot Loading

github-actions bot commented Nov 18, 2024

sumitd2 commented Nov 18, 2024

simon-mo commented Nov 18, 2024

mgoin Nov 18, 2024

Choose a reason for hiding this comment

mkumatag Nov 19, 2024

Choose a reason for hiding this comment

mikejuliet13 Nov 19, 2024

Choose a reason for hiding this comment

mgoin Nov 19, 2024

Choose a reason for hiding this comment

DarkLight1337 commented Nov 19, 2024

mgoin left a comment

Choose a reason for hiding this comment

mikejuliet13 commented Nov 18, 2024 •

edited by github-actions bot

Loading