Skip to content

Boot failure on s390 with llvm-objcopy #1996

New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom
Closed
nathanchance opened this issue Feb 16, 2024 · 4 comments
Closed

Boot failure on s390 with llvm-objcopy #1996

nathanchance opened this issue Feb 16, 2024 · 4 comments
Assignees
@nathanchance
Labels
[ARCH] s390 This bug impacts ARCH=s390 [BUG] linux A bug that should be fixed in the mainline kernel. [FIXED][LINUX] 6.9 This bug was fixed in Linux 6.9 [TOOL] llvm-objcopy The issue is relevant to LLVM objcopy

Comments

@nathanchance
Copy link
Member

I have been testing the s390 llvm-objcopy pull request, which results in no new build warnings or errors on next-20240215:

$ make -skj"$(nproc)" ARCH=s390 LLVM=1 OBJDUMP=s390x-linux-gnu-objdump mrproper defconfig all

However, the resulting kernel image does not boot:

$ boot-qemu.py -a s390 -k .
...
Linux version 6.8.0-rc4-next-20240215 (nathan@dev-fedora.aadp) #1 SMP Thu Feb 15 19:50:28 MST 2024
Out of memory allocating 6d7800 bytes 8 aligned in range 0:20000000
Reserved memory ranges:
0000000000000000 a394c3c30d90cdaf DECOMPRESSOR
Usable online memory ranges (info source: sclp read info [3]):
0000000000000000 0000000020000000
Usable online memory total: 20000000 Reserved: a394c3c30d90cdaf Free: 0
Call Trace:
(sp:0000000000033e90 [<0000000000012fbc>] physmem_alloc_top_down+0x5c/0x104)
 sp:0000000000033f00 [<0000000000011d56>] startup_kernel+0x3a6/0x77c
 sp:0000000000033f60 [<00000000000100f4>] startup_normal+0xd4/0xd4


 -- System halted

With OBJCOPY=s390x-linux-gnu-objcopy, there are no issues when booting:

$ boot-qemu.py -a s390 -k .
...

KASLR disabled: CPU has no PRNG
KASLR disabled: CPU has no PRNG
[    0.851887] Linux version 6.8.0-rc4-next-20240215 (nathan@dev-fedora.aadp) (ClangBuiltLinux clang version 19.0.0git (https://github.com/llvm/llvm-project db4ea21dfde307f4fc873a6fe56791cd7ae3f0a1), ClangBuiltLinux LLD 19.0.0) #1 SMP Thu Feb 15 19:53:34 MST 2024
[    0.854515] setup: Linux is running under KVM in 64-bit mode
[    0.878762] setup: The maximum memory size is 512MB
[    0.879514] setup: Relocating AMODE31 section of size 0x00003000
...

Looking at the files that are generated by objcopy during the build, I notice that arch/s390/boot/info.bin is different between the two builds...

$ file {gnu,llvm}-objcopy/arch/s390/boot/info.bin
gnu-objcopy/arch/s390/boot/info.bin:  data
llvm-objcopy/arch/s390/boot/info.bin: empty

If I take a directory built with OBJCOPY=llvm-obcopy and move the arch/s390/boot/info.bin from a directory built with OBJCOPY=s390x-linux-gnu-objcopy and run make again, the resulting image boots.

Looking at the build logs, info.bin is generated from vmlinux with

llvm-objcopy -O binary --only-section=.vmlinux.info --set-section-flags .vmlinux.info=load vmlinux arch/s390/boot/info.bin

$ llvm-readelf --section-headers vmlinux | rg -F .vmlinux.info
  [42] .vmlinux.info     PROGBITS        0000000000000000 1d45000 000078 00   W  0   0  1

$ llvm-objcopy -O binary --only-section=.vmlinux.info --set-section-flags .vmlinux.info=load vmlinux llvm-info.bin

$ s390x-linux-gnu-objcopy -O binary --only-section=.vmlinux.info --set-section-flags .vmlinux.info=load vmlinux gnu-info.bin

$ file {gnu,llvm}-info.bin
gnu-info.bin:  data
llvm-info.bin: empty

This was done initially as part of https://git.kernel.org/linus/369f91c374514f9491d52fec12f7ee9ef6d44b23.

cc @MaskRay and @uweigand (even though this doesn't seem like anything specific to s390 but just in case)
@nathanchance nathanchance added [BUG] Untriaged Something isn't working [TOOL] llvm-objcopy The issue is relevant to LLVM objcopy [ARCH] s390 This bug impacts ARCH=s390 labels Feb 16, 2024
@uweigand
Copy link

I had a quick look, and it seems llvm-objcopy with -O binary only outputs sections marked as ALLOC. Due the --sec-section-flags argument, the flags of .vmlinux.info are set to LOAD only, which means the ALLOC flag is removed.

Now, I'm not quite sure what LOAD without ALLOC even means exactly. (ALLOC without LOAD is used for .bss, ALLOC and LOAD together are used for sections mapped at run-time, and neither ALLOC nor LOAD is used for debug and similar sections.)

Should the kernel Makefile use --set-section-flags .vmlinux.info=alloc,load instead here?

@nathanchance
Copy link
Member Author

Should the kernel Makefile use --set-section-flags .vmlinux.info=alloc,load instead here?

Yeah, seems entirely reasonable to me, especially since I see no visible difference with GNU objcopy.

Patch submitted: https://lore.kernel.org/20240216-s390-fix-boot-with-llvm-objcopy-v1-1-0ac623daf42b@kernel.org/

@nathanchance nathanchance added [BUG] linux A bug that should be fixed in the mainline kernel. [PATCH] Submitted A patch has been submitted for review and removed [BUG] Untriaged Something isn't working labels Feb 16, 2024
@nathanchance nathanchance self-assigned this Feb 16, 2024
nathanchance added a commit to nathanchance/continuous-integration2 that referenced this issue Feb 16, 2024
@nathanchance
generator: yml: Use more LLVM utilities for s390 builds on linux-next…
e85965b 
… with LLVM tip of tree

s390 support in ld.lld has recently landed in the main branch of LLVM.
With this, it makes sense to start cutting s390 over to full LLVM=1,
versus the current situation of just CC=clang.  linux-next is the only
tree that currently has the necessary changes to support ld.lld but they
will trickle down to other trees in time.

llvm-objcopy currently works on LLVM main as well but that needs a
kernel side fix to boot properly, so we will stick with GNU objcopy
until that is fixed.

Closes: ClangBuiltLinux#710
Link: ClangBuiltLinux/linux#1996
Link: llvm/llvm-project@fe3406e
Link: llvm/llvm-project@3c02cb7
Signed-off-by: Nathan Chancellor <nathan@kernel.org>
@nathanchance nathanchance mentioned this issue Feb 16, 2024
Closed
@nathanchance
Copy link
Member Author

Patch accepted: https://git.kernel.org/s390/c/9ea30fd166e9b869546059f714beab78cb150b11

@nathanchance nathanchance added [PATCH] Accepted A submitted patch has been accepted upstream and removed [PATCH] Submitted A patch has been submitted for review labels Feb 20, 2024
intel-lab-lkp pushed a commit to intel-lab-lkp/linux that referenced this issue Feb 21, 2024
@nathanchance @hcahca
s390/boot: add 'alloc' to info.bin .vmlinux.info section flags
9ea30fd 
When attempting to boot a kernel compiled with OBJCOPY=llvm-objcopy,
there is a crash right at boot:

  Out of memory allocating 6d7800 bytes 8 aligned in range 0:20000000
  Reserved memory ranges:
  0000000000000000 a394c3c30d90cdaf DECOMPRESSOR
  Usable online memory ranges (info source: sclp read info [3]):
  0000000000000000 0000000020000000
  Usable online memory total: 20000000 Reserved: a394c3c30d90cdaf Free: 0
  Call Trace:
  (sp:0000000000033e90 [<0000000000012fbc>] physmem_alloc_top_down+0x5c/0x104)
   sp:0000000000033f00 [<0000000000011d56>] startup_kernel+0x3a6/0x77c
   sp:0000000000033f60 [<00000000000100f4>] startup_normal+0xd4/0xd4

GNU objcopy does not have any issues. Looking at differences between the
object files in each build reveals info.bin does not get properly
populated with llvm-objcopy, which results in an empty .vmlinux.info
section.

  $ file {gnu,llvm}-objcopy/arch/s390/boot/info.bin
  gnu-objcopy/arch/s390/boot/info.bin:  data
  llvm-objcopy/arch/s390/boot/info.bin: empty

  $ llvm-readelf --section-headers {gnu,llvm}-objcopy/arch/s390/boot/vmlinux | rg 'File:|\.vmlinux\.info|\.decompressor\.syms'
  File: gnu-objcopy/arch/s390/boot/vmlinux
    [12] .vmlinux.info     PROGBITS        0000000000034000 035000 000078 00  WA  0   0  1
    [13] .decompressor.syms PROGBITS       0000000000034078 035078 000b00 00  WA  0   0  1
  File: llvm-objcopy/arch/s390/boot/vmlinux
    [12] .vmlinux.info     PROGBITS        0000000000034000 035000 000000 00  WA  0   0  1
    [13] .decompressor.syms PROGBITS       0000000000034000 035000 000b00 00  WA  0   0  1

Ulrich points out that llvm-objcopy only copies sections marked as alloc
with a binary output target, whereas the .vmlinux.info section is only
marked as load. Add 'alloc' in addition to 'load', so that both objcopy
implementations work properly:

  $ file {gnu,llvm}-objcopy/arch/s390/boot/info.bin
  gnu-objcopy/arch/s390/boot/info.bin:  data
  llvm-objcopy/arch/s390/boot/info.bin: data

  $ llvm-readelf --section-headers {gnu,llvm}-objcopy/arch/s390/boot/vmlinux | rg 'File:|\.vmlinux\.info|\.decompressor\.syms'
  File: gnu-objcopy/arch/s390/boot/vmlinux
    [12] .vmlinux.info     PROGBITS        0000000000034000 035000 000078 00  WA  0   0  1
    [13] .decompressor.syms PROGBITS       0000000000034078 035078 000b00 00  WA  0   0  1
  File: llvm-objcopy/arch/s390/boot/vmlinux
    [12] .vmlinux.info     PROGBITS        0000000000034000 035000 000078 00  WA  0   0  1
    [13] .decompressor.syms PROGBITS       0000000000034078 035078 000b00 00  WA  0   0  1

Closes: ClangBuiltLinux#1996
Link: llvm/llvm-project@3c02cb7
Suggested-by: Ulrich Weigand <ulrich.weigand@de.ibm.com>
Signed-off-by: Nathan Chancellor <nathan@kernel.org>
Link: https://lore.kernel.org/r/20240216-s390-fix-boot-with-llvm-objcopy-v1-1-0ac623daf42b@kernel.org
Signed-off-by: Heiko Carstens <hca@linux.ibm.com>
nathanchance added a commit to nathanchance/continuous-integration2 that referenced this issue Feb 21, 2024
@nathanchance
generator: yml: Use LLVM=1 for ARCH=s390 on linux-next with tip of tr…
c41724b 
…ee LLVM

With LLVM main, which has the necessary changes to ld.lld and
llvm-objcopy to support s390, and linux-next, which has the necessary
changes to the kernel to support these tools, ARCH=s390 builds cleanly
with LLVM=1. Cut builds that satisfy these conditions over to LLVM=1 so
that regressions with this working setup do not occur.

Link: llvm/llvm-project@fe3406e
Link: llvm/llvm-project@3c02cb7
Link: ClangBuiltLinux/linux#859
Link: ClangBuiltLinux/linux#1991
Link: ClangBuiltLinux/linux#1992
Link: ClangBuiltLinux/linux#1993
Link: ClangBuiltLinux/linux#1994
Link: ClangBuiltLinux/linux#1995
Link: ClangBuiltLinux/linux#1996
Signed-off-by: Nathan Chancellor <nathan@kernel.org>
@nathanchance nathanchance mentioned this issue Feb 21, 2024
Merged
@nathanchance
Copy link
Member Author

Merged into mainline: https://git.kernel.org/linus/9ea30fd166e9b869546059f714beab78cb150b11

@nathanchance nathanchance added [FIXED][LINUX] 6.9 This bug was fixed in Linux 6.9 and removed [PATCH] Accepted A submitted patch has been accepted upstream labels Mar 15, 2024
# for free to join this conversation on GitHub. Already have an account? # to comment
Assignees

@nathanchance nathanchance

Labels
[ARCH] s390 This bug impacts ARCH=s390 [BUG] linux A bug that should be fixed in the mainline kernel. [FIXED][LINUX] 6.9 This bug was fixed in Linux 6.9 [TOOL] llvm-objcopy The issue is relevant to LLVM objcopy
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@uweigand @nathanchance