Kernel Build Failure After Increasing Rootfs Size

[mrbilandi]

Hi,

I encountered an issue when attempting to rebuild the kernel in the CVA6 SDK after adding a significant amount of data (around 10GB) to the root filesystem (rootfs). During the build process, I receive multiple warnings, followed by a fatal error related to memory layout conflicts. Here is the relevant part of the build log:

/media/.../riscv64-buildroot-linux-gnu-ld: warning: dot moved backwards before .init.pi' /media/.../riscv64-buildroot-linux-gnu-ld: section .init.data VMA wraps around address space /media/.../riscv64-buildroot-linux-gnu-ld: section .init.pi LMA [0000000051af7600,0000000051af9a72] overlaps section .init.data LMA [0000000000800000,00000000d1af75ff] ... arch/riscv/mm/init.o: in function .L251':
init.c:(.init.text+0xbfc): relocation truncated to fit: R_RISCV_CALL_PLT against symbol __pi_set_satp_mode_from_cmdline' defined in .init.pi.text section in arch/riscv/kernel/pi/cmdline_early.pi.o arch/riscv/kernel/pi/cmdline_early.pi.o: in function _pi.L0 ':
__pi_cmdline_early.c:(.init.pi.text+0x8e): relocation truncated to fit: R_RISCV_CALL_PLT against symbol __pi_strlen' defined in .text section in arch/riscv/lib/strlen.o arch/riscv/kernel/pi/string.pi.o: in function __pi_strlcpy':
__pi_string.c:(.init.pi.text+0x14a): relocation truncated to fit: R_RISCV_CALL_PLT against symbol __pi_strlen' defined in .text section in arch/riscv/lib/strlen.o arch/riscv/kernel/pi/lib-fdt.pi.o: in function __pi_fdt_first_subnode':
__pi_fdt.c:(.init.pi.text+0x5b2): relocation truncated to fit: R_RISCV_CALL_PLT against symbol __pi_strlen' defined in .text section in arch/riscv/lib/strlen.o arch/riscv/kernel/pi/lib-fdt_ro.pi.o: in function __pi_fdt_get_name':
__pi_fdt_ro.c:(.init.pi.text+0x404): relocation truncated to fit: R_RISCV_CALL_PLT against symbol `__pi_strlen' defined in .text section in arch/riscv/lib/strlen.o
make[4]: *** [scripts/Makefile.vmlinux:36: vmlinux] Error 1
make[3]: *** [Makefile:1250: vmlinux] Error 2
make[2]: *** [package/pkg-generic.mk:293: /media/.../cva6-sdk/buildroot/output/build/linux-v6.5/.stamp_built] Error 2
make[1]: *** [Makefile:82: _all] Error 2
make[1]: Leaving directory '/media/.../cva6-sdk/buildroot'
make: *** [Makefile:113: /media.../cva6-sdk/install64/vmlinux] Error 2

This error appears to be related to the increased size of the rootfs and its interaction with the .init sections, particularly .init.text, .init.data, and .init.pi in the memory layout.
Steps to Reproduce:

1.Add approximately 10GB of files to the rootfs.
2. Rebuild the linux : make images

Do I need to adjust the memory layout for the .init.text, .init.data, or any other sections to prevent overlapping, and if so, what parameters should be changed?

Is there a maximum rootfs size that the current setup supports, and what would be the best approach to handle such a large rootfs?

Thanks for your help.

[Moschn]

For the linux image built in this repo, we use initramfs as a file system. This makes handling easier, as you only need a single file to boot it. However, as its name hints already, initRAMfs puts the files into memory, and thus, it is not designed to support extremely large root file systems. Your error seems to hint that the linker tries to add the 10gb to the image but hits some edgecase (section .init.data VMA wraps around address space). Due to its backing in RAM there also is an inherent limit on how big the initramfs can become. So even if you manage to fix the build errors reported here, you will need a device with more than 10gb RAM to boot this image. For a rootfs this large, I would propose you either switch to a different rootfs, or you could also add NFS or other file systems that can be mounted over the network.

[mrbilandi]

Hi @Moschn,

Thanks a lot for your explanation, I understand the issue now, especially with the RAM limitations when using initramfs.

Just to clarify, what would be the maximum allowed file size I can put into the rootfs within the current setup using initramfs? I'm trying to gauge how much I can include before hitting these limitations.

Thanks again for your help.

[Moschn]

Hmm, I do not know the exact limit, but I would probably be careful after the rootfs size exceeds 50% of the available memory on your board. The Genesys2 board has 1gb of RAM, thus I would try to stay within 512mb.

[mrbilandi]

Is it possible to add a new partition with, for example, an ext4 file system, and then use that partition to store large files? I'm not too familiar with the process of switching to a different rootfs like ext4. Could you provide me with some hints or steps on how to do this?

[Moschn]

Sorry, I also do not have experience doing this. If I had to do this, I would start looking at buildroot (which we are also using in this repo). I believe it can be configured to use a different rootfs.

[Moschn]

Oh, you could also just create a new partition on the SD card and mount that. In this way, you do not have to add it to the rootfs.

[mrbilandi]

I have created a partition in the remaining part of the SD card using Disks in Ubuntu. The new partition is 1GB in size, formatted as FAT, and named data. However, after booting Linux, I cannot see the partition to mount it. To investigate, I used the following commands:
dmesg | grep sd
blkid
cat /proc/partitions
lsblk

But there is no output when I run them.

Is this the right way to create a new partition as you mentioned?
Could you please guide me on how to mount this new partition?

Thanks a lot.

[Moschn]

Sorry, I do not have an FPGA at hand and cannot help you that easily.

I believe you are creating the partition correctly. Maybe you just have to add the SD driver to your linux image using buildroot (e.g., cd buildroot, make menuconfig or make linux-menuconfig) .

[mrbilandi]

I have enabled the "MMC block device driver" (CONFIG_MMC_BLOCK), the "Secure Digital Host Controller Interface" (SDHCI) (CONFIG_MMC_SDHCI), and "MMC/SD/SDIO over SPI" (which was already enabled). However, when I run the previous commands I mentioned, there is still no output, and I cannot see the new partition. just When I run dmesg | grep sd, I see the following:
[ 90.722921] sdhci: Secure Digital Host Controller Interface driver
[ 90.740952] sdhci: Copyright(c) Pierre Ossman
[ 90.763114] sdhci-pltfm: SDHCI platform and OF driver helper
These are the drivers I have enabled.