Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Rufus silently fails to write an ISO containing large files in file mode w/ FAT32 #1007

Closed
4 of 10 tasks
sqweek opened this issue Aug 23, 2017 · 7 comments
Closed
4 of 10 tasks
Assignees
Labels
Milestone

Comments

@sqweek
Copy link

sqweek commented Aug 23, 2017

Checklist

  • I looked at https://github.com/pbatard/rufus/wiki/FAQ to see if my question has already been answered.
  • I performed a search in the issue tracker for similar issues, using keywords relevant to my problem.
  • I clicked the Log button in Rufus and copy/pasted the log into the line that says <FULL LOG> below.
  • The log I am copying is the FULL log, starting with the line Rufus version: x.y.z - I have NOT removed any part of it.

Additionally (if applicable):

  • I ran a bad blocks check, by clicking the "bad blocks" check box in Rufus, and confirmed that my USB is not defective
  • I also tried one or more of the following:
    • Using a different USB drive
    • Plugging the USB into a different port
    • Running Rufus on a different computer
  • If using an ISO image, I clicked on the # button (at the bottom of the Rufus interface), to compute the MD5, SHA1 and SHA256 checksums, which are therefore present in the log I copied. I confirmed, by performing an internet search, that these values match the ones from the official image.

Issue description

I used Rufus to write a 7.1 GB .iso to USB (an archlinux derivative). The UI is nice and simple so thank you and props for no installation required :). Anyway, I used mostly default/recommended settings which meant FAT32 and the file-based mode rather than direct-disk mode.

Unfortunately it turns out that the ISO was predominantly comprised of a single 6.9 GB .sfs file, which is of course too large to store on a FAT32 partition. Rufus however happily ignored this problem and just copied ~3.5 GB of the .sfs across. There were no errors or warnings (or at least none prominent) -- it looked like the process had succeeded and it wasn't until attempting to boot from the USB that I noticed something was wrong.

I'm rewriting the USB now in direct-disk mode which I expect will solve my problem so no big deal, it would just save some time to catch this scenario earlier :)

Log

I lied -- I didn't save the log of the run on account of I thought it succeeded!

@pbatard
Copy link
Owner

pbatard commented Aug 23, 2017

I'm very surprised by your report. Rufus does detect > 4 GB files residing on an ISO, and will take measures to work with them. As a result, I expect that Rufus switched to using NTFS instead of FAT32 for the file system, but you didn't notice (of course, having a log would tell this very explicitly, and nothing prevented you from trying the same thing twice to get a log).

Now the issue is that while some Linux distros (Debian and IIRC Arch) have no issues with NTFS as the file system used on a bootable USB Flash Drive, there's a minimum level of support that needs to be added to the kernel and the init script for this to work, which your Arch derivative might not follow).

Now, if you could give me a link to the ISO you downloaded I could definitely test and see what's up. So can you please provide that information, especially if you don't have a log to provide. Makes it very difficult to treat your request as a relevant one otherwise, as I must confirm an issue before I can address it. Thank you.

@pbatard pbatard self-assigned this Aug 23, 2017
@sqweek
Copy link
Author

sqweek commented Aug 24, 2017

OK, interesting. The result was definitely a FAT32 usb stick, as my first attempt to fix the problem was to manually copy the full 6.9 GB .sfs file across, which failed because it was too large for the filesystem. I didn't have time to retry the burn last night but I'll give it a shot now to get a log and see if the problem is reproducible...

Yep, it is. The log suggests Rufus has copied the first 4.0 GB of the sfs across, then overwritten it with the last 2.9 GB? xD In case it's something particular to the iso, it's from https://blackarch.org/downloads.html#iso-download (BlackArch Linux 64 bit Live ISO, 2017.06.14).

Rufus version: 2.16.1170
Windows version: Windows 10 64-bit (Build 15063)
Syslinux versions: 4.07/2013-07-25, 6.03/2014-10-06
Grub versions: 0.4.6a, 2.02
System locale ID: 0x0809
Will use default UI locale 0x0809
SetLGP: Successfully set NoDriveTypeAutorun policy to 0x0000009E
Localization set to 'en-US'
Found USB 2.0 device 'USB DISK 2.0 USB Device' (13FE:4200)
1 device found
Disk type: Removable, Sector Size: 512 bytes
Cylinders: 942, TracksPerCylinder: 255, SectorsPerTrack: 63
Partition type: MBR, NB Partitions: 1
Disk ID: 0x28A85F9F
Drive has an unknown Master Boot Record
Partition 2:
  Type: EFI FAT (0xef)
  Size: 64 MB (67108864 bytes)
  Start Sector: 164, Boot: No
Scanning image...
ISO analysis:
  Image is an ISO9660 image
  Will use '/isolinux/isolinux.cfg' for Syslinux
  Detected Syslinux version: 6.03 (from '/isolinux/isolinux.bin')
Disk image analysis:
  Image has an unknown Master Boot Record
  Image is a bootable disk image
ISO label: 'BLACKARCH_201706'
  Size: 7.1 GB (Projected)
  Uses: Syslinux/Isolinux v6.03
  Uses: EFI
Using image: blackarchlinux-live-2017.06.14-x86_64.iso
Will reuse 'ldlinux.sys' and 'ldlinux.bss' from './rufus_files/rufus_files/syslinux-6.03/' for Syslinux installation

Format operation started
Requesting disk access...
Opened \\.\PHYSICALDRIVE1 for exclusive write access
Requesting lock...
Will use 'D:' as volume mountpoint
I/O boundary checks disabled
Requesting lock...
Analyzing existing boot records...
Drive has an unknown Master Boot Record
Volume has an unknown Partition Boot Record
Clearing MBR/PBR/GPT structures...
Erasing 128 sectors
Deleting partitions...
Partitioning (MBR)...
Closing existing volume...
Waiting for logical drive to reappear...
Formatting (FAT32)...
Using cluster size: 4096 bytes
Quick format was selected
Creating file system...
Format completed.
Writing master boot record...
Drive has a Zeroed Master Boot Record
Partition is already FAT32 LBA...
Set bootable USB partition as 0x80
Using Syslinux MBR
Found volume GUID \\?\Volume{8af8524e-144d-11e7-83de-2816ad2a662d}\
Installing Syslinux 6.03...
Opened \\?\Volume{8af8524e-144d-11e7-83de-2816ad2a662d} for shared write access
Using existing './rufus_files/syslinux-6.03/ldlinux.sys' ✓
Using existing './rufus_files/syslinux-6.03/ldlinux.bss' ✓
Successfully wrote 'ldlinux.sys'
Successfully wrote Syslinux boot record
Successfully remounted \\?\Volume{8af8524e-144d-11e7-83de-2816ad2a662d}\ on D:
Copying ISO files...
Extracting files...
Image is an ISO9660 image
This image will be extracted using Joliet extensions (if present)
Extracting: D:\blackarch\boot\intel_ucode.img (964.5 KB)
Extracting: D:\blackarch\boot\intel_ucode.LICENSE (6.8 KB)
Extracting: D:\blackarch\boot\memtest (146.5 KB)
Extracting: D:\blackarch\boot\memtest.COPYING (17.7 KB)
Extracting: D:\blackarch\boot\syslinux\archiso.cfg (209 bytes)
Extracting: D:\blackarch\boot\syslinux\archiso_head.cfg (821 bytes)
Extracting: D:\blackarch\boot\syslinux\archiso_pxe32.cfg (1 KB)
Extracting: D:\blackarch\boot\syslinux\archiso_pxe64.cfg (1.1 KB)
Extracting: D:\blackarch\boot\syslinux\archiso_pxe_32_inc.cfg (118 bytes)
Extracting: D:\blackarch\boot\syslinux\archiso_pxe_both_inc.cfg (158 bytes)
Extracting: D:\blackarch\boot\syslinux\archiso_pxe_choose.cfg (207 bytes)
Extracting: D:\blackarch\boot\syslinux\archiso_sys32.cfg (302 bytes)
  Patched D:\blackarch\boot\syslinux\archiso_sys32.cfg: 'BLACKARCH_201706' ➔ 'BLACKARCH_2'
Extracting: D:\blackarch\boot\syslinux\archiso_sys64.cfg (310 bytes)
  Patched D:\blackarch\boot\syslinux\archiso_sys64.cfg: 'BLACKARCH_201706' ➔ 'BLACKARCH_2'
Extracting: D:\blackarch\boot\syslinux\archiso_sys_32_inc.cfg (118 bytes)
Extracting: D:\blackarch\boot\syslinux\archiso_sys_both_inc.cfg (118 bytes)
Extracting: D:\blackarch\boot\syslinux\archiso_sys_choose.cfg (207 bytes)
Extracting: D:\blackarch\boot\syslinux\archiso_tail.cfg (615 bytes)
Extracting: D:\blackarch\boot\syslinux\cat.c32 (1.6 KB)
Extracting: D:\blackarch\boot\syslinux\chain.c32 (24.2 KB)
Extracting: D:\blackarch\boot\syslinux\cmd.c32 (1.2 KB)
Extracting: D:\blackarch\boot\syslinux\cmenu.c32 (3.6 KB)
Extracting: D:\blackarch\boot\syslinux\config.c32 (1.4 KB)
Extracting: D:\blackarch\boot\syslinux\cptime.c32 (4.1 KB)
Extracting: D:\blackarch\boot\syslinux\cpu.c32 (4.4 KB)
Extracting: D:\blackarch\boot\syslinux\cpuid.c32 (1.7 KB)
Extracting: D:\blackarch\boot\syslinux\cpuidtest.c32 (2.7 KB)
Extracting: D:\blackarch\boot\syslinux\debug.c32 (1.5 KB)
Extracting: D:\blackarch\boot\syslinux\dhcp.c32 (4.0 KB)
Extracting: D:\blackarch\boot\syslinux\disk.c32 (1.9 KB)
Extracting: D:\blackarch\boot\syslinux\dmi.c32 (8.4 KB)
Extracting: D:\blackarch\boot\syslinux\dmitest.c32 (11.7 KB)
Extracting: D:\blackarch\boot\syslinux\elf.c32 (3.2 KB)
Extracting: D:\blackarch\boot\syslinux\ethersel.c32 (2.7 KB)
Extracting: D:\blackarch\boot\syslinux\gfxboot.c32 (10.2 KB)
Extracting: D:\blackarch\boot\syslinux\gpxecmd.c32 (1.6 KB)
Extracting: D:\blackarch\boot\syslinux\hdt\modalias.gz (101.7 KB)
Extracting: D:\blackarch\boot\syslinux\hdt\pciids.gz (243.2 KB)
Extracting: D:\blackarch\boot\syslinux\hdt.c32 (161.1 KB)
Extracting: D:\blackarch\boot\syslinux\hexdump.c32 (3.6 KB)
Extracting: D:\blackarch\boot\syslinux\host.c32 (1.8 KB)
Extracting: D:\blackarch\boot\syslinux\ifcpu.c32 (4 KB)
Extracting: D:\blackarch\boot\syslinux\ifcpu64.c32 (1.7 KB)
Extracting: D:\blackarch\boot\syslinux\ifmemdsk.c32 (4.2 KB)
Extracting: D:\blackarch\boot\syslinux\ifplop.c32 (1.8 KB)
Extracting: D:\blackarch\boot\syslinux\kbdmap.c32 (1.6 KB)
Extracting: D:\blackarch\boot\syslinux\kontron_wdt.c32 (4.8 KB)
Extracting: D:\blackarch\boot\syslinux\ldlinux.c32 (113.6 KB)
Extracting: D:\blackarch\boot\syslinux\lfs.c32 (5.0 KB)
Extracting: D:\blackarch\boot\syslinux\libcom32.c32 (177.7 KB)
Extracting: D:\blackarch\boot\syslinux\libgpl.c32 (65.3 KB)
Extracting: D:\blackarch\boot\syslinux\liblua.c32 (99.5 KB)
Extracting: D:\blackarch\boot\syslinux\libmenu.c32 (23.6 KB)
Extracting: D:\blackarch\boot\syslinux\libutil.c32 (23.1 KB)
Extracting: D:\blackarch\boot\syslinux\linux.c32 (4.6 KB)
Extracting: D:\blackarch\boot\syslinux\lpxelinux.0 (73.2 KB)
Extracting: D:\blackarch\boot\syslinux\ls.c32 (2.9 KB)
Extracting: D:\blackarch\boot\syslinux\lua.c32 (6.7 KB)
Extracting: D:\blackarch\boot\syslinux\mboot.c32 (10.0 KB)
Extracting: D:\blackarch\boot\syslinux\memdisk (25 KB)
Extracting: D:\blackarch\boot\syslinux\meminfo.c32 (2.4 KB)
Extracting: D:\blackarch\boot\syslinux\menu.c32 (25.4 KB)
Extracting: D:\blackarch\boot\syslinux\pci.c32 (3.2 KB)
Extracting: D:\blackarch\boot\syslinux\pcitest.c32 (3.4 KB)
Extracting: D:\blackarch\boot\syslinux\pmload.c32 (2.9 KB)
Extracting: D:\blackarch\boot\syslinux\poweroff.c32 (1.6 KB)
Extracting: D:\blackarch\boot\syslinux\prdhcp.c32 (3 KB)
Extracting: D:\blackarch\boot\syslinux\pwd.c32 (1.4 KB)
Extracting: D:\blackarch\boot\syslinux\pxechn.c32 (12.1 KB)
Extracting: D:\blackarch\boot\syslinux\reboot.c32 (1.3 KB)
Extracting: D:\blackarch\boot\syslinux\rosh.c32 (13.5 KB)
Extracting: D:\blackarch\boot\syslinux\sanboot.c32 (1.6 KB)
Extracting: D:\blackarch\boot\syslinux\sdi.c32 (3.1 KB)
Extracting: D:\blackarch\boot\syslinux\splash.png (22.6 KB)
Extracting: D:\blackarch\boot\syslinux\sysdump.c32 (15.1 KB)
Extracting: D:\blackarch\boot\syslinux\syslinux.c32 (7.3 KB)
Extracting: D:\blackarch\boot\syslinux\syslinux.cfg (74 bytes)
Extracting: D:\blackarch\boot\syslinux\vesa.c32 (3 KB)
Extracting: D:\blackarch\boot\syslinux\vesainfo.c32 (2.1 KB)
Extracting: D:\blackarch\boot\syslinux\vesamenu.c32 (26.0 KB)
Extracting: D:\blackarch\boot\syslinux\vpdtest.c32 (1.8 KB)
Extracting: D:\blackarch\boot\syslinux\whichsys.c32 (2.4 KB)
Extracting: D:\blackarch\boot\syslinux\zzjson.c32 (3.5 KB)
Extracting: D:\blackarch\boot\x86_64\archiso.img (37.6 MB)
Extracting: D:\blackarch\boot\x86_64\vmlinuz (4.8 MB)
Extracting: D:\blackarch\pkglist.x86_64.txt (110.1 KB)
Extracting: D:\blackarch\x86_64\airootfs.md5 (47 bytes)
Extracting: D:\blackarch\x86_64\airootfs.sfs (4.0 GB)
Extracting: D:\blackarch\x86_64\airootfs.sfs (2.9 GB)
Extracting: D:\EFI\archiso\efiboot.img (64 MB)
Extracting: D:\EFI\boot\bootx64.efi (94.3 KB)
Extracting: D:\EFI\boot\HashTool.efi (92.8 KB)
Extracting: D:\EFI\boot\loader.efi (78.3 KB)
Extracting: D:\EFI\shellx64_v1.efi (753.1 KB)
Extracting: D:\EFI\shellx64_v2.efi (900.7 KB)
Extracting: D:\isolinux\boot.cat (2 KB)
Extracting: D:\isolinux\isohdpfx.bin (432 bytes)
Extracting: D:\isolinux\isolinux.bin (38 KB)
Extracting: D:\isolinux\isolinux.cfg (135 bytes)
Extracting: D:\isolinux\ldlinux.c32 (113.6 KB)
Extracting: D:\loader\entries\archiso-x86_64.conf (193 bytes)
Extracting: D:\loader\entries\uefi-shell-v1-x86_64.conf (56 bytes)
Extracting: D:\loader\entries\uefi-shell-v2-x86_64.conf (56 bytes)
Extracting: D:\loader\loader.conf (33 bytes)
Created: D:\syslinux.cfg
Finalizing, please wait...
Created: D:autorun.inf
Created: D:autorun.ico

Found USB 2.0 device 'USB DISK 2.0 USB Device' (13FE:4200)
Using autorun.inf label for drive D: 'BLACKARCH_201706'
1 device found
Disk type: Removable, Sector Size: 512 bytes
Cylinders: 942, TracksPerCylinder: 255, SectorsPerTrack: 63
Partition type: MBR, NB Partitions: 1
Disk ID: 0x035ED7C1
Drive has a Syslinux Master Boot Record
Partition 1:
  Type: FAT32 LBA (0x0c)
  Size: 7.2 GB (7747174400 bytes)
  Start Sector: 2048, Boot: Yes
Found USB 2.0 device 'USB DISK 2.0 USB Device' (13FE:4200)
Using autorun.inf label for drive D: 'BLACKARCH_201706'
1 device found
Disk type: Removable, Sector Size: 512 bytes
Cylinders: 942, TracksPerCylinder: 255, SectorsPerTrack: 63
Partition type: MBR, NB Partitions: 1
Disk ID: 0x035ED7C1
Drive has a Syslinux Master Boot Record
Partition 1:
  Type: FAT32 LBA (0x0c)
  Size: 7.2 GB (7747174400 bytes)
  Start Sector: 2048, Boot: Yes
Found USB 2.0 device 'USB DISK 2.0 USB Device' (13FE:4200)
Using autorun.inf label for drive D: 'BLACKARCH_201706'
1 device found
Disk type: Removable, Sector Size: 512 bytes
Cylinders: 942, TracksPerCylinder: 255, SectorsPerTrack: 63
Partition type: MBR, NB Partitions: 1
Disk ID: 0x035ED7C1
Drive has a Syslinux Master Boot Record
Partition 1:
  Type: FAT32 LBA (0x0c)
  Size: 7.2 GB (7747174400 bytes)
  Start Sector: 2048, Boot: Yes

@pbatard
Copy link
Owner

pbatard commented Aug 24, 2017

Thanks for the log. This looks weird, and I'm thinking the issue doesn't come from Rufus, but from the way the ISO was mastered, as the BlackArch people seemed to have split the SFS in 2 parts that use the same name. As such I'm think that ISO9660 file system shenanigans are to blame rather than Rufus. Of course I'm currently downloading the ISO to have a closer look at its content, but, from having worked and tested ISOs that do contain > 4GB file, the behaviour being displayed here seems just too weird not to come from ISO-9660 filesystem abuse.

As I was mentioning, Rufus does detect > 4GB files when processing an ISO, and should have displayed an Has a >4GB file notice like:

ISO label: 'SSS_X64FRE_EN-US_DV9'
  Size: 5.3 GB (Projected)
  Has a >4GB file
  Uses: EFI
  Uses: Bootmgr
  Uses: Install.wim (version 0.13.1)
Using image: en_windows_server_2016_x64_dvd_9327751.iso

The fact that it didn't, as well as the following two lines from the log tells me that the people who created the ISO used some kind of trickery to split the file in 2 while using the same name for the 2 parts:

Extracting: D:\blackarch\x86_64\airootfs.sfs (4.0 GB)
Extracting: D:\blackarch\x86_64\airootfs.sfs (2.9 GB)

Anyway, I'll know more after I have a closer look at your ISO.

@sqweek
Copy link
Author

sqweek commented Aug 24, 2017

Gotcha. FWIW opening the ISO in 7zip shows a single file of 7,445,232,000 bytes:

rufus_4gb_sfs

The properties pane of airootfs.sfs suggests that something like you describe may be happening at the ISO9660 level though (I note "Files: 2", although that may be talking about the folder...):

7z_airootfs_props

@pbatard
Copy link
Owner

pbatard commented Aug 24, 2017

I got the ISO now, and can confirm that, as far as 7-zip is concerned, airootfs.sfs is a single file of ~7 GB, but libcdio (the library that Rufus uses for ISO processing) sees it as two separate files:

/d/libcdio/example # ./isolist /d/ISOs/blackarchlinux-live-2017.06.14-x86_64.iso
Application: ARCH LINUX LIVE/RESCUE CD
Preparer   : PREPARED BY MKARCHISO
Publisher  : ARCH LINUX <HTTP://WWW.ARCHLINUX.ORG>
Volume     : BLACKARCH_201706
d [LSN     30]     2048 /.
d [LSN     21]     2048 /..
- [LSN  55927]       47 /airootfs.md5
- [LSN  55928] 4294965248 /airootfs.sfs
- [LSN 2153079] 3160266752 /airootfs.sfs

Looks like this could be a libcdio bug. I'll get in touch with the libcdio mailing list to find out what they think the issue is, and hopefully get a fix, that I can then integrate in Rufus. It may be a little while before I am able to do so though...

@pbatard pbatard added this to the 2.18 milestone Sep 8, 2017
@pbatard
Copy link
Owner

pbatard commented Sep 11, 2017

While I'm still waiting for libcdio to fix this issue (though, by the looks of it, I may have to submit a fix myself), I have now added a warning about multi-extent ISO-9660 files in Rufus 2.17.

pbatard added a commit that referenced this issue Sep 11, 2017
* This is related to issue #1007, which libcdio still needs to fix.
@lock
Copy link

lock bot commented Apr 6, 2019

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue if you think you have a related problem or query.

@lock lock bot locked and limited conversation to collaborators Apr 6, 2019
# for free to subscribe to this conversation on GitHub. Already have an account? #.
Labels
Projects
None yet
Development

No branches or pull requests

2 participants