"inflate error" when writing a compressed image larger than 4GB

[chocbic172]

Checklist

• I looked at https://github.com/pbatard/rufus/wiki/FAQ to see if my question has already been answered.
• I performed a search in the issue tracker for similar issues using keywords relevant to my problem, such as the error message I got from the log.
• I clicked the 'Log' button (🗒️) or pressed Ctrl-L in Rufus, or used DebugView, and copy/pasted the log into the section that says <FULL LOG> below.
• The log I am copying is the FULL log, starting with the line Rufus version: x.y.z - I have NOT removed any part of it.

Additionally (if applicable):

• I ran a bad blocks check, by clicking Show advanced format options then Check device for bad blocks, and confirmed that my USB is not defective.
• I also tried one or more of the following:
  • Using a different USB drive.
  • Plugging the USB into a different port.
  • Running Rufus on a different computer.
• If using an image, I clicked on the (✓) button to compute the MD5, SHA1 and SHA256 checksums, which are therefore present in the log I copied. I confirmed, by performing an internet search, that these values match the ones from the official image.

Issue description

When attempting to flash a .zip compressed disk image to a USB (tested on SanDisk Extreme Pro and SanDisk UltraFit devices, both >=64 GB in size), there is generally no issue. However, when the compressed image exceeds a size of 4 GB, a warning dialogue titled "Error: Write error." appears about 60 seconds into the write, accompanied by "Error: inflate error" in the logs. The disk image is a custom OS image and was compressed using the "zip" utility on Ubuntu 22.04.

Extracting this .zip archive with Windows Explorer or WinRar and flashing the .img file contained within (which is about 11GB) does not present any errors and allows the USB to be successfully flashed. Using an alternative USB flashing software succeeds to flash the same .zip file to the USB.

Log

Rufus x64 v4.1.2045
Windows version: Windows 10 Pro x64 (Build 19044.3086)
Syslinux versions: 4.07/2013-07-25, 6.04/pre1
Grub versions: 0.4.6a, 2.06
System locale ID: 0x0809 (en-GB)
Will use default UI locale 0x0809
SetLGP: Successfully set NoDriveTypeAutorun policy to 0x0000009E
Localization set to 'en-US'
Notice: The ISO download feature has been deactivated because 'Check for updates' is disabled in your settings.
Found USB 3.0 device 'SanDisk Extreme Pro USB Device' (0781:5588)
1 device found
Disk type: Removable, Disk size: 128 GB, Sector size: 512 bytes
Cylinders: 15567, Tracks per cylinder: 255, Sectors per track: 63
Partition type: GPT, NB Partitions: 4
Disk GUID: {B792CFC6-3FA2-4860-A9FC-025EFA0D31E4}
Max parts: 128, Start Offset: 17408, Usable = 128043678208 bytes
Partition 1:
  Type: EFI System Partition
  Name: 'Partition 1'
  Detected File System: FAT32
  ID: {199EF6B7-3B12-4746-AC30-667EC0805238}
  Size: 4 GB (4294967296 bytes)
  Start Sector: 2048, Attributes: 0x0000000000000000
Partition 2:
  Type: EFI System Partition
  Name: 'Partition 2'
  Detected File System: FAT32
  ID: {49E44F7D-0C7C-4D58-9790-9984A5C19F10}
  Size: 128 MB (134217728 bytes)
  Start Sector: 8390656, Attributes: 0x0000000000000000
Partition 3:
  Type: Microsoft Basic Data Partition
  Name: 'Partition 3'
  Detected File System: FAT32
  ID: {00A0E0C2-8A8C-4096-B15E-3571E50CDDB0}
  Size: 512 MB (536870912 bytes)
  Start Sector: 8652800, Attributes: 0x0000000000000000
Partition 4:
  Type: Linux Data Partition
  Name: 'Partition 4'
  Detected File System: ext4
  ID: {1F4C16F6-853A-4837-98E2-25D23EF3DBB2}
  Size: 6 GB (6442450944 bytes)
  Start Sector: 9701376, Attributes: 0x0000000000000000
Scanning image...
ISO analysis:
  'C:\Users\ethan\Downloads\example-disk-image-release.zip' doesn't look like an ISO image
Disk image analysis:
  Image is a compressed bootable disk image
Using image: example-disk-image-release.zip (4 GB)
Search for conflicting processes was interrupted due to timeout

Format operation started
Requesting disk access...
No drive letter was assigned...
Will use 'D:' as volume mountpoint
Opened \\.\PhysicalDrive1 for exclusive write access
Requesting logical volume handle...
Writing compressed image:
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Error: corrupted data

Error: inflate error

Could not write compressed image: -1
Opened \\.\PhysicalDrive1 for shared write access
Re-mounted volume as D: after error

Found USB 3.0 device 'SanDisk Extreme Pro USB Device' (0781:5588)
1 device found
Disk type: Removable, Disk size: 128 GB, Sector size: 512 bytes
Cylinders: 15567, Tracks per cylinder: 255, Sectors per track: 63
Partition type: GPT, NB Partitions: 4
Disk GUID: {9CBC7FCA-AEF5-4B9B-84F1-3A9DD55A70D8}
Max parts: 128, Start Offset: 17408, Usable = 128043678208 bytes
Partition 1:
  Type: EFI System Partition
  Name: 'Partition 1'
  Detected File System: FAT32
  ID: {598343B7-AC7B-45BB-934B-E581A436EDFC}
  Size: 4 GB (4294967296 bytes)
  Start Sector: 2048, Attributes: 0x0000000000000000
Partition 2:
  Type: EFI System Partition
  Name: 'Partition 2'
  Detected File System: FAT32
  ID: {1669E821-E7C4-469F-A11C-8ADCA22AE927}
  Size: 128 MB (134217728 bytes)
  Start Sector: 8390656, Attributes: 0x0000000000000000
Partition 3:
  Type: Microsoft Basic Data Partition
  Name: 'Partition 3'
  Detected File System: FAT32
  ID: {28836096-D108-4A82-B08E-A0D80D465301}
  Size: 512 MB (536870912 bytes)
  Start Sector: 8652800, Attributes: 0x0000000000000000
Partition 4:
  Type: Linux Data Partition
  Name: 'Partition 4'
  Detected File System: ext4
  ID: {B1883C25-677E-4028-B9A2-296D9C4B5026}
  Size: 6 GB (6442450944 bytes)
  Start Sector: 9701376, Attributes: 0x0000000000000000
Found USB 3.0 device 'SanDisk Extreme Pro USB Device' (0781:5588)
1 device found
Disk type: Removable, Disk size: 128 GB, Sector size: 512 bytes
Cylinders: 15567, Tracks per cylinder: 255, Sectors per track: 63
Partition type: GPT, NB Partitions: 4
Disk GUID: {9CBC7FCA-AEF5-4B9B-84F1-3A9DD55A70D8}
Max parts: 128, Start Offset: 17408, Usable = 128043678208 bytes
Partition 1:
  Type: EFI System Partition
  Name: 'Partition 1'
  Detected File System: FAT32
  ID: {598343B7-AC7B-45BB-934B-E581A436EDFC}
  Size: 4 GB (4294967296 bytes)
  Start Sector: 2048, Attributes: 0x0000000000000000
Partition 2:
  Type: EFI System Partition
  Name: 'Partition 2'
  Detected File System: FAT32
  ID: {1669E821-E7C4-469F-A11C-8ADCA22AE927}
  Size: 128 MB (134217728 bytes)
  Start Sector: 8390656, Attributes: 0x0000000000000000
Partition 3:
  Type: Microsoft Basic Data Partition
  Name: 'Partition 3'
  Detected File System: FAT32
  ID: {28836096-D108-4A82-B08E-A0D80D465301}
  Size: 512 MB (536870912 bytes)
  Start Sector: 8652800, Attributes: 0x0000000000000000
Partition 4:
  Type: Linux Data Partition
  Name: 'Partition 4'
  Detected File System: ext4
  ID: {B1883C25-677E-4028-B9A2-296D9C4B5026}
  Size: 6 GB (6442450944 bytes)
  Start Sector: 9701376, Attributes: 0x0000000000000000

[JonnyTech]

Do you have enough space on your local drive (specifically the temp folders) to store the extracted image from within the zip archive?

[pbatard]

@JonnyTech, this shouldn't matter, as Rufus does not need to extract to a temporary file.

I'll try to validate this when I get a chance, but it looks like some of the decompression code we have, which we picked from BusyBox because we wanted something lightweight, might not be 64-bit compliant...

[pbatard]

This looks like a BusyBox bug that was fixed in a more recent version of the code than the one we use for Rufus (which comes from Bled which is derived from BusyBox).

Especially, the symptoms describe by OP seem to match

19 August 2021 -- BusyBox 1.34.0 (unstable)
(...)
unzip: fix for .zip archives with >4GB file
(...)

from BusyBox's ChangeLog.

I'll try to update Bled with the latest BusyBox decompression code when I have some time, so that I can then update Rufus.

[pbatard]

Actually, the issue is more complex than that.

The fix mentioned above is for decompressed files that are larger than 4 GB, and we already integrated it last time we updated Bled (for instance you can download the ~1.5 GB [ChromeOS Flex installer image that contains a 8 GB DD image, and you'll find that Rufus has no trouble with it). And unfortunately that fix does nothing for archives that are > 4GB themselves, i.e. our actual issue here.

And it's not as simple as adding support for ZIP64 data (basically the traditional ZIP header is limited to 32-bit for compressed and uncompressed file sizes, but you can add extra ZIP64 data after it to define the 64-bit sizes), because I've done just that and the decompressing still fails.

So it looks like the problem is in the gunzip decompression, that is being called behind the scenes from unzip, and since we're working with bite-size decompression windows there, I'm not seeing an obvious "This is a 32-bit integer variable that should converted to 64-bit". In short, this will take a bit more time to figure out, and it is far from an easy fix.

[pbatard]

This should be fixed in the next version of Rufus.

I basically had to add full blown ZIP64 support to the BusyBox code, since BusyBox doesn't support it, as well as improve CDF/CDE handling. This was a major undertaking, as evidenced in the commit that closes this issue below.

But at least, people should now be able to use Rufus with .zip files that are larger than 4 GB...

[chocbic172]

Thanks for getting back and doing this so quickly, much appreciated.

[github-actions]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue if you think you have a related problem or query.