Why and how does Rufus use FAT32 for a bootable Linux ISO?

Question

I never paid attention to this detail, and always thought that Rufus writes "the original ISO image" to the destination disk without changing anything.

But today I noticed that when writing an Ubuntu ISO to a USB flash drive, it uses FAT or FAT32 by default:

How is that possible?

I am pretty sure that ubuntu-20.04-desktop-amd64.iso contains ext4 partitions and no FAT32 partition.

Answer 1

CDs and DVDs use ISO-9660 or UDF, not FAT32 or ext4, otherwise most non-Linux platforms can't read them. But modern Linux installation discs are also isohybrid so they can also boot directly if the image was written into a normal non-optical disk. That's the dd method and most other Linux distros suggest to use dd mode to create a bootable USB installation drive. There are also some other methods such as copy the whole ISO file into the data partition and map it then use the boot-from-mapped-ISO feature of Grub, or copy all the files of the installer into the FAT32 partition and use syslinux to boot

Rufus also supports the dd mode but its author already states why it doesn't use that mode by default in the FAQ in its Github repo. The reason is very long, see the full in the above link or below. But in summary it's simpler for the average users, is less astonishing and users can use the pen drive for data normally. With the dd method users would need to delete the partition and reinitialize the disk whereas this method only requires a simple format or deletion of the files

Rufus' author also has an excellent answer on SuperUser that I just saw from Kamil Maciorowski's comment after posting this answer

Here's the full reason on Github:

Why doesn't Rufus recommend DD mode over ISO mode for ISOHybrid images? Surely DD is better!

Congratulations. If you are coming to this FAQ with the idea that DD mode has no drawbacks, then you have drunk the ISOhybrid kool aid, which has been a massive plague for people who are effectively trying to ensure that users can actually create a bootable drive in the best possible condition, without being constrained to the shortcomings of a "one method to rule them all" fallacy.

And here, I hear you protesting: "But dd is a lot faster than copying individual files, and it enables the use of a native Linux file system as well as an ESP, plus it makes sure that the resulting drive is a bit for bit copy of the one created by the person who produced the ISO. How could this not objectively be the better option???"

Well, unfortunately for you, it is very easy to disprove that dd mode is the better option for users (and that's not even counting the similar reports I get through e-mail). The fact that Windows cannot natively mount the usual Linux partition that follows the ESP is EXCEEDINGLY CONFUSING FOR MANY USERS. So, writing an ISOHybrid in dd mode will usually break the principle of least astonishment, which Linux maintainers, who are less tuned to hearing reports from Windows users, tend to disregard as a non issue, when it most certainly is an issue.

Furthermore, whereas pretty much any OS provides native tools to easily create a FAT32 partition onto a USB drive, and extract ISO content onto it (which, if you have a UEFI system, should be more than enough to create a bootable drive, provided that the image creators did their job properly), so that you shouldn't even have to use a utility like Rufus, using dd for copying an image requires a little more involvement and, if you are using it manually, can lead to dramatic mishaps (dd is not also called Disk Destroyer by mistake), which are a lot less likely to happen when using file extraction mode.

There is also the problem of restricting the media you create to UEFI boot only, which can be very desireable to do (to prevent installing your OS in BIOS/Legacy/CSM mode when you wanted to install it in UEFI mode), but that can only be accomplished if you can set the partition scheme to GPT... whereas most ISOHybrid media is designed to boot from either BIOS or UEFI and therefore uses MBR.

Oh, and you can of course forget about adding any extra content (such as, say, proprietary Wifi firmware binaries, which you may need to load in order for your platform to have connectivity during installation) or using a bootable drive for data on Windows, if it was written in DD mode. For instance, one can't simply use DD mode to install a generic Linux distribution on the Raspberry Pi, whereas, when that same distribution supports ISO mode, one can just take the vanilla ISO, add the handful of extra files that are required for Pi boot (which of course would be impossible to accomplish in DD mode) and install that OS in the same manner as you would do on a PC.

Finally, when using GPT as a partition scheme, using dd to write an ISOhybrid image will imediately result in a "broken" drive, on account that the backup GPT table will not be written where it should (in the very last 33 sectors of the drive) unless you use a drive that is the exact same size as the image, which is never ever the case. This means that, should a UEFI firmware be pedantic (and some are!), it may very well choose not to boot the drive altogether, on account that the GPT is broken. So much for DD mode being a panacea!

The above often results in a first time experience, for Windows users who are trying to try or transition to Linux, that can be very subpar and it is very unfortunate, though not entirely surprising, that a lot of Linux maintainers have so far been turning a deaf ear to the plight of said users, by disimissing these issues as something unfamiliar users should just "plow through", on account of treating ISOHybrids as a mere DD image (which is what the Manjaro and PopOS maintainers currently do) making their own lives so much easier...

Still, because we do believe that Windows users should have the best experience when creating a bootable drive, and not be confronted with something very unexpected that will leave them, at best, inconvenienced, or, at worst, believing that their drive is "broken", where possible, Rufus will continue to recommend ISO mode over DD mode (while obviously still giving the choice, for users who wish to do so, to write their ISOHybrid in DD mode).

Answer 2

There are many things to unpack here. First, how does a Live Linux boot anyway? Unlike a regular installed-to-disk Linux, it does not use an ext4 (or similar) file system on a disk partition. Instead, the root filesystem (containing directories like /bin, /usr etc) is stored in an compressed image file (Squashfs). Squashfs supports all the features required for a Linux root fs like permissions, ACLs, hard links etc. At runtime, this read-only image is made to appear writable by overlaying a tmpfs RAM filesystem.

By storing the root filesystem in an image format, it can then be placed on any old filesystem—including FAT32, ISO 9660 (CDs) and UDF (DVDs).

For whatever reason, Linux distributions still create CD/DVD images for all their installation media. So they use an ISO 9660 filesystem (with Joliet and sometimes Rock Ridge extensions). Because of some peculiarities, only certain filesystems are practical on optical media.

The Live Linux has to boot. Bootable CDs/DVDs have to use the “El Torito” standard—they boot in a different way from regular mass storage like USB drives or internal HDDs/SSDs. So the Linux images have to follow this standard. It has since been adapted to also support UEFI boot.

However, you can also write (dd) the image to a USB flash drive and it works there too. This is accomplished by smuggling bootcode and a MBR or GPT partition table into gaps that ISO 9660 does not use. And then, by tacking on an EFI System Partition (located at the end of the image, after the ISO 9660 filesystem), this monstrosity is also UEFI-bootable.

The problem with all this? The flash drive is mostly unusable until wiped. Rufus can (optionally) solve this by copying the contents of the EFI System Partition and the contents of the ISO 9660 filesystem to a single FAT32 filesystem that is then bootable for both BIOS and UEFI as well as usable by Windows.