79

After failing to copy a file bigger than 4G to my 8G USB flash drive, I formatted it as ext3. While this is working fine for me so far, it will cause problems if I want to use it to copy files to someone which does not use Linux.

I am thinking of formatting it as UDF instead, which I hope would allow it to be read (and possibly even written) on the three most popular operating systems (Windows, MacOS, and Linux), without having to install any extra drivers. However, from what I found on the web already, there seem to be several small gotchas related to which parameters are used to create the filesystem, which can reduce the compability (but most of the pages I found are about optical media, not USB flash drives).

I would like to know:

  • Which utility should I use to create the filesystem? (So far I have found mkudffs and genisoimage, and mkudffs seems the best option.)
  • Which parameters should I use with the chosen utility for maximum compability?
  • How compatible with the most common versions of these three operating systems UDF actually is?
  • Is using UDF actually the best idea? Is there another filesystem which would have better compatibility, with no problematic restrictions like the FAT32 4G file size limit, and without having to install special drivers in every single computer which touches it?
1

7 Answers 7

57

First, I zeroed completely the drive before creating the UDF filesystem with:

dd if=/dev/zero of=/dev/sdx bs=512

This is to avoid any leftover superblocks or other metadata which could confuse the operating systems' filesystem type detection (at least zeroing the first sector should be needed, to obliterate the partition table; the first few sectors are not used by UDF, and a leftover partition table could really confuse things). You could also use the count=1 switch on the dd command, in order to more-quickly zero just the first 512 bytes of the drive (where the MBR usually is located within), though this was not tested.

To create the file system, the command I used was:

mkudffs --media-type=hd --blocksize=512 /dev/sdx

mkudffs command will become available on Debian-based Linux distros (such as Ubuntu) after installing a udftools package:

sudo apt-get install udftools

The default blocksize for mkudffs is 2048, which is wrong for a USB flash drive (which uses 512-byte sectors). Since the block size is used to find the filesystem metadata, using a wrong block size can make it not be recognized as a UDF filesystem (since the anchor will not be where the filesystem driver is expecting). Note that the mkudffs man page is wrong; 512 is a valid value for the block size (and the code explicitly accepts it).

I also used the whole drive instead of a partition; this should be more compatible.

The result of my testing so far:

  • Linux with the most recent kernel (2.6.31, from Ubuntu 9.10): works.
  • Linux with an older kernel: needs the bs=512 option to mount, because it incorrectly used 2048 instead of the device sector size (fixed in commit 1197e4d).
  • Windows Vista: works.
  • A brand-new Mac: works.
  • Windows XP: can read fine, but gives "access denied" when trying to write; also seems to think the disk is full.

While I have not so far tried to create a file larger than 4G in it, I see no reason why it would not work.

Given that it worked perfectly on all recent operating systems (only having to mount manually on Linux, which will not be needed anymore as soon as Ubuntu 9.10 and Fedora 12 are out), and worked read-only in Windows XP (which was a surprise to me; I was expecting it to not recognize the filesystem at all), using UDF instead of FAT32 or NTFS in big USB flash drives seems a good idea.

14
  • 1
    how did you format/partition the drive? I have zeroed my 32GB USB drive, using dd if=/dev/zero of=/dev/sdb bs=1M, which leaves it without any partition table.
    – romeovs
    Apr 6, 2012 at 11:00
  • 4
    @romeovs: I did not partition the drive. That is the whole point of zeroing it, to remove the partition table and old filesystem leftovers and put the UDF filesystem in the whole unpartitioned drive. Creating a partition table would only risk confusing things.
    – CesarB
    May 1, 2012 at 16:52
  • 3
    Didn't work for me with a 500 GB SeaGate FreeAgent Desktop USB 2.0 external hard drive (not sure if it should since this thread is about Flash drives). Windows 7 always showed the drive as "unallocated" in the Disk Management utility. I tried several different option combinations. Sep 17, 2012 at 19:11
  • 2
    Thanks a lot for the follow up. It's rare, ifnot the first time, that I see a smart question which is then elaborately answered by the OP after a few days/weeks of testing. Great job, thanks!
    – Luc
    Nov 22, 2013 at 22:57
  • 2
    Most USB sticks expose their memory as 512 byte blocks. Whereas internally things are much more complicated (different size, depending if you speak about write blocks (generally a few kb), erase blocks (lots of kb) or allocation units (a few mb)). On some system, UDF needs that block size == sector size. So "-b 512"
    – DrYak
    Mar 23, 2016 at 16:40
7

CesarB did a great job getting to the crux of the issue. One thing that can't be underscored enough is how important it is to use the proper block size when formatting UDF.

Inspired by CesarB's post (and my other research/testing), I wrote a script to automate the process of formatting in UDF--using the properly detected sector size. See format-udf on GitHub. Notable features:

  • Formats a block drive (hard drive or Flash drive) in Universal Disk Format (UDF)
    • UDF revision 2.01 used for maximal compatibility
    • First 4096 sectors are zeroed out to erase any existing MBR (necessary for proper UDF detection)
  • Resulting file system can be read/written across multiple operating system families (Windows, OS X, and Linux)
  • Runs on any OS having a Bash environment

Because of the last point, this script I wrote cannot be used on Windows. However, the script will run on OS X and Linux. After doing so, Windows should be able to magically detect the newly formatted UDF drive.

To directly answer the questions posted, format-udf will:

  • choose the appropriate tool for formatting based on operating system and environment
  • automatically detect and populate all parameters necessary for formatting
  • maximize OS compatibility (see GitHub page for compatibility chart)
  • yield the maximum feature set (and minimal limitations) that the asker is looking for
4
  • 1
    I took a look at your format-udf utility on Github and I have one question about it. The script detects the physical block size of the drive. Are you sure the parameter Linux calls "physical sector (block) size" and not "logical sector size" is the correct one to use? Physical and logical can mean a lot of things. What hdparm calls "Logical sector size" is the unit of addressing used by the SATA protocol, whereas "Physical sector size" is a drive internal thing. To me it makes more sense that the "physical blocks" in the UDF spec really means Linux's "logical blocks". May 24, 2017 at 19:09
  • you're spot-on, @JohanMyréen. i invite you to join the discussion on this very topic on GitHub. github.com/JElchison/format-udf/issues/13 There is an imminent change along the lines of your question, pending additional testing on Windows 7 and 10.
    – j0nam1el
    May 25, 2017 at 21:07
  • 1
    format-udf is really nice. Just tested it on Linux, and could read/write the formatted drive on OS X and on Windows 10.
    – mivk
    Jun 26, 2017 at 11:03
  • While the ability to "run on any OS with bash" has some appeal it would be even better to implement the same approach in mkudffs directly; I mean adding some new --best-block-size option to mkudffs.
    – MarcH
    Aug 13, 2017 at 12:41
3

I seem to recall having done that, the problem I found is that the linux version I had mounted it read only, as the driver had not been built for r/w. It did work in windows, and I think mac.

Yeah, a good solution is hard to find. For a while I had an external drive with a fat32 partition that had drivers for win and mac, a mac partition, and a big ext3 partition. It worked, but it meant installing drivers. Neat trick was it was also bootable on a mac (fw&usb), you have to leave space and take some notes, then you can add partitions via the command line and a mac partition table as well.

The world needs a free, usable by everything, file system. ZFS would be a nice choice. :-)

3
  • ZFS would be nice to have, but would confuse a lot of people. It is also a bit overkill for external media, don't you think? More suited for giant file servers, from what I can tell. Sep 12, 2009 at 2:52
  • 2
    Well, ZFS does check summing and error recovery, which makes a lot of sense for flaky consumer stuff. :-D We are all storing enough these days that bit rot will eat something eventually (see the study Sun did on the issue that lead to ZFS, also note the undetected error rate for the encoding on HD). We need a real fs that everything can use, and fatX isn't it. Not here yet, really. Sep 23, 2009 at 23:31
  • ZFS does not work in Linux
    – ignis
    Nov 9, 2012 at 15:49
2

Pieter Wuille wrote a tool to partition and format a disk to make a UDF layout that will be compatible with both Windows (>= Vista, read only for XP), MacOS X 10.5, Linux 2.6.30+:

Alternatively:

0
1

To achieve maximal compatibility you should use mkudffs from udftools project at least in version 2.0. No special parameters are needed, everything is autodetected.

There are 3 big restrictions:

  1. Microsoft Windows systems do not recognize non-removable hard disk if it does not have MBR or GPT partition table.

  2. Apple Mac OS X systems do not recognize UDF filesystem on partitioned disk.

  3. Probably all systems (except recent Linux kernels) do not recognize UDF filesystem if UDF block size does not match logical sector size of disk.

Tool mkudffs since version 2.0 handle all 3 restrictions. When formatting non-removable hard disks, it creates "fake" MBR table which starts at sector 0 and spans whole disk. So UDF filesystem can be read either from first partition (needed for Microsoft Windows) or from whole disk (needed for Apple Mac OS X). See mkudffs 2.0 man page for more details.

-2

NTFS, with NTFS-3G you can write to it using Linux and take a look at http://macntfs-3g.blogspot.com/ for your Mac.

3
  • 4
    This seems to involve extra drivers for Mac. Sep 12, 2009 at 1:53
  • If he doesn't want to use FAT he is stuck and NTFS-3G works on Mac already he needs what I linked for write.
    – user10547
    Sep 12, 2009 at 2:22
  • 5
    One slight problem is that it is not my Mac. I would rather not have to install a driver in other people's computers.
    – CesarB
    Sep 26, 2009 at 13:35
-2

There are drivers for Windows (and Mac) that can access EXT3 partitions, so you can format it to EXT3 and use it (with drivers) to everywhere. Another way would be to use an archiever to store the large file in two or more files up to 4GB each. This way you can use the FAT32 filesystem which is universal. On the host computer you have to extract the archive in order to use it, but it's a way to do it without drivers. Use a RAR format archiver since it works on Windows, Linux, Mac, although I think a ZIP format could work as well. But I would go with the drivers. Once installed you can do anything without restrictions. In the PC I have used Ext2Fsd for full EXT2, EXT3 and EXT4 access and Macdrive for full MacOS format access. Surely similar tools exists for Linux and MacOS as well to fully access NTFS partitions etc. If you only need read access, you don't need any drivers, Linux and MacOS support reading of NTFS partitions, so format the USB as NTFS! If all these computers are at the same network, things are easier! Make the USB either format and share it on the network. Other computers should not have problem accessing it!

1
  • 2
    -1 because the answer doesn't even mention UDF!
    – dolmen
    Jun 2, 2014 at 16:04

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .