17

This question already has an answer here:

I recently bought a SanDisk 128GB USB flash drive.

And after formatting the USB flash drive in exFAT format, I copied a folder whose capacity is around 10GB. There are lots of small files in it, so it took some time.

However, when I see in the Windows Explorer after copying the folder, it says that around 43GB of the storage is occupied and now only 70GB of the storage is free to use.

What is happening and how should I deal with it? Is my USB flash drive physically broken?

It is still weird because when I copied a single file with 7 GB capacity, it showed the remaining capacity correctly at around 110 GB available..

marked as duplicate by phuclv, JakeGould, Jan Doggen, bertieb, fixer1234 Oct 30 '18 at 2:07

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • 7
    If you right click on a small file and go to properties what does it display for "size" and "size on disk" – Scott Chamberlain Oct 28 '18 at 20:24
  • 4
  • 20
    You said a 10GB file in the title but actually copied a 10GB folder of small files. They're completely different. If your cluster size is 4KB and your files are 1KB on average then obviously it'll take 40GB on disk. By default the allocation size of exFAT is much higher than other file systems – phuclv Oct 29 '18 at 1:58
  • Why did you do that? Do you have to use the USB stick with something other than laptops, desktops and similar hardware? AFAIK only some car radios & such do not support NTFS or a similar alternative... – Bakuriu Oct 29 '18 at 21:45
  • I don't think folders have a "capacity" (some maximum they can hold) expect maybe number of files. What do you mean? – jpmc26 Oct 30 '18 at 2:03
58

You already answered your own question: There are lots of small files in it

Every file on an exFAT volume takes at least one blocksize. So a file of a single byte in size takes at least 4K - a size amplification of 1:4096. You are seing a size amplification of 4.3, which is very plausible with lots of small files.

You can check this hypothesis by packing the files with WinRAR and the zero compression settings, then copy this file to the USB stick.

  • 14
    It means exactly what it means. Disk space is allocated in increments of 4kb, approx. A one byte file takes up 4kb of disk space. A two byte file takes up the same 4kb of disk space. Ditto for 3 bytes, and up to 4096 bytes. A 4097 byte file takes up 8192 bytes of disk space, and so on (this is ignoring the overhead of creating directory entries). The average size of your files seems to be about 1kb, so you end up using up four times as much as the sum total of your data. All filesystems work this way, FAT or NTFS, differing only in blk sizes, but some optimizations are possible, occasionally. – Sam Varshavchik Oct 29 '18 at 1:28
  • 7
    NTFS is substantially more efficient than any version of FAT at handling lots of small files. If you're only ever going to use this USB drive with full-size computers running Windows, formatting it as NTFS is a perfectly reasonable thing to do. If you planned to plug it into a camera, on the other hand, or an Apple product, they wouldn't be able to read it. – zwol Oct 29 '18 at 1:53
  • 3
    Is it possible the exFAT was gratuitously formatted with a block size much larger than 4k? That could be fixed by reformatting as exFAT with sane options, with no loss in compatibility. – R.. Oct 29 '18 at 3:07
  • 3
    @zwol Apple products can read NTFS drives. They just can't write to them by default – awksp Oct 29 '18 at 13:36
  • 4
    MSDN says that the default cluster size for an exFAT partition with 128 GB is 128 kB. That will behave very badly with small files. Zipping is your friend here. – Peter A. Schneider Oct 29 '18 at 15:37
14

When formatting as exFAT, you almost surely chose some large allocation unit (block size) like 128k or 512k. Reformat with the standard 4k allocation units and the problem should go away.

  • 4
  • 2
    Yeah, that's a huge problem. Reformat with 4k. – R.. Oct 29 '18 at 5:21
  • 1
    For me, the default allocation size was 512KB.. – Felix Lee Oct 29 '18 at 6:46
  • @FelixLee that means even a 1 byte file will take 512 KB. – Captain Man Oct 29 '18 at 13:56
  • 1
    @GalacticCowboy: It may not presently be a requirement, but it's just general best practices. It may especially become an issue in the event of needing to access the files from a different system or if there's corruption on the drive and OP is trying to recover as much data as possible (good luck if it's nested in an archive file, especially a compressed one). – R.. Oct 29 '18 at 18:21
7

Why is this happening?

Because you're storing a lot of tiny files.

Filesystems have a minimum file size that they can store. For NTFS filesystems, it's usually 4KB. For exFAT, it can be much larger. That's called the block or cluster size. Files that are smaller than this size will still use up the minimum size, so a 1KB file might use 4KB of disk space. A 3KB file would also use 4KB of disk space. If you have a 5KB file, it'll use 8KB of disk space.

You can imagine it like a grid of holes. Each hole can hold a certain amount of data. Files are spread across as many holes as necessary to hold all the file's data, but holes can't have data from more than one file. So, if a file's data doesn't completely fill a hole, some of that space is wasted. No other file can use it that hole so the unused space is unavailable.

What can you do about it?

In your case, you have a lot of files that don't fill the holes, so there's lots of wasted space. If you were to put all the files into a ZIP file, then all that data would be contained in a single file and it would use a lot less space on the drive.

Some USB drives are formatted as exFAT by default, so alternatively, if you're just using this drive to copy files between Windows computers (or just for storage), you could try reformatting the drive as NTFS (but copy all the files off first, obviously!) to try to get a smaller cluster size.

2

As the other answer suggested, use an archiver, but I'll recommend using 7z instead of WinRAR because it's free, and also you can avoid installing any third-party archivers if you use Windows' built-in "Send to > Compressed (zipped) folder" option when you right-click files and folder. It's faster than 7z but it archives slightly slower.

In case you need to store mostly JPEG images or something else that doesn't compress at all, you should benefit from using 7z and picking the "no compression" option explicitly.

Using .zip archive format over .rar or .7z is important because Windows supports browsing them as if it was just any other folder (albeit with some limitations).

If you are okay with not being able to browse files like that on the flash drive, you can use another format, but the important part about the files not taking so much space is having a single archive file instead of all the original files separately.

  • 3
    If the size grew 4x over nominal size, the vast majority of the files are 1k or smaller. These almost surely aren't jpeg files. – R.. Oct 29 '18 at 3:06
  • The other thing is, why would you ever select "no compression"? It'll still be slightly smaller, even if the files are not overly compressible. – Clonkex Oct 29 '18 at 4:08
  • 1
    @clonkex because speed and latency are a thing – PlasmaHH Oct 29 '18 at 8:18
  • 2
    @Clonkex Because compression algorithms are relatively slow and resource intensive by their very nature, if you know it's not going to get a meaningful gain from being compressed with the extra time to compress/decompress the files, why not tell the zipper to skip that step? – Trotski94 Oct 29 '18 at 9:34
  • 1
    @JamesTrotter for ordinary (non-LZMA) compression algorithms, on ordinary machines, the compression code runs faster than the disk can write, so there is no "extra time" — writing the compressed archive is faster because it writes fewer bytes and the disk is the bottleneck. – hobbs Oct 29 '18 at 15:11

Not the answer you're looking for? Browse other questions tagged or ask your own question.