Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

As you can see below, there is so much difference between the Size and Size on disk fields in my folder. Why is that?

Screenshot showing 50,875 files in 1,504 folders, 105 MB being 1.43 GB on disk

I know that Size on disk should be a little more than Size because of allocation units in Windows, but why that much of a difference? Could it be because of the large number of files?

BTW, this folder is on my Android phone’s SD card. Inside this, my maps app stores its cached maps and the app gets its map from Google Maps.

share|improve this question
8  
Hello thelastblack, and welcome to SuperUser. I edited your question to remove the part about defragmenting, since the two existing answers focus on the size/size on disk discrepancy and the Stack Exchange format works best when each question posted is about a single thing. You certainly can re-ask that as a separate question however, although I think that the answers you have received so far on this question show that defragmentation will not help you. (It also generally does no good on solid-state media.) Feel free to edit your question further if you feel I changed your intent in any way. –  Michael Kjörling Jan 20 at 10:08
1  
@MichaelKjörling Heh, I just edited in a minor discussion on fragmentation (got distracted a bit earlier) –  Bob Jan 20 at 10:13
14  
@MichaelKjörling Don't edit questions retroactively to fit answers. One of the answers addresses the fragmentation part of OP's question. Your edit needs to be rolled back to avoid confusion. –  DanteTheEgregore Jan 20 at 14:07
4  
@DanteTheEgregore If you are referring to Bob's answer, which indeed has been edited to also discuss the effects of fragmentation, then before jumping the gun, please check the edit histories and timestamps on that answer and the question. At the time of my edit, Bob's answer did not cover the issue of fragmentation at all. If the OP wants to do so, editing back in "will defragmenting the media help me with this?" should resolve any outstanding confusion, although I still feel that is better asked as a separate question; IMO the matter of the difference between the two values is unrelated. –  Michael Kjörling Jan 20 at 16:15
8  
Looks to me like this app is seriously badly programmed – consider filing a bug report. I'm by no means a professional programmer, but I once hacked something similar together in JavaME, and of course one of the problems I had to solve was how to store all those little map tiles efficiently (storage & access) in a container. I ended up using uncompressed zip files. –  A. Donda Jan 20 at 18:15

8 Answers 8

up vote 260 down vote accepted

I will be assuming that you are using the FAT/FAT32 filesystem here, since you mention this is a SD card. NTFS and exFAT behave similarly with regards to allocation units. Other filesystems might be different, but they aren't supported on Windows anyway.

If you have a lot of small files, this is certainly possible. Consider this:

  • 50,000 files.

  • 32 kB cluster size (allocation units), which is the max for FAT32

Ok, now the minimum space taken is 50,000 * 32,000 = 1.6 GB (using SI prefixes, not binary, to simplify the maths). The space each file takes on the disk is always a multiple of the allocation unit size - and here we're assuming each file is actually small enough to fit within a single unit, with some (wasted) space left over.

If each file averaged 2 kB, you'd get about 100 MB total - but you're also wasting 15x that (30 kB per file) on average due to the allocation unit size.


In-depth explanation

Why does this happen? Well, the FAT32 filesystem needs to keep track of where each file is stored. If it were to keep a list of every single byte, the table (like an address book) would grow at the same speed as the data - and waste a lot of space. So what they do is use "allocation units", also known as the "cluster size". The volume is divided into these allocation units, and as far as the filesystem is concerned, they cannot be subdivided - those are the smallest blocks it can address. Much like you have a house number, but your postman doesn't care how many bedrooms you have or who lives in them.

So what happens if you have a very small file? Well, the filesystem doesn't care if the file is 0 kB, 2 kB or even 15 kB, it'll give it the least space it can - in the example above, that's 32 kB. Your file is only using a small amount of this space, and the rest is basically wasted, but still belongs to the file - much like a bedroom you leave unoccupied.

Why are there different allocation unit sizes? Well, it becomes a tradeoff between having a bigger table (address book, e.g. saying John owns a house at 123 Fake Street, 124 Fake Street, 666 Satan Lane, etc.), or more wasted space in each unit (house). If you have larger files, it makes more sense to use larger allocation units - because a file doesn't get a new unit (house) until all others are filled up. If you have lots of small files, well, you're going to have a big table (address book) anyway so may as well give them small units (houses).

Large allocation units, as a general rule, will waste a lot of space if you have lots of small files. There usually isn't a good reason to go above 4 kB for general use.


Fragmentation?

As for fragmentation, fragmentation shouldn't waste space in this manner. Large files may be fragmented, i.e. split up, into multiple allocation units, but each unit should be filled before the next one is started. Defragging might save a little space in the allocation tables, but this isn't your specific issue.


Possible solutions

As gladiator2345 suggested, your only real options at this point are to live with it or reformat with smaller allocation units.

Your card might be formatted in FAT16, which has a smaller limit on table size and therefore requires much larger allocation units in order to address a larger volume (with an upper limit of 2 GB with 32 kB allocation units). Source courtesy of Braiam. If that is the case, you should be able to safely format as FAT32 anyway.

share|improve this answer
3  
The wasted space due to minimum allocation sizes is actually technically called "internal fragmentation", so you could say that fragmentation is the culprit. But it's still not something that any "defragment" tool can do anything about. –  hobbs Jan 23 at 6:21
3  
(Less technically, it's just called "slack".) –  hobbs Jan 23 at 6:21
1  
Cluster sizes also limit the maximum filesystem size. For example, if your address space is 32-bit, you have a total of ~4.29 billion possible total clusters. Now, if you use the smallest cluster size supported by NTFS (512 bytes), you can address a maximum of 512 * 2^32 bytes = 2 GiB. If you need a volume that can store more than 2 GiB of data, you have to increase the cluster size. This is all independent of the actual largest file you attempt to store, granted you cannot store a file larger than 2 GiB that is the least of your problems. –  Andon M. Coleman Jan 24 at 20:23
    
4 KiB clusters will allow you to address files in a volume that is up to 16 TiB in size, which should be enough for the foreseeable future. –  Andon M. Coleman Jan 24 at 20:31
    
Well, he could compress his archive of small files into one large file. –  einpoklum Jan 26 at 14:55

This is one of those situations where compressing/archiving into a single file may help. What Bob said in his answer is true but the solution may be easier than reformating the disk as other answers suggests. If you compress or archive the directory (using zip, tar, or any other method) the file system will see that you have a single big file, instead of several smaller ones. Even without compressing you will be getting back almost 1.4 GiB of space back, because all those "small files" will be counted as a single big file.

Inside this, my maps app stores its cached maps and the app gets its map from Google Maps

Maybe you should discuss with the developer to use an archive or a database instead of multiple files. This probably will also help to have the disk less fragmented and will surely save space especially if it's a NAND flash drive. If you explain the ridiculous situation where 100MB of payload/useful data becomes 1.4GiB, there's something wrong with how the data is stored, and the developers should bring a nicer solution.

share|improve this answer
1  
> Inside this, my maps app stores its cached maps and the app gets its map from Google Maps. - unfortunately, in this case, compression (which is effectively a file system above the base one) would require support from this mapping app. –  Bob Jan 20 at 15:04
1  
@Bob then the solution should comes from developer side D: –  Braiam Jan 20 at 15:12
4  
That's totally true. I think for the time being, I should change my app. –  thelastblack Jan 20 at 16:34
14  
@Braiam It's not tricking the file system into thinking there's only one file; there is only one file. As to why the developers don't store the cache information in an archive, it's probably because most archive formats are not designed for fast random writes, which a cache certainly needs. A better alternative might be to use a lightweight database library like SQLite. –  bcrist Jan 21 at 1:30
1  
Absolutely true..... +1 –  gladiator2345 Jan 21 at 4:40

In case anyone is confronted with this problem, it could be useful to also know that another reason to see big difference in file size / space on disk is the use of alternate data streams (ADS)

This applies only to NTFS to my knowledge. ADS are known for both legitimate and not legitimate uses:

  • to tag a file as downloaded from Internet
  • to store metadata (Microsoft wanted to include some of the Apple OS feature, like not using file extension to determine the type of a file)
  • to hide data or code in the context of a malware.

ADS simply: any NTFS file can hold multiple data streams (understand "subfiles"). One is the main stream, used by Windows Explorer and other Windows tools, it holds the usual content of a file. Alternate data streams may contain other information, exactly as the main stream, but they cannot be handled directly by Windows tools (in particular Explorer display the file size as equal to the size of the main stream, regardless of the size of the ADS), you have to use specialized tools or code to write, read, and locate ADS.

The main point is that in case of big file size difference observed, don't overlook the possibility of ADS, and hidden malware.

Another link.

To safely experiment with ADS, try this at DOS/CMD level...

Create and then display the content of a file in the root of C:

C:\> echo The main data stream> test.txt
C:\> type test.txt

Result:

C:\> The main data stream

Now add an ADS with the same method, just specify the ADS name in addition of the file name:

C:\> echo The secret message> test.txt:secret

You have just hidden the secret message in the file. Note that the file size in Explorer has not changed in spite we added bytes in the ADS "secret".

Try to display the ADS content:

C:\> type test.txt:secret

Result:

The filename, directory name, or volume label syntax is incorrect.

CMD type is not able to display the content of the ADS. We will use Notepad instead:

notepad test.txt:secret

In Notepad we can see the content of the ADS:

The secret message

You can also hide a full executable in an ADS of an innocent text file, and run it at any time. Wealth does not harm for hackers :-)

share|improve this answer
    
I'm not a win-man myself, my work is mostly done in Linux. This was very useful. Thank you –  thelastblack Jan 21 at 9:47
4  
It's worth using a tool such as Streams from Sysinternals to check for ADS usage. For instance files downloaded on a Windows system may be tagged with a source in ADS, though this is tiny and shouldn't take up space. It won't show in dir or Explorer output ordinarily. It may take up blocks and aggravate the disk usage problem you are investigating. . –  adric Jan 21 at 13:38

The problem may be because of the cluster size.

According to Microsoft:

If you are not using NTFS compression for any files or folders contained on the volume, the difference between SIZE and SIZE ON DISK is wasted space because of a larger-than-necessary cluster size. You should attempt to use an optimal cluster size so that the SIZE ON DISK value is as close to the SIZE value as possible. An excessive discrepancy between the SIZE ON DISK and the SIZE value is an indication that the default cluster size is too large for the average file size that you are storing on the volume, and that it should be decreased. This can be done only by backing up the volume and then reformatting the volume by using the format command and the /a switch to specify the appropriate allocation size: IE: format D: /a:2048 (This example uses a 2-KB cluster size).

Try formatting your drive with smaller cluster size.

share|improve this answer
4  
That been said, one shouldn't make cluster size less than 4096 bytes or just not multiple of this number. 32 bit OS works with pages which (in non-PAE case) are of 4096 bytes, so using non-multiple clusters may negatively affect file system performance. This is why default size is set to 4096 bytes. –  Ruslan Jan 20 at 14:17
2  
To add on to what @Ruslan said, newer hard drives now have a 4 kB sector size, and it would be optimal to align the filesystem to the physical sectors, and have a multiple of the physical sector size as the allocation unit size. –  Bob Jan 20 at 15:22
1  
@Ruslan I believe you mean to say that it should be a power of two times 4096. 12288 (3×4096) and 20480 (5×4096) are not great choices. –  Scott Jan 23 at 22:57

I see many people recommending to reformat your drive with a smaller cluster size. Since this is an SD card, note that many vendors pre-format the card to the recommended cluster size to match the size of the NAND's cluster size (keeping both in sync is very important for optimum read/write performance and reducing wear-out)

You can't change the NAND's cluster size (it's a physical attribute of your SD card's hardware).

First run scandisk/chkdsk on your SD card to be sure the size report problem doesn't lie within a corrupted filesystem.

Second, I'd suggest you report the bug to Google Map devs, for them being the one to blame here. They should be using a superior storage method. Fixing it should also make the app to run faster on many devices due to less I/O and file system's driver activity.

share|improve this answer
    
Actually, it was not Google Maps, but another app using Google's maps. I informed the developer, and just removed those files from my SD. –  thelastblack Jan 21 at 19:33

This is a general issue with many filesystems. There are two factors at work here, the maximum number of "blocks" a filesystem can handle per logical volume and physical restrictions of the storage medium. Only 1 file can be allocated to any given block (files generally take as many blocks as they need). So a text file with 64 bytes can often take anything from 4k to 32k, depending on the block size of the filesystem it resides on.

One way to think about this is think of each block in the filesystem as a box, and the filesystem as a room. All your boxes are the same size, and you try to fit as many as you can in a room. If you fit them all in with more room left over, you have to get bigger boxes so that the room is filled completely with boxes.

One of the rules for putting things in boxes is that you can't put two unrelated things in a box. They have to be part of the same document. So if I were to type up a page of text, it would have it's own box. If my typed text had so many pages I couldn't fit it all in one box, I'd simply find another box and continue putting pages in there instead, repeating until I'd filed all my pages. I'd also have written down the boxes I'd used for that document and the order of the boxes to read it in sequence.

Depending on how I'd organize the boxes, I may only have enough room in my manifest for a certain number of boxes. So if I had a big room to fill, but only a small number of boxes I'd have to use very large boxes to reach the room capacity.

So in that case my one page document would still occupy a single box, with nothing else sharing it.

The same situations play out amongst various storage solutions. FAT32 can only manage what is considered a low number of "boxes" on today's huge hard drives, so it ends up with very large "boxes" to compensate for this.

share|improve this answer

Aside from cluster sizes, you can also have a discrepancy due to the following conditions:

  • Compressed or encrypted files can use up a different space than that of the logical file size.
  • Linked files will report n times the number of links times the size of the file for the logical file size, but the physical space used is usually less.
share|improve this answer
    
Generally, that could be true. But in my case, high allocation unit was the problem. –  thelastblack Jan 21 at 6:29
3  
Yup I am just trying to add to the answer by giving more possible reasons for the discrepancy. –  Archimedes Trajano Jan 21 at 16:19

You should have a look at Block Suballocation entry in Wikipedia. That is exactly what's happening to you. Using a file system with support for Tail Packaging is a file system level solution for this problem besides changing allocation cluster size.

All have the inconvenient of needing to reformat the disk.

In some case merely storing those files in an archive would fix the problem (and the small files would also be compressed beside stopping loosing space at end of files). This has the inconvenient of spending some time for decompression.

Another option if you have so many small files because of some specific application related problem is store your software data using another method (may be in a database). But of course it's a solution for programmers, not end users.

http://en.wikipedia.org/wiki/Tail_packing

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.