Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

What limits a file to have some maximum size depending on the Operating System?

From this page:

alt text

I do not exactly understand this. If you have the storage space, what else can be the limitation? You should be able to store as much data as you want the way you want (even in a single file) unless you run out of storage space.

share|improve this question
up vote 17 down vote accepted

Filesystems need to store file sizes (either in bytes, or in some filesystem-dependent unit such as sectors or blocks). The number of bits allocated to the size is usually fixed in stone when the filesystem is designed.

If you allow too many bits for the size, you make every file take a little more room, and every operation a little slower. On the other hand, if you allow too few bits for the size, then one day people will complain because they're trying to store a 20EB file and your crap filesystem won't let them.

At the time the filesystems you mention were designed, having a disk big enough to run into the limit sounded like science-fiction. (Except FAT32, but the company that promoted it intended it as an intermediate measure before everyone adopted their shiny new NTFS, plus they were never very good at anticipating growing requirements.)

Another thing is that until the end of the last century, most consumer (and even server) hardware could only accomodate fast computation with 32-bit values, and operating systems tended to use 32-bit values for most things, including file sizes. 32 bits means 4GB, so operating systems tended to be limited to 4GB files regardless of the filesystem, often even 2GB because they used signed integers. Any serious desktop or server OS nowadays uses 64 bits for file sizes and offsets, which puts the limit at 8EB.

share|improve this answer
+1. Clarification to the uninitiated - EB as in exabytes, as in a million terabytes. – Shinrai Aug 13 '10 at 22:16

The on-disk data structures are usually the limit. Research how these operating systems format their disks and how they track the portions of files on the disk, and you'll understand why they have these limitations. The FAT filesystem is pretty well documented on-line (see Wikipedia, for instance) and you can see that their choice of integer sizes for some disk structure fields ends up limiting the overall size of the file that you can store with this disk format.

share|improve this answer

The limitation is simply due to the fact that when the specifications of the filing systems were written, it was never thought that hard drives would be that much bigger.... or other technical limitations whilst designing the specifications.

I think that nowadays, limitations in new filing systems typically go towards what the expected use will be .

... It would be hard for any technical team to release a filing system and saying that it supports 500 Petabyte hard drives without ever doing the testing on it.

My first laptop was a 286 with a 40MB hard drive... I would never imagine ever needing (or hitting the limit) of FAT at the time!

I think the current NTFS limitation is around 16TB per volume, 2TB per file... quite frankly, that is (and should be) good for some time - anything capable (or needing) of writing files larger than 2TBs usually has the ability to split files and/or similar administrative features (e.g. SQL server).

share|improve this answer

Simple answer: you need to be able to read the file, so you need to be able to address the file. This access will be through datastructures that have limits. You'll be stuck with the lowest common denominator of; physical (disk, SD card, etc) limits, file system limits, and OS limits.

share|improve this answer

Knowing the size of every file is just a simplification that we have gotten used to. It doesn't have to be that way and wasn't always the case.

The old mainframes never new how big a tape file was until it had read to the end of it (and even then it may not have kept track of the size). Even the disk files were often accessed sequentially and would just grow as they were extended. Their sizes (in bytes) were not recorded and could not, in general, be calculated without reading them in full due to variable sector sizes and other odd features.

It wouldn't be difficult to design a file-system that didn't keep track of the file size and allowed them to grow unbounded.

share|improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .