31

Once again, here I am trying to wrap my head around some HDD and filesystem concepts.

When Windows displays a drive as having "unallocated space" and prompts for the drive to be initialised, what exactly does this mean - that the drive has no filesystem on it, or just that the filesystem isn't recognised by Windows?

Is the concept of unallocated space a Windows one, or a hard drive one - in other words, is it possible for other operating systems to write to unallocated space?

If so, does this mean that outside of Windows, a filesystem isn't necessary to write to a hard drive?

  • 7
    I don't know Windows well enough to answer this, still I have one observation: on certain level Windows likes (or used to like? – I'm not up to date with Windows) to treat partitions and filesystems almost as if they were one thing. In effect some Windows users tend to equate the two concepts. Technically these are different things. You can have partitions without filesystems; a filesystem without any partition; a filesystem smaller than its partition; a filesystem not corresponding with the partition type. This comment is to avoid confusion in case the simplified view meets the strict one. – Kamil Maciorowski Oct 9 at 20:57
  • 4
    Trivially, a swap partition doesn't have a filesystem and is written to on a hard drive, so that's the an example that answers your final question. Other examples are fdisk and mkfs, which create partitions and filesystems - they wouldn't be able to work if there was no way to write to an unformatted disk! – Toby Speight Oct 10 at 15:15
  • 1
    "does this mean that outside of Windows, a filesystem isn't necessary to write to a hard drive" – It's true inside of Windows as well. How do you think Windows writes a filesystem to a hard drive? – Jörg W Mittag yesterday
54

is it possible for other operating systems to write to unallocated space?

Yes, any OS can always write to any disk location, whether it's inside or outside a partition.

Partitions are nothing more than a list of regions that the OS agrees to use – the disk always behaves as a single continuous storage medium and doesn't know care about being partitioned.

(Indeed the partition table itself is stored on the same disk, on sectors 0–32ish, and when tools like fdisk or diskpart edit partitions, all they do is update the table stored in those sectors.)

does this mean that outside of Windows, a filesystem isn't necessary to write to a hard drive?

It's not strictly necessary for most operating systems. A program with sufficient rights can open the raw device file such as /dev/sda4 on Linux or \\.\Device\Harddisk5\DR5 on Windows and read/write bytes into any location.

The program could practically invent its own private filesystem that way, or use some other structure (e.g. there are SQL database engines which write the DB into a raw partition). Or it could just dump data wherever.

Filesystems as you know them merely allow all programs within the OS to share the same drive or partition. The OS takes care of space allocation, reliability, etc. and programs don't need to think about anything other than files and folders.

When Windows displays a drive as having "unallocated space" and prompts for the drive to be initialised, what exactly does this mean - that the drive has no filesystem on it, or just that the filesystem isn't recognised by Windows?

When the "Disk Management" app prompts for the drive to be initialised, that means it does not even recognize the partition table (which is supposed to indicate which areas are allocated and which are not).

Windows understands MBR and GPT partition tables, but when this prompt shows up, it means neither was found. Therefore Windows doesn't even get to the point of looking at filesystems – it cannot know where the filesystems are (if there are any).

However, when Windows finds a partition with unrecognized filesystem, "Disk Management" does not prompt to format or otherwise overwrite it – it simply shows that partition as having an unrecognized filesystem, often shortened as "RAW".

When other operating systems create their partitions on an MBR or GPT disk, they will generally honor the existing partition table and won't start using a different one. So if e.g. Linux or FreeBSD created its own partitions, they will be visible as actual partitions in Disk Management – as long as they're in the MBR or GPT, they won't simply turn into "unallocated space" just because the filesystem was not recognized.

However, do not confuse DiskMgmt's prompt to initialize the disk with Windows Explorer's prompt to format the disk. The former is about unknown partition tables, but the latter is about unknown filesystems in a known partition.

(Sometimes Windows accidentally assigns a drive letter to an unknown filesystem, and trying to open it via Windows Explorer will cause a "Format" prompt.)

So in short:

  • Prompt to initialize disk ⇒ No partition table (MBR or GPT) was found. The disk might be empty or it might use a non-standard partition table such as APM.

  • No prompt, but unallocated space shown ⇒ The space is really unallocated. The disk's partition table has no entries that would correspond to that area.

  • No prompt, but unknown partition shown ⇒ The partition contains an unrecognized filesystem.

  • Prompt to format disk ⇒ The partition contains an unrecognized filesystem.

  • 1
    Writing to a device was actually fairly common under *nix with tar and magnetic tapes (tar="tape archiver") or later floppies. It's probably still common for writing disk images to boot media etc. – Peter A. Schneider Oct 11 at 11:56
  • 1
    "Prompt to format disk" this type of Microsoft arrogance just makes me want to rant and rave... - It makes one wonder who thought it would be a good idea to try and convince the (non-careful) user to potentially destroy data... (Windows loves to try and suggest to format my Linux USB drive... - And one better be careful not to type accidentally lest the wrong button be pressed...) – DetlevCM 2 days ago
  • @PeterA.Schneider I guess, reading/writing raw partitions/disks is and will be the base of disk backup tools (so learning dd and being damn careful can save you money). And actually, tools like fdisk must work with the raw device. – maaartinus yesterday
13

Let's first explain how a harddrive works.

A harddrive has space, obviously. You create a partition scheme across the entire drive, and you fill that with partitions. A partition is a reservation in the partition space with a fixed size. (this is a very basic explanation. If I add RAID to the mix, I fear the answer gets unnecessary complex)

If your harddrive has just one partition, and its size is smaller than the size of the harddisk, there is space left for another partition. This bit is called unallocated space.

Now, does this mean you can always create a partition there? No. It is possible that another OS created a partition in that space that is completely unfamiliar by your OS.

If this is the case, your OS will either see this as an unknown partition, or in the worst case as unallocated space. If you create a partition in unallocated space that is actually allocated by another OS, you will destroy the previous partition in most cases.

There is another possibility, the drive has a partition scheme that is not familiar by your OS. If this is the case, the OS will think the drive is not initialized, and will ask you to initialize it. In Windows, it will either be GPT or MBR, but there are known cases where the drive (or SD Card etc) is formatted for a special device and its unique format is a form of security.

  • 7
    Regarding "It is possible that another OS created a partition in that space that is completely unfamiliar by your OS" ... that's possible, yes, but other OSes will still see it in the partition table, Windows included. They just won't be able to open it. – grawity Oct 10 at 4:55
  • 1
    @grawity: That assumes Windows understands the partitioning scheme. In fact, some partitioning schemes don't even have a "partition table". – Jörg W Mittag Oct 10 at 10:43
  • 1
    @JörgWMittag: Your version is already covered by the last paragraph of the post. In the context that I'm replying to (paragraphs 2–4), the disk has a Windows-compatible partition scheme with at least one Windows-compatible partition in it. – grawity Oct 10 at 10:57
5

A hard drive (or its equivalent such as a USB thumb drive) by convention has a partition table on it. An MBR (master boot record) style partition table fits in one sector (512 bytes) at the beginning of the drive (sector number 0). An MBR also contains a boot loader so the partition table space is limited to four "primary" partitions (though some 1980s software supported eight partitions). Each MBR partition entry has:

  • a type code (a single byte) indicating that the corresponding partition is either NTFS, FAT-32 (four flavors), FAT-16 (eight flavors), FAT-12, Linux swap area, etc. Partition code 0 (zero) means "raw", or "no meaningful content", "or disregard partition contents".
  • whether the partition is flagged as bootable. Up to one partition per disk can have this flag enabled.
  • The disk address of the beginning sector of the partition
  • The partition's ending sector—or the number of sectors. Early PCs used cylinder/head/sector numbers; after about 1995, use of "logical block numbers" a single 32-bit value replaced cyl/head/sect addressing

Since a maximum of four partitions was soon intolerable, a mechanism for additional partitions was "extended partitions" in MBR where a specially marked partition can have an embedded MBR containing another partition table. Thus it is possible to divide a disk into dozens of partitions though some partitioning tools might inflict some artificial limit.

A GPT is intrinsically not limited to four partitions. Its basics occupy more than 512 bytes on disk, usually something in the 4 to 16 megabyte range. But the meaning of partitions is the same.


A disk's partition table which has every sector of the disk included in it has "no unallocated space" by the most common use of the phrase. So the usual answer to your question is that unallocated space is that outside of all partitions. However, ambiguity of terminology occurs if a partition is marked as type zero (raw), so there possibly could exist a tool which considers a raw partition as, in some sense, unallocated.

When Windows displays a drive as having "unallocated space" and prompts for the drive to be initialised, what exactly does this mean - that the drive has no filesystem on it, or just that the filesystem isn't recognised by Windows?

It means there is no partition table or all the partition entries are empty. Windows is happy to ignore unrecognized filesystems.

Is the concept of unallocated space a Windows one, or a hard drive one - in other words, is it possible for other operating systems to write to unallocated space?

It is a hard drive convention observed by all standard operating systems supporting PC software. Not sure what "in other words" means. Yes, it is possible to write to unallocated space (even in Windows), but the proper etiquette is to create a partition containing the unallocated space, format a filesystem within it (which writes to the "unallocated space"), and then write files and directories to it.

If so, does this mean that outside of Windows, a filesystem isn't necessary to write to a hard drive?

For a well behaved system, no. Cooperative systems require filesystems to be in place, permissions set appropriately, etc. However, it would be trivial in Linux (for example) to open the disk itself as a big file (raw disk access) and write directly to it. It would require administrative access and some guidance or calculations to determine where to write.

However, ignoring filesystems and partitions is bad. They greatly improve disk management and make life much easier and more sane. Ultimately it is possible to write renegade software which completely ignores sanity and good manners and writes anywhere on the disk it wants.

However, the meta-ness I perceive from this question might be misunderstanding whether you intend to obey the permissions and limits imposed by non-administrator/non-system mode in Windows, Linux, etc.

  • 1
    You state that by strong convention disks including USB drives have partition tables. I'm not so sure; I think both Linux and Windows are capable of dealing with unpartitioned USB disks. They're simply treated as a single partition. – MSalters Oct 10 at 10:36
  • @MSalters: I noticed that occasionally a "disk" would act as a dedicated partition not subject to a partition table. I never investigated further because my involvement was developing digital forensic evidence gathering software and it was good enough to notice what the system thought it was, rather than trying to steer partitions in any particular direction. – wallyk Oct 10 at 23:18
  • 2
    @MSalters it's probably more correct to say they are treated as a block device. Creating a partition table and file system would still be necessary if the USB drive is zeroed. It does not have to be "DOS" though. It is also possible to write "tar" files directly to a USB drive without a filesystem as if it were tape. I think PLAN9 used unpartitioned space. – mckenzm Oct 10 at 23:32
  • 1
    Floppies pretty much never had a partition table. CDs, DVDs, BluRays still don't. "Floppy-like but bigger" disks that were once popular (such as Iomega's ZIP and Bernoulli drives) usually didn't have partition tables on them, but sometimes did, depending on use case. These "larger-than-floppy" removable disks were sometimes called "super floppies", and a USB stick without a partition table is still today said to be in "Superfloppy format". Back when I used Linux exclusively, I never had a partition table on my thumb drives, because … why would you need one? – Jörg W Mittag 2 days ago
4

Other answers already tell a great deal, I'll just add a bit more details to fill some gaps.

One of the question the author asks is:

If so, does this mean that outside of Windows, a filesystem isn't necessary to write to a hard drive?

Actually, you don't need it even inside Windows. But the devil is in the details:

At a low level, the hard drive is just a bunch of bytes that can be read and written. Each byte has a number - the 1st byte, the 2nd byte, etc. And... that's it. Just a bunch of numbered bytes. Note that there's no concept of a "file" or "directory" or "partition" or whatever. When an operating system wants to read or write to the drive, at the lowest level it just says to it "please give me bytes 123 to 456" or "please write this data into bytes 789 to 1234". Or something like that.

In fact, you can easily write a program like that too. If you're a programmer, all you need to do is open a special filename \\.\PhysicalDrive0 like a regular file and you get full access to ALL the bytes on the drive, just as they are written. (Note: this is under Windows. Under Linux you open one of the files under /dev and other OSes might have other mechanisms still)

Mind you, you'll need admin privileges for that, since you do get access to all the sensitive information on the drive. But other than that, that's all there is to it. I've done it myself.

Now, actually, I've lied to you a bit. A little white lie. The drive doesn't actually deal in individual bytes. It reads and writes data in "sectors". That's just a fancy name for "a bunch of bytes at a time". For the "spinning rust" kind of drives, the sectors are usually 512 bytes large. So when you read or write to the drive, you need to do it in 512 byte large chunks. So, Windows actually tells the drive "get me sectors 1 to 5" or "write to sectors 6 to 8". But it doesn't really change the whole idea.

SSDs usually have larger sectors, and the sizes can vary between drives. Typically it's 4096 bytes, but there are drives with 2048 byte sectors, and probably other values too. Again - doesn't really matter, just something to keep in mind. The OS usually abstracts it away and allows you to read and write whatever, and then does a little extra work to make sure the result is what you expect it to be.

Next step are the "partition tables" that wallyk has nicely explained in his answer.

The partition tables aren't anything that the drive cares about - all it cares about is reading and writing sectors of bytes. But we, humans, would like to split the drive into "partitions" and that's where the partition tables come in. A partition table is, by convention, stored at the very start of the drive. It's probably documented in some standard somewhere too, so that all the tools and OSes have the same understanding of them.

The MBR (Master Boot Record) table is an older standard, there since 1982. Since at that time nobody had even dreamed of SSDs or sectors different than 512 bytes, the MBR was designed to take up exactly 1 sector (512 bytes) and had a bunch of other limitations, particularly relating to larger drive sizes.

Eventually the drives got large enough that MBR just wasn't enough, and GPT (GUID Partition Table) was introduced to get around those limits.

Now, whichever format you use, all that the partition table actually says is something along the lines of "The partion 1 exists on sectors 50 to 5000 and is formatted with the FAT32 filesystem; The partition 2 exists on sectors 5001 to 10000 and is formatted with the EXT4 filesystem" and so on.

From this the OS can find the actual partitions and parse the filesystems. And that's where you get files and folders and permissions and other stuff that the OS provides you with. This is all done in software.

And here's another thing to note - as I said, the partition tables are a convention, not a necessity. Hardware doesn't care about it, software does. So you can use a drive even without one. For example, this day and age, it's common practice to use LVM in Linux to manage your hard drives (it's a bit of a long story). And one possibility with that is to add a whole drive to the pool of devices managed by LVM. In this case, there will be no partition table on the drive - LVM will manage it instead.

3

Imagine a HDD like a magnetic board for kids

Magnetic whiteboard for children

When you get a board the first time, it will not be completely white, but a scrambled mix of black and white nonsense. Before you can read or write something, you will first have to erase a part of the board, where you want to write. And since a hard-drive can store all kind of data, you will also have to create some initial information (usually right at the beginning), which defines what kind of data will be on the drive.

I could for example write a pattern of dots in the top left corner to designate the content of the board to be written English. Or another pattern, which would indicate an image. Or a complex pattern, which would divide the board into four parts and defining that the top left quarter will contain an image, the top right will contain English text and the two in the lower half are empty and ready for new data. (And it is enough to write in the pattern "the lower half is empty" and I don't have to clean this part completely, since no one will try to read the content of this space, because it is designated as "empty")

In this way the OS will usually write a clear pattern in the beginning of a drive, which can be used to identify what other data is written on the drive and where it can be found. These parts are the Partition-Table and File System Descriptor.

Unknown data is not easily distinguishable from noise

When an OS encounters an HDD it usually tries to read the first bytes on the drive to identify a certain pattern. Most operating systems know a bunch of different formats how a drive can be partitioned and it tries to find the known markers (Master Boot Record, Partition Tables,...) to identify the contents of the drive. - If an OS (like Windows) can not find a pattern which it recognizes, it does not know, if the drive is just "empty" (filled with random noise, like an magnetic board after shaking it) or if it is just in a format it does not know.

So the "unallocated space" is just unrecognizable data with no identifiable pattern for the OS. If you choose to create a partition in this space, the OS will simply overwrite the first few bytes and create a pattern of data, which essentially says "This drive will contain files in this structure and currently everything after this note is currently empty and can be overwritten" and after that all files which you copy onto the drive will be written in the structure which is defined in this note.

Special cases of "unallocated space"

Unrecognizable space on an HDD can be an unknown file-system. Or it could be a hidden partition generated by an encryption software like VeraCrypt - which is designed so that the hidden encrypted data looks just like random noise on an unallocated disk. Another case can be a damaged file-system. If the hard-drive experienced some damage the pattern in the beginning could be broken, so the system can no longer recognize the file-system, even if it is of a known type. In this case you would need some recovery software, which tries to reconstruct the information about the file-system, so the original data can be accessed again.

Tools to access raw data

There are also tools for Windows/Linux, which allow you to access the raw data of an HDD. These tools will not care if a drive is partitioned or contains just random noise, if you use these you can view the drive in its entirety, just like the magnetic board. There you will see a giant amount of random-looking numbers and symbols, which are usually interpreted by a file-system-driver in the OS to allow you to see a structure of files and folders.

If you are interested in how file-systems work, you could look at a drive with such tools and see the partition tables for yourself.

  • 3
    Downvoter care to comment how this answer could be improved ? – Falco Oct 10 at 9:32
  • 2
    Didn't downvote, but I think part of your analogy is unhelpful: Before you can read or write something, you will first have to erase the board completely. That's not how hard drives logically work (or physically / internally for magnetic drives. And SSDs hide that detail). You just write new data to a sector. The OS only tries to read metadata from places its expecting to find it (GPT or MBR partition table at start of the drive, and if so then filesystem signature within partitions). As you say yourself, the OS doesn't erase when formatting. That's unnecessary. – Peter Cordes Oct 11 at 3:51
  • Other parts of your analogy / explanation do work, though. That's why I didn't downvote. But I think it would be better to distinguish between not recognizing anything in places where there might be metadata vs. what's actually in the whole rest of the drive. An OS doesn't look at most sectors when detecting a disk. – Peter Cordes Oct 11 at 3:53
  • @PeterCordes I tried reformulating this part, to make it clearer that only a part of the whole board/hdd has to contain recognizable data. – Falco Oct 11 at 8:47
  • 1
    Emphasizing the "erase first" requirement is how SSDs work internally, but magnetic HDDs literally don't work that way: they simply remagnetize domains up or down without caring what was there before. It's also not how OSes actually manage data at a logical level; I don't think it's normal for a partition-table editor to write any sectors outside those containing the partition table itself. Detection of an MBR or GPT partition table doesn't require looking at any other data; random garbage everywhere else won't confuse the process. Maybe if you were wiping a disk to leave it unpartitioned.. – Peter Cordes Oct 11 at 14:13

protected by Community Oct 10 at 9:21

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.