55

I put 4096 characters in a text file and save it. Since every character is 1 byte, the Size of the text file must be 4K bytes. As you see below that's OK:

enter image description here

I connect my flash memory to my computer. the free space on the flash memory is 1,717,518,336 bytes:

enter image description here

I created a copy of the file in my flash memory. And again take a look at the free space. it has 1,717,514,240 bytes free space:

enter image description here

Let see what is the difference:

1,717,518,336 - 1,717,514,240 = 4096 bytes


My question:

Q1:

As you see in the last picture above, the only space that the file occupy on the flash, is the space for its contents [characters].So where is metadata file?

I mean, when I move the file to another computer, how it understand name of file, Owner of file, Date created and modified and ... ?

Doesn't it occupy any size?!!

Q2:

Can I see the metadata file in the flash memory?

enter image description here

Appreciate your time and consideration.

14
  • 12
    metadata would be contained in the filesystem itself. that isn't part of the file size reported by Windows. Furthermore FAT32 and NTFS metadata would be different.
    – Ramhound
    Aug 4, 2014 at 10:47
  • @Ramhound when I move the text file to flash memory, its metadata file also move to flash memory, right? how I can see it? Aug 4, 2014 at 10:53
  • 6
    Possible duplicate of How are the file metadata stored in Windows?, How are filenames stored?
    – and31415
    Aug 4, 2014 at 13:06
  • 1
    Aside from understanding OS internals (worthwhile in its own right), there is probably no reason to ever look at or modify such information unless you're writing a disk repair/recovery program. For normal use, you would never want to alter information at this level as it could easily undermine the integrity of the filesystem itself and of everything that uses it.
    – Joe
    Aug 4, 2014 at 21:16
  • 3
    Short answer: It does take up space, but it's not counted as part of the file size.
    – user253751
    Aug 6, 2014 at 2:43

6 Answers 6

54

Yes, metadata occupies space. On NTFS it occupies 1024 bytes, to be specific. However, the information is not stored in the file, but in the master file table MFT. Specifically in MFT record #4 $AttrDef.

See this Technet article for details: table 3.5 holds all MFT records defined.

When a volume is formatted with NTFS, a Master File Table (MFT) file and other pieces of metadata are created. Metadata are the files NTFS uses to implement the file system structure. NTFS reserves the first 16 records of the MFT for metadata files.

NTFS creates a file record for each file and a directory record for each directory created on an NTFS volume. The MFT includes a separate file record for the MFT itself. These file and directory records are stored on the MFT. The attributes of the file are written to the allocated space in the MFT. Besides file attributes, each file record contains information about the position of the file record in the MFT.

Note that other file systems can and do deal differently with metadata.

EDIT: It has been pointed out in the comments section that this answer is missing the point because the OP asked for metadata on FAT32 filesystems, not NTFS. If I knew how, I would remove the "correct answer" attribute. Therefore I provide additional information that answers the question with regard to FAT32.

FAT32 saves simple metadata such as visibility or modification time for each file and folder in an entry in the parent folder of the the file or folder, creating a tree down from the root folder of the FS. As pointed out with regards to NTFS this is not a file but saved within the folder data structure. The entry originally was 32 byte large and contained the following attributes:

Name (8.3) xxxxxxxx.yyy. (88 bits)

Attribute byte (8 bits of information, described later in this section).

One reserved byte.

Create time (24 bits).

Create date (16 bits).

Last access date (16 bits).

Two reserved bytes.

Last modified time (16 bits).

Last modified date (16 bits).

Starting cluster number in the file allocation table (16 bits).

File size (32 bits).

The list was taken from this Microsoft Technet article and pertains to FAT16. Since the cluster size of FAT32 can be 32 bits and the name of files can be longer than 8.3 the table is not wholly accurate. To accomodate long file names and larger disks FAT32 modifies some behaviour which can be read up in the Wikipedia here but the basic idea holds.

13
  • 2
    You can use the Sysinternal suite's NTFSinfo to look at Metadata and find any extends. You can download it here: technet.microsoft.com/en-us/sysinternals/default There are several metadata editors around, but I can't recommend any one because I haven't used them.
    – bjanssen
    Aug 4, 2014 at 11:48
  • 9
    @TheGoodUser-Sp metadata, on NTFS at least, isn't stored in "a file" in the conventional sense, which is what I get the impression you are looking for. There isn't something tucked away in the \windows folder somewhere that we're all just not telling you about; file metadata is an integral part of the filesystem itself.
    – Rob Moir
    Aug 4, 2014 at 13:59
  • 2
    In this case the Flash Drive is formatted as FAT-32. So the metada are stored in File Allocation Table (FAT) en.wikipedia.org/wiki/File_Allocation_Table
    – jnovacho
    Aug 5, 2014 at 7:41
  • 2
    It's important to note that the allocation tables for NTFS and FAT are preallocated. It differs from format to format, but several sectors are usually occupied with empty preallocated blocks that are to be used for storing metadata in order to reduce fragmentation.
    – Kaslai
    Aug 5, 2014 at 13:49
  • 2
    What makes you think metadata is fixed size? The access control list certainly can vary quite a lot in complexity; I fail to see how it can always fit in 1024 bytes (along with all the other metadata like access and modification times)
    – Ben Voigt
    Aug 6, 2014 at 0:01
27

Doesn't it occupy any size?!!

Yes, but it's a small entry in a large pre-allocated block. That block is counted in the "used" portion of your disk. Adding an entry inside that block doesn't require the block to be expanded.

Depending on the filesystem, eventually the block will be filled and extended somehow after a lot of filenames are added.

Can I see the metadata file in the flash memory?

Not easily

As Ruslan and Blorgbeard commented. You can install a hex viewer such as HxD which will allow you to view (and edit - be very careful) the raw filesystem data. But you'll have to do the interpretation yourself concerning which bytes belong to which filesystem structures. For that you would need some good documentation of the specific filesystem used on the flash disk. FAT32 is likely to be simpler to understand than any of the many variants of NTFS. See Understanding FAT32 Filesystems for example.

7
  • 3
    @TheGood If deprived of Google, I'd probably start by booting Linux and using dd to copy the raw flash filesystem to a file on another disk then use a hex-viewer to examine it in conjunction with a good reference-work on the specific filesystem (if not proprietary and undisclosed). I might make a metadata change with touch, repeat the dd and use a binary diff. Aug 4, 2014 at 11:47
  • 2
    Good thing I'm not deprived of Google.
    – Cthulhu
    Aug 4, 2014 at 12:50
  • 6
    @Cthulhu: The old ones have no need of the necronomicon. fhtagn. Aug 4, 2014 at 15:06
  • 2
    There's no need to change OS to use a hex-viewer on a hard drive directly. Just use a decent hex viewer. (It seems even hiew can do it, but I'm not sure since I was on windows long ago).
    – Ruslan
    Aug 5, 2014 at 7:51
  • 1
    HxD will definitely do it, on Windows. Extras menu, Open Disk.
    – Blorgbeard
    Aug 6, 2014 at 2:39
7

Metadata is not stored (nor reported by usual file management tools) as files, it is stored on filesystem's data on disk.

Depending on the nature / version of the filesystem, each entry will take some amount of disk space to represent the metadata information.

Moreover alongside the space allocated in the Master File Table, some filesystems will also keep journal about files change (taking extra space), and some filesystems can even be extended with extra fields containing special purpose metadata.

So technically metadata takes disk space, but it is not taken in account by most file management utilities which works calling system's API that query the filesystem for file space, not for metadata space.

And this abstraction is only the tip of the iceberg, as filesystem itself works on an abstraction of underlying physical disk space, provided by low level disk routines, so actually only the disk's internal logic will know how many actual memory positions are available for higher levels and how many are marked as not reliable, reserved, or used for checksum.

3

The metadata depends on the file system. The most basic file systems usually used on exchangeable media is based on a DOS file system (FAT). DOS doesn't have users, and permissions. Or, more correctly, some of that information is carried in the 8th data bit of the file name. The only resource overhead that is used in a basic DOS FS is to account for the blocks of the file, or the directory that it is in - and that block is probably already committed for handling ".", already. IOW, adding an empty file wouldn't add another bit of storage, but it would change some existing consumed and allocated bits.

When you use more advanced file systems, with journalling and users, you get metadata, and journal entries and possibly a forensic chain to recover previous file versions, etc. Then a small file can explode in used storage.

So, check the FS type. If it is FAT, then you probably don't have a user recorded in metadata on the media. Hence... no space used. :)

A good example of a FAT FS is an open source implementation - and you can see that the list of operations available does not include "get/set user" and "check access by user". No identity = no storage of that identity.

3

Where are metadata kept?

When we talk about metadata, there are two types of metadata.

The first type includes created date, last modified date, last accessed date. Depending on file system (i.e. NTFS / FAT / Ext3 etc...) there will be different "metadata" available, for example Windows owner and permission on NTFS.

The first type applies to all files, e.g. the .txt file in your example.

All file systems pre-allocate spaces to contain these metadata at the Master File Table (MFT for NTFS, some other names for other file system), which is not directly accessible by users. Since MFT is not counted as part of drive capacity, there is no "additional" storage space required anyway.

The second type of "metadata" is defined by the file type or application. For example, Office documents keeps "Author", "Subject" and other metadata; JPEG images keeps a set of EXIF data including "date picture taken", "camera model", "shutter speed"; while MP3 sound contains "album", "track #", "bitrate"...

The second type DOES take up additional space, because these "metadata" forms part of the file.


Different size in different drive

When your text file is on C:\ it takes up 4K. It becomes 1K in size when you put it on your flash drive H:\. This is because different "block size" for the different partitions.

Files are allocated spaces in block. Therefore, on a file system of block size 4K, 1 byte is allocated 4K while 4,097 bytes (4K + 1 byte) is allocated 8K.

Apparently your C: is formatted with 4K block size while H: is formatted with 1K block size, resulting in the difference.

2
  • When I copy a blank text file to a flash memory, its metadata also will moved to flash memory. now, Is there any way , in Linux for example, to see the metadata file? in binary for example. Aug 6, 2014 at 5:58
  • Whether the metadata is copied depends on how you do it. For example robocopy gives you option whether to copy timestamps, attributes, owner information, access control lists (ACL) etc. It also depends on whether the file system on your flash memory supports it (e.g. FAT32 does not keep owner info and ACL so they are discarded anyway). Metadata is not in the form of a file so you cannot see the raw data; if you want to see low level things you may need dd in Linux but how to interpret them is another story.
    – Kenneth L
    Aug 6, 2014 at 6:23
2

As you see in the last picture above, the only space that the file occupy on the flash, is the space for its contents [characters].So where is metadata file ?

The "metadata file" is the directory that contains the file. That's basically what a directory is -- a collection of metadata describing the contents of the directory.

I mean, when I move the file to another computer, how it understand name of file, Owner of file, Date created and modified and ... ?

Doesn't it occupy any size?!!

Yes, in the directory. In most filesytems, the same file can have two different names if it's linked into two different directories.

Can I see the metadata file in the flash memory?

If your filesystem supports it, you can see it by looking at the size of the directory.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .