29

Say Johnny makes an EMPTY file. It is called foobar.py. When Johnny allows it to be executed, he runs chmod 755 foobar.py. The file now has the metadata of

-rw-r--r-- 1 johnny staff    0 Dec 27 22:53 foobar.py

Where is all that metadata stored in that file? The size of the file is 0, so how does it keep the metadata when it is transferred to another drive?

  • 1
    i'm no expert but i guess the general answer is that when you have a hard disk and you make 1+ partitions, then you format the partition with a file system, e.g. windows tends to use ntfs and linux might use ex2, then the bulk of that partition is for file contents, but some small amount of it is reserved for other stuff including metadata. – barlop Dec 28 '16 at 7:04
  • @barlop essentially correct. Both systems use some space for recording where files are stored; in NTFS the "master file table" stores the metadata, in ext2+ it's in "inodes". – pjc50 Dec 28 '16 at 14:57
  • @pjc50 thanks. and metadata aside, what is the name for the thing that is outside of the partitions? I suppose it depends whether the thing is MBR or GPT.. In MBR the thing is called the MBR.. What's it called in GPT? (I understand GPT has a legacy MBR but does it have its own thing too, outside of all the partitions?) – barlop Dec 28 '16 at 15:33
  • Related: (basically same thing, but question is specifically about Windows) How are the file metadata stored in Windows? – gronostaj Dec 28 '16 at 16:37
  • 2
    "chmod 755 ... The file now has the metadata of ... -rw-r--r-- ..." you mean -rwxr-xr-x. – JoL Dec 28 '16 at 17:07
42

It's not stored in that file. It's stored in the filesystem, and all parameters are copied manually one-by-one (though some cannot be copied at all).

That is, most operating systems don't really have a "copy file with metadata" call. The file-copy program just creates a new file named foobar.py, copies the whole 0 bytes of data, then uses utime() or SetFileTime() to make its modification time look the same as the original's. Likewise, file permissions would be "copied" by setting them anew using chmod() or by copying the POSIX ACL attribute.

Some metadata isn't copied. Setting ownership requires root privileges, so copies of someone else's files belong to you and occupy your disk quota. The ctime (attribute change time) is impossible to set manually on Unixes; btime (birth/creation time) is usually not copied either.

Compare cp -a foo bar (which copies metadata) and cp foo bar (which doesn't):

$ strace -v cp foo bar
…
open("foo", O_RDONLY)                   = 3
open("bar", O_WRONLY|O_TRUNC)           = 4
read(3, "test\n", 131072)               = 5
write(4, "test\n", 5)                   = 5
read(3, "", 131072)                     = 0
close(4)                                = 0
close(3)                                = 0
…
$ strace -v cp -a foo bar
…
 -- original metadata is retrieved
lstat("foo", {st_dev=makedev(254, 0), st_ino=60569468, st_mode=S_IFREG|0644,
             st_nlink=1, st_uid=1000, st_gid=1000, st_blksize=4096, st_blocks=8,
             st_size=5, st_atime=2016-12-28T09:16:59+0200.879714332,
             st_mtime=2016-12-28T09:16:55+0200.816363098,
             st_ctime=2016-12-28T09:16:55+0200.816363098}) = 0
 -- data is copied
open("foo", O_RDONLY|O_NOFOLLOW)        = 3
open("bar", O_WRONLY|O_TRUNC)           = 4
read(3, "test\n", 131072)               = 5
write(4, "test\n", 5)                   = 5
read(3, "", 131072)                     = 0
 -- modifiction time is copied
utimensat(4, NULL, [{tv_sec=1482909419, tv_nsec=879714332},
                    {tv_sec=1482909415, tv_nsec=816363098}], 0) = 0
 -- ownership is copied (only with 'sudo [strace] cp')
fchown(4, 1000, 1000)                   = 0
 -- extended attributes are copied (xdg.origin.url is set by browsers, wget)
flistxattr(3, NULL, 0)                  = 0
flistxattr(3, "user.xdg.origin.url\0", 20) = 20
fgetxattr(3, "user.xdg.origin.url", "https://superuser.com/", 22) = 22
fsetxattr(4, "user.xdg.origin.url", "https://superuser.com/", 22, 0) = 0
 -- POSIX ACLs are not present, so a basic ACL is built from st_mode
 -- (in this case, a simple fchmod() would work as well)
fgetxattr(3, "system.posix_acl_access", 0x7ffc87a50be0, 132) = -1 ENODATA (No data available)
fsetxattr(4, "system.posix_acl_access", "\2\0\0\0\1\0\6\0\377\377\377\377\4\0\4\0\377\377\377\377 \0\4\0\377\377\377\377", 28, 0) = 0
close(4)                                = 0
close(3)                                = 0
…
  • 3
    to complement this answer you should mention : - when copying to another drive: the metadata is read from the source, and reproduced on the target if the appropritate settings (or options) (for ex: keep date, keep rights, or even keep "everything") were used (as you mentionned). 2) An alternative is to first do an archive (.zip, .tar, etc) of the files, and extract from this archive on the target, once again giving the program some place (in the archive format) to find the metadata, and specific options/settings allow one to keep (or not) those metadatas. – Olivier Dulac Dec 28 '16 at 12:53
  • To the second paragraph: What about stat(2)? – cat Dec 28 '16 at 15:37
  • Thanks for giving me a detailed answer to this one question I've pondered about. – juniorRubyist Dec 28 '16 at 18:49
12

It generally differs from filesystem to filesystem where the metadata is stored. On the ext2-family of filesystems, the metadata you mentioned (owner, group, permissions, time) are stored in the inode. The inode also stores (pointers to) the blocks the file occupies on disk. The inode does not store the filename.

You can access this data with the stat system call (man 2 stat), and use the stat tool to print it (man stat). A detailed description of the inode fields can be found in linux/include/linux/fs.h in the kernel source.

There are other kinds of metadata (e.g. ACL permissions) that are stored in different places.

Metadata is not copied by default when you copy the file. Instead, a new file with default metadata values is created. There are various options to cp (-p, --preserve) which instruct cp to also copy metadata, by reading the old metadata with stat and modifying the new metadata accordingly.

4

Depending on the file system, areas are reserved either (semi-)statically or dynamically to hold metadata such as permissions, size, and others (sometimes the file name too).

In Unix, metadata is stored in the inode controlling the data area where the file resides (while file names and related inode numbers are stored in a directory entry).

In some filesystems directory entries are files like any other, but hidden from view. FAT and FAT32 are such filesystems (FAT's root directory is "special" though). When you create a file, you add/edit an entry in the file that describes the folder where the file resides. Each entry is large enough to store file size, name and date, and nothing else (long names occupying multiple entries; the default entry size of 32 bytes can hold a single name in the old 8+3 character format. All this, of course, assuming my memory is working). Ext system is similar, but the directory entry is dynamically sized and holds only the name and the inode pointer; all other information is in the inode. This way, two entries may point to the same file, which is useful to manage duplicate files.

In some filesystems, inodes can be large enough to hold a small amount of data in addition to the metadata, so that if the file can fit there, it does not occupy extra disk space. You create a 45-byte file and the free disk space does not change at all; those bytes are stored inside the inode. I think that the ext* family supports this (and NTFS too). This helps to manage large number of very small files.

In yet other file systems, there's what amounts to a "phantom" file system along the main one, that stores these extra attributes. Not only file information but possibly file icons as well.

Some systems have both: NTFS has the full directory metadata working in inode-like fashion, and the possibility to create alternate data streams holding further information that does not (apparently) change anything in the "main" file.

  • 2
    File names aren't stored with the file, they're part of the directory inode. That's why hard links work – Sobrique Dec 28 '16 at 15:40
  • this answer conflicts with dirkt's about where filenames are stored, i wonder which is correct – cat Dec 28 '16 at 15:47
  • Sorry, I mixed up things, and @dirkt has the right of it. Fixing answer. – LSerni Dec 28 '16 at 17:28
  • They're part of the directory, but usually not part of the directory's inode. It's FS-specific, but if you think of a directory as a special file, then its contents would be the list of files (names and their inodes). – grawity Dec 28 '16 at 19:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.