33

I've noticed that on non-Windows OS.... ie Linux/Mac I can do things like:

  • Send a zip to a friend over aim
  • Delete the file while it's in transfer

And the transfer does not fail.

Or, I can do operations like..

  • start a movie
  • erase the file
  • the movie still plays to completion (read from disk, not just buffered in memory)

Although the files are being "deleted", as i mentioned, they are actually being moved to a different location on the file system... ie a Trash directory or something. So it seems to me like the OS uses a pointer @ the file that is updated when it moves rather than accessing the files directly.

Can anyone shed some light on how this AWESOME capability is actually implemented? I'm not even sure what to google to learn more about it.

1
  • 1
    In theory I suppose there are 2 ways it could happen. Both are reality. The file's in memory. Or the file system has references to the files and only the reference is deleted. Both are reality. The references fact is how undelete programs can undelete things. The idea that the file is actually deleted when all the programs acessing it close, sounds like nonsense. You could look up about linked lists(a programming data structure, to understand the concept in more depth). Or you could look up about particular file systems.
    – barlop
    Feb 27, 2011 at 21:09

4 Answers 4

32

The directory entry is just a pointer to an inode. The inode contains the meta-information about the file (other than the name), and pointers to the file's data (if any). When you start to copy a file you get a handle to the inode.

The operating system maintains a count of references to the inode. As long as there are references to the inode, the inode and the file's data are kept. Once all references to the the inode are removed, the inode is and the space required by the file is released.

As you have the file open for copying it will be kept until your process closes the file. This should occur when the file transfer finishes, and will happen if the copy process fails. If the file transfer fails part way through and you have deleted all hard links to the file, you will be unable to successfully restart the transfer.

EDIT: As other have noted, file moves on the same device are done without moving the data. Instead a new directory entry is created in the destination directory, and the original directory entry is removed.

It is possible to have multiple directory entries for the same file. These are called hard links. They are created by making a new directory entry for the file without removing the original entry. The file system's inode has a reference count to record the number of directory entries pointing to the file.

EDIT2: If the process crashes or is killed, the file will be cleanly removed as the in memory access count will be reduced to zero. This is the action that occurs when the the program ends normally.

In the case of a power fail or other unorderly system shutdown, the disk will need an fsck (file system check) before it can be fully mounted. Depending on the state of the on-disk inode and directory structures, the space will be recovered, the file will remain in the directory, or a new entry will be made in the lost+found directory. The results will depend on which changes have been flushed to disk or written to the file systems journal.

3
  • 5
    Exactly. I couldn't put it better myself. Incidentally, a common trick, if you need some scratch file storage within a program, is to create a file in /tmp, and immediately unlink(2) it. At that point, there's no file in a directory (so nothing to clean up at exit or crash), but your process still has access to the file, and no other process can accidentally or deliberately mess with it. That illustrates the property of interest. Mar 1, 2011 at 15:58
  • Does this mean that if an open file is removed while a program is using it and a power failure occurs, stranded data will remain on the drive occupying space? What about if the process using the file crashes or is killed?
    – Jason C
    Jun 10, 2014 at 22:03
  • 1
    @JasonC I have updated the response to answer your questions.
    – BillThor
    Jun 10, 2014 at 23:56
6

As explained by Matt Jenkins, the OS (the filesystem) keeps track of files that are kept open by applications. As long as a process keeps a file open, its data stays on disk (even though it has been deleted and is no longer visible or accessible to other programs.

Note that a consequence of this is that the space occupied by a file can only be reclaimed once the last process using it has closed it. That's a FAQ for Linux/Unix file system operations: "'df' command says partition is full, while 'du' reports free space" (see e.g. http://batleth.sapienti-sat.org/projects/FAQs/ext3-faq.html ). If you need to free up space, it's not enough to delete big files (e.g. logfiles), you must also make sure no process keeps them open (typically a problem with logfiles).

1
  • Not true in case of OS X/HFS+: You can move files, e.g. to the Trash, but not across partitions or delete them (i.e. emptying the Trash).
    – Daniel Beck
    Feb 27, 2011 at 22:23
2

It's quite simple actually. The file maintains a list of references - processes that are accessing the file. When you delete the file it just removes the listing from the directory but not the file itself. The programs that still have the file open can still access it. The file is only actually deleted when all the programs that are accessing it close it.

Also, with moving the file - if it's within the same filesystem - the file doesn't actually move as such, it just changes the pointer to the directory the file is in.

6
  • "The file maintains a list of references"?
    – Daniel Beck
    Feb 27, 2011 at 20:43
  • Should be the "OS or Explorer" not file.
    – Moab
    Feb 27, 2011 at 20:48
  • 2
    That would actually be the file-system.
    – JRobert
    Feb 27, 2011 at 21:08
  • The OS uses the filesystem to store the references in the file (nlinks in the file header structure) - there - we're all happy ;)
    – Majenko
    Feb 27, 2011 at 21:26
  • 1
    "References" are both processes with an open handle to the file and hard links (including the first file name) to the file. The data blocks are not marked free and available for reuse until the reference count goes to zero.
    – mpez0
    Feb 27, 2011 at 21:40
0

In unix file systems, a file can have one or more hard links to it. The file lives as long as there is at least one. When the last one is removed, the file's space is freed. link above will give you a starting point for more reading - see especially the "Link counter" section.

1
  • 2
    True, but that's not what's happening here. The point is that even after the last hardlink is removed, the file still stays until all pocesses have closed their handles to them.
    – sleske
    Feb 27, 2011 at 21:22

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .