1

I was running a long backup with tar (-cvJf) and was not paying attention while cleaning up folders as the backup was running and renamed the destination file (with mv). I figured I had borked the tar process, but to my surprise it was still running and the new file was still growing. Afterwards the file appeared fine so I ran a restore to confirm all the bits where as they should be, which they were.

So my questions are these:

  • How/why does this work?
  • Is this a tar specific feature?
  • Is this a file-system specific feature (ext4)?
  • Is this a distribution specific feature (I'm running Ubuntu 18.04.2)?
  • It seems to work when moving the file to different folders too. I don't have enough free space on other mount points to try this across mount points, but would like to know how far one can push this?

I am sure there is a term for this, but trying to google the effect is giving me a lot unrelated material.

1

It's a feature of ext4 and other file-systems in Linux.

In such file-system files are identified by their inode numbers. Inode is a structure that points to data blocks and to some metadata (e.g. permissions) associated with the file. This doesn't include name nor path.

When we say the file foo belongs to a directory, this means there is a directory entry with the name foo inside the directory, so the entry points to a certain inode.

This approach gives us hardlinks without extra effort. One can add another entry (possibly with another name, in the same or another directory) pointing to the same inode. All such entries are equal and one cannot tell which one is the "original one". Moving a file from one directory to another within a single file-system is technically performed by creating a hardlink (entry) in the destination directory and then removing the entry in the source directory.

When you pass a path like /some/path/to/foo to a program and it tries to open the file, the kernel reads the directory /some/path/to/, finds the foo entry and learns which inode the path points to (note it needs to read / to learn about some/, then it reads some/ to learn about path/ and so on; compare this question). The program obtains a file descriptor which identifies such-and-such inode on such-and-such file-system.

At this moment the program may discard information about the path. As long as it holds the descriptor it can access the file.

I figured I had borked the tar process, but to my surprise it was still running and the new file was still growing.

It was the same file with the same inode. You created an additional path leading to the file and destroyed the original one. After tar had gained access to the file, it no longer needed any of these paths.

If tar memorized the path and (for whatever reason) tried to open the file for the second time using the (now nonexistent) path, then it would fail. This was not the case. Other tools, however, may do this (e.g. some implementations of tail support --follow=name and --follow=descriptor options).


Another feature of inode-based file-systems is the ability to remove (unlink) a file in use. One can remove all hardlinks (paths, names) leading to a particular inode but if the file is open then it will still take space and be able to grow. Descriptors stay valid, programs continue to work with the file. When all programs close the file and it's no longer in use, the kernel informs the file-system and the space is freed. In case of problems (e.g. hard reset) fsck should detect and fix the orphaned inode.

This may lead to a situation where removing files doesn't free disk space. On the other hand you can create a temporary file, open it, unlink it, yet still use it. This is useful and expected in Linux, to the point where the behavior is emulated for file-systems that don't really support this.

I don't have enough free space on other mount points to try this across mount points, but would like to know how far one can push this

Moving a file to another file-system is performed by copying, then removing (unlinking) the original. You are able to copy the file (i.e. read it sequentially and write the data elsewhere). If you unlink the original file before tar ends then tar will still work with it (on the original file-system) because this is the file it opened.

In your case you might end up with truncated data in the copy. If the copying process reached the end of the original file while tar was still running, it would consider copying complete despite the fact tar is about to append to the file (the copying process wouldn't know, wouldn't care). There are ways to recover the original file but if you do nothing and let tar finish, it will close the descriptor, the file will become unaccessible, its data marked as empty space which may be overwritten.

Another general problem with copying a file in use is coherence. tar sequentially writes to the file, so even if you copy only the initial part of it, you may still be able to extract some data out. But imagine a tool that seeks and modifies the file non-sequentially. It may alter fragments you have already copied and fragments you are just about to copy. The result may not only be truncated, it may be incoherent (funny analogy: panorama fail).

| improve this answer | |
2

Because in Unix filesystems (ext4 and others), the actual file is an "inode" (its identifier is just a unique number) and the path/name you put on it is just a convenient way to access it.

When an application opens the file it actually opens the inode, and unless it tries to access the file by name later (to rename it for instance) it won't notice if you rename/move it.

This is also why you can have "hard links": these are just have several path/names pointing to the same inode. When the last name disappears (rm or unlink), the inode file is returned to the free space.

An application won't also notice the file has been erased... if you rm the file while it is written to, it can continue to grow. It will be erased only when the application closes it, which is why sometimes erasing a "live" application log doesn't free any disk space.

If you want to explore: ls -i gives you the inode number of the files, and ls -l tells you how many links there are to the inode of the file.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.