Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

And that system crash happens while your file is being written to by your program?

Also, is it the case that when your program writes to the file, it actually isn't writing directly to disk, but rather to memory instead?

Because it just seems like it would be more effective for the system to transfer the written data to memory, and then when it's completely written to transfer things back to the disk.

In any case, how does the disk recover data whether your program writes directly to disk, or whether the finished file is in the process of being copied over by the system to various locations on the disk?

share|improve this question

migrated from programmers.stackexchange.com Oct 24 '11 at 20:17

This question came from our site for professional programmers interested in conceptual questions about software development.

    
What operating system are you using? What filesystem are you using? –  FrustratedWithFormsDesigner Oct 24 '11 at 20:04
    
Hmmm right now my computer is a Windows 7 (which I'm assuming is NTFS). But I was wondering in a general sense for this question (an answer specified for anyone's machine is fine). –  Kaitlyn Mcmordie Oct 24 '11 at 20:06
    
Check out Shadow Copy (VSS) for windows, and I think linux journaling filesystems have some metadata that is written before the write to signify that the file is being written somewhere else, and then another piece to signify that the file was written. –  Malfist Oct 24 '11 at 20:07
    
Btw, what do you mean by "linux journaling filesystems"? Do you mean linux logging filesystems...? –  Kaitlyn Mcmordie Oct 24 '11 at 20:10
1  
Here you go, I'll update my answer too: en.wikipedia.org/wiki/Journaling_file_system –  Malfist Oct 24 '11 at 20:12
add comment

2 Answers

up vote 7 down vote accepted

Because it just seems like it would be more effective for the system to transfer the written data to memory, and then when it's completely written to transfer things back to the disk.

In modern operating systems, file access is buffered. Your program writes to a part of memory that's maintained by the OS, and when that area gets full, or when you close the file, the buffer is written to disk. This way, you can write several gigabytes of random data into a file if you want, but you don't consume that much memory while you're doing it. At the same time, the disk subsystem is free to do other things while your program is writing a few bytes at a time.

In any case, how does the disk recover data whether your program writes directly to disk, or whether the finished file is in the process of being copied over by the system to various locations on the disk?

Depends on the operating system and on the particular file system. Often, the data is written into a new file on disk. When your program closes the file, the buffer is flushed and the directory is updated to point to the new file instead of the old one, which can subsequently be deleted. This avoids data corruption if the system crashes during writing -- changing the directory at the last moment creates the illusion of "atomic" writes.

You might want to read about journaled file systems and transactional file systems.

Other strategies are also possible. Entire textbooks are available on file systems and fault tolerance.

Finally, sometimes the file system can't recover your file. Sometimes it can't recover at all. The strategies I mentioned above are responses to the kinds of problems you're asking about, and as far as I know they're pretty effective at keeping the disk in a consistent state. Older file systems weren't so good at that, and if you happened to lose power just when the disk was being written, the file or the whole disk could be compromised. There used to be a large market for disk repair utilities such as Norton Disk Doctor. That market seems to have waned with more reliable OS implementations, but some of those products are still available.

share|improve this answer
1  
Thanks! Just two real quick questions here: 1) So let's say that the file does not get written to atomically, and everything must be "rolled back". Does that mean the next time the system is rebooted, the system will check the log and make that change again? 2)Also, I'm not sure what you mean by fault tolerance as that's something I've been seeing when referring to distributed systems, but not so much with disks... –  Kaitlyn Mcmordie Oct 24 '11 at 20:46
    
@KaitlynMcmordie 1) Again, what happens depends on the file system and OS, but generally if a write can't be completed for some reason (like power failure) then the file system won't show the change at all. So, if the file was just being created, then it won't exist on the disk. If it was being modified, then the file on the disk will be the version from just before the write happened. 2) Robustness might be a better term than fault tolerance here; I just meant the ability of the system to remain consistent despite various kinds of errors. –  Caleb Oct 24 '11 at 22:23
2  
1) When the OS gets done with a filesystem, it marks it as clean. After a crash, it sees that it's dirty and the journal gets indeed processed. However, this doesn't say you get the original or the new data in the file written to during the crash. You only get a filesystem in a sane state (no used blocks marked as unused, no dangling references, correct link counts, ...). –  maaartinus Oct 24 '11 at 22:27
    
2) Asking google about disk and fault tolerance gives three million results. The same for file system and fault tolerance. In fact, fault tolerance is a very broadly applicable term, see e.g. RAID. –  maaartinus Oct 24 '11 at 22:32
2  
@KaitlynMcmordie: A "dangling reference" would be a filename that's still in a directory, but no longer has a physical file associated with it. This would happen if the computer crashed halfway during the deletion of a file - the OS needs to remove both the file and the filename, and they're usually stored in different places. –  MSalters Oct 25 '11 at 8:25
show 1 more comment

Most modern file systems are journaled. This means that there is a journal of changes to the file/filesystem. On crashes, the last entry can be rolled back if it is not marked as completed; or if there is sufficient information, can be enacted.

Wikipedia has this:

To prevent this [data loss on crash], a journaled file system allocates a special area—the journal—in which it records the changes it will make, ahead of time. After a crash, recovery simply involves reading the journal from the file system and replaying changes from this journal until the file system is consistent again. The changes are thus said to be atomic (or indivisible) in that they either: succeed (succeeded originally or are replayed completely during recovery), or are not replayed at all (are skipped because they had not yet been completely written to the journal before the crash occurred).

In windows (vista and beyond), there is a technology called Volume Shadow Copy (VSS), which makes the file system act in a "copy-on write" manner, meaning when you write to a file, that old file is not being destroyed, and is in fact maintained intact. This allows restoring after crashes, but more importantly, it also allows access to the last stable version of the file in the event of a file lock and a process (such as a backup service) needs to read that file.

share|improve this answer
    
I see! So by journaling filesystem I'm guessing it's the same as a logging filesystem, just using different terms to describe the same thing? –  Kaitlyn Mcmordie Oct 24 '11 at 20:48
1  
@KaitlynMcmordie: Yes. –  surfasb Oct 25 '11 at 2:57
    
Typically I hear 'journaling' much more than I hear 'logging', fwiw. (I suppose 'logging' could be ambiguous in more contexts.) –  Shinrai Oct 25 '11 at 14:30
    
...as evidenced perhaps by en.wikipedia.org/wiki/Logging_file_system –  Shinrai Oct 25 '11 at 14:32
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.