17

A while ago, there has been some discussion about ext4 potentially leaving empty files after an unclean unmount, summed up pretty well in this article. Basically, because of delayed allocation, writes can be kept in the write cache for a much longer time than the default commit interval of the ext journal (5 seconds).

The problems seem to have been fixed in a patch that forces block allocation in certain situations, thereby forcing the data to disk after at most 5 seconds by default.

I'm wondering what happens when an application overwrites existing parts of a file, without truncating or appending the file itself. Will that be forced to disk within 5 seconds as well?

It seems like a different situation than appending to a file: when appending, the file size changes, which is a metadata change; therefore, a journal commit will be necessary within 5 seconds, and because of data=ordered, the data will have to be written before that because of security concerns (otherwise parts of deleted files of other users could show up for the owner of the appended file).

When just overwriting file data, there is no reason why the data write should have to happen before the metadata journal commit, as the old data belongs to the same user as the new one. So does the write happen before the commit anyway, or can it be delayed longer than the journal commit interval? If so, how long?

Update: I know that all this is irrelevant when doing the right thing, that is, using fsync(). (This was the main reason for all the discussion about ext4 and data loss - the problem only concerned applications not fsync()ing, or not at the right moments.) I'm not writing my own application, I'm asking because I don't know whether all of my applications do the right thing, and I want to know an approximate timeframe for such "dangerous" writes. The reason for asking is that my graphics driver causes kernel panics regularly, and I want to know whether I have to worry about more than the last 5 seconds of data writes.

2 Answers 2

19

You can set the commit interval to a custom value which, I believe, can be as high as a 32-bit unsigned integer number of seconds; so about 4 billion seconds, or 136 years. This is available through the commit mount option, which you can place into effect as follows (this is merely an example; you can also set this in fstab):

mount /dev/sda1 -t ext4 -o rw,data=writeback,nobh,commit=12345678

The commit interval is not based on any type of condition like whether the data is appended or overwriting existing data or whatever. The commit mount option (which defaults to 5 seconds if you don't supply the mount option at all) is equivalent to doing something like this in a bash shell:

#!/bin/bash
while :
do
    echo "Syncing all uncommitted data and journal to disk"
    sync
    sleep 5
done

Don't confuse data=ordered and this global filesystem sync interval ("commit interval" is perhaps a less meaningful term for those of us who understand the functionality of the command line program sync, in which case it may be better named the "sync interval"). data=ordered is about the order in which data and metadata are updated (where data=writeback is "less safe / faster" and data=journal is "more safe / slower"). commit=12345678 is about the frequency with which the filesystem driver itself forces a FULL sync of ALL dirty data/journal/metadata/whatever to the physical media. And you can most certainly set it to 136 years if you want, and mount with data=writeback,nobh and programs that don't call fsync() or sync() will have dirty pages sitting in RAM for... several lifetimes.

Update: Based on your context in your question edit, I'd say that you should run your filesystem with mount options data=journal,commit=1 or even with the sync mount option, until you are able to resolve your graphics driver kernel panics. This will maintain maximum data integrity but at the cost of performance. You'll especially want to do this if you are frequently writing data to disk that you can't afford to lose, and that's doubly important if you don't "trust" the apps you're using to employ fsync() appropriately.

Source: here and personal experience

6
  • 1
    Thanks, the "ALL dirty data" part was exactly what I was worrying about! I was worried that there were more exceptions in addition to delayed allocation (which can cause new data to remain in the write cache even after the commit interval).
    – lxgr
    Sep 25, 2012 at 14:04
  • 1
    I'm pretty sure that delayed allocation is completely irrelevant when calling sync (or, equivalently, when the commit interval timer is fired). At the point in time when the sync completes, there are absolutely no dirty data, metadata or journal pages. Any changes to the filesystem during the synchronous data transfer are blocked until it completes. Sep 25, 2012 at 14:21
  • 1
    Really? In bugs.launchpad.net/ubuntu/+source/linux/+bug/317781/comments/45 it is specifically mentioned that unallocated pages will NOT be written to disk on a commit (but of course on an fsync()). The patch fixes some common cases where that behavior is problematic by forcing allocation; however, there is nothing said about overwriting data.
    – lxgr
    Sep 25, 2012 at 14:28
  • 1
    Ah, so commit=... and sync are NOT equivalent? Or did tytso imply that even with a sync it doesn't commit unallocated pages? I can't imagine that being the case, since it would violate POSIX specifications. Maybe you could use that bash script I provided for better data safety :P Sep 25, 2012 at 14:39
  • 1
    I'm pretty sure he meant the former, the latter would make ext4 on Linux a pretty dangerous file system to use ;) The script looks like a nice workaround; I'll give it a try and maybe evaluate some of my most important applications with strace - maybe they are all using fsync(), and I'm worrying too much...
    – lxgr
    Sep 25, 2012 at 14:44
1

Whatever the answer to your question is, it doesn't matter.

The guaranteed exposed behavior of the ext4 filesystem is that "data will be on disc after a successful sync/fsync call". So, if you have an application which makes you ask this question, you should insert synchronization calls at the critical points where data integrity needs to be ensured. If you're a user worried about the same issue, you can call the sync command-line utility before doing whatever dangerous behavior might cause an unclean shutdown.

1
  • I know about fsync(); I'm asking as a user of applications that might or might not use it. I've updated my question.
    – lxgr
    Sep 25, 2012 at 13:58

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .