23

When I redirect a command's output to a file (e.g., echo Hello > file) will that file be guaranteed to have such data right after the command exits? Or is there still a very small window between the command exits and data written to the file? I'd like to read the file right after the command exits, but I do not want to read an empty file.

10
  • 1
    It probably executes the command right away, but the amount of time it takes to actually open the file, write, and close will depend on the speed and type of your hard drive, any running programs, etc. – freginold Jan 25 '18 at 12:44
  • In terms of the given example, what is 'the process'? Are echo and > not seperate (short lived) processes? And where does the output of echo remain before > is executed? – oɔɯǝɹ Jan 25 '18 at 17:26
  • 1
    @oɔɯǝɹ > is shell redirection. It is the same as if the program had opened the named file for writing and replaced stdout with it which is exactly what the shell does. – Dan D. Jan 26 '18 at 0:11
  • 7
    I think it is the OS responsibility to give you the file containing Hello regardless of whether it is flushed or not. – Salman A Jan 26 '18 at 6:11
  • 1
    If the program is running on machine A, and you are reading the file on machine B, with machine A's filesystem mounted over the network, then you may end up reading an empty file, depending on the network filesystem type and the mount settings. So you may want to disable caching for that mount. – pts Jan 27 '18 at 13:56
22

There are multiple layers of buffers/caches involved.

  1. The CPU cache.

    The data is put together byte by byte, and stored in the CPU cache. If the CPU cache is full and the data has not been accessed for a while, the block containing our data may get written to main memory. These are, for the most part, hidden from the application programmers.

  2. The in-process buffers.

    There is some memory set aside in the process where the data is collected so we need to make as few requests to the OS as possible, because that is comparatively expensive. The process copies the data to these buffers, which again may be backed by CPU caches, so there is no guarantee that the data is copied to main memory. The application needs to explicitly flush these buffers, for example using fclose(3) or fsync(3). The exit(3) function also does this before the process is terminated, while the _exit(2) function does not, which is why there is a big warning in the manual page for that function to call it only if you know what you are doing.

  3. The kernel buffers

    The OS then keeps its own cache, to minimize the number of requests it needs to send to the disks. This cache belongs to no process in particular, so data in there may belong to processes that have finished already, and since all accesses go through here, the next program will see the data if it has reached here. The kernel will write this data to the disks when it has time to do so or when explicitly asked.

  4. The drive cache

    The disk drives themselves also keep a cache to speed up accesses. These are written fairly quickly, and there is a command to write the remaining data in the caches and report when that is complete, which the OS uses on shutdown to make sure no data is left unwritten before powering down.

For your application, it is sufficient for the data to be registered in the kernel buffers (the actual data may still live in CPU caches at this point, and might not have been written to main memory): the "echo" process terminates, which means that any in-process buffers must have been flushed and the data handed over to the OS, and when you start a new process then, it is guaranteed that the OS will give the same data back when asked.

6
  • 8
    Considering the CPU caching seems not relevant to me. That is an unnecessary level of detail here. As would going through all details until some physical quantity representing a bit on a hard disk platter or ssd memory is changed to flip it. – mvw Jan 27 '18 at 12:57
  • 3
    Indeed, the CPU cache is fairly orthogonal. – Simon Richter Jan 27 '18 at 16:10
  • 2
    And more importantly, the CPU cache is coherent between cores, which is why it's totally out of the picture. On x86, it's even coherent with DMA (and x86 has a total-store-order memory-ordering mode), so anything that can read memory will see the data most recently stored to that address in the global order of memory operations. (A CPU core will see its own stores even before they become globally visible, because of store-forwarding from the store queue). On non-x86 platforms without cache-coherent DMA, the Linux kernel makes sure cache is flushed before DMA to those addresses. – Peter Cordes Jan 28 '18 at 10:29
  • 1
    "These are, for the most part, hidden from the application programmers." Why the "for the most part"? I am an embedded developer and except during boot loader (so not "application") I completely ignore CPU cache. I don't think any application developer can be affected by the effects of CPU cache. – Sam Jan 28 '18 at 15:43
  • 1
    @Sam cache misses / hits together with speculative execution may be exploited in some CPUs to bypass read access restrictions. Perhaps this is what the answer has been referring to? – John Dvorak Jan 28 '18 at 15:48
23

If the application doesn't have any internal caches, then the changes will be immediately written to the file. The same for your example. The file is a logical entity in memory which will be immediately updated. Any subsequent operations on the file will see the changes made by the program.

However, this does not mean the change was written to the physical disk. The changes might linger in the OS filesystem caches or hardware caches. To flush the filesystem buffers, use the sync command.

I'd like to read the file right after the command exits, but I do not want to read an empty file.

You shouldn't run into any practical problems here.

6
  • 1
    “If the application doesn't have any internal caches” — that is a very big “if”: the vast majority of I/O library implementations use buffer stdout by default. That said, the C standard for instance mandates that the stdout buffer be flushed upon exit (but potentially not if exit isn’t at least implicitly called). Other libraries/languages (e.g. Java!) give fewer guarantees. – Konrad Rudolph Jan 25 '18 at 13:38
  • What if just limit it to the redirect primitive (i.e., the command in my question)? It has not internal caches, right? – Eric Jan 25 '18 at 13:45
  • @Eric No, you should be fine. – mtak Jan 25 '18 at 13:53
  • 10
    I'm not sure if I get this answer. The question is about "when the process exits". Every application with internal write caches will flush them to disk on process exit, if that didn't happen earlier. IOW, those caches do not matter here. – MSalters Jan 25 '18 at 15:20
  • 2
    Moreover, an internal buffer will either be flushed on exit or just fade from existance, right? So even if internal buffers don't flush, the content will not be observable, no matter how long one would wait. – WorldSEnder Jan 26 '18 at 2:35
21

Will buffer be automatically flushed to disk when a process exits?

In general the answer is no.

It depends on the command. As the other answers mention, if the command does not internally buffer the data, all data will be available when the command terminates.

But most, if not all, standard I/O libraries do buffer stdout by default (to some extent), and give different guarantees about the automatic flushing of buffers when the application closes.

C guarantees that a normal exit will flush the buffers. “Normal exit” means that exit is called — either explicitly, or by returning from main. However, abnormal exit can circumvent this call (and therefore leave unflushed buffers behind).

Here’s a simple example:

#include <signal.h>
#include <stdio.h>

int main() {
    printf("test");
    raise(SIGABRT);
}

If you compile this and execute it, test will not necessarily be written to stdout.

Other programming languages give even fewer guarantees: Java, for instance, does not auto-flush upon program termination. If the output buffer contains an unterminated line, it may therefore be lost, unless System.out.flush() was called explicitly.

That said, your question body asks something slightly different: if the data arrives in the file at all, it should do so immediately after the command terminates (subject to the caveats described in the other answers).

2
  • 7
    I've also seen abnormal exit when a command-line tool is writing to a file and to stdout or stderr, like a debug log, and the user has done a pipe to head or less then typed 'q' to quit less. The disk file is not always fully flushed if the command-line tool didn't handle SIGPIPE. – Zan Lynx Jan 25 '18 at 19:11
  • +1, but " it should do so immediately after the command terminates" isn't quite right: any write() or pwrite() system call will happen before the process exits, and that's when the file changes become visible. So the last file change is definitely before process termination, immediately-before at the latest. I think even with an mmap(MAP_SHARED) file, there's no way for anything to observe process termination happen before all file changes that are going to happen. – Peter Cordes Jan 28 '18 at 10:37
9

I think that no question addresses this issue sufficiently yet:

I'd like to read the file right after the command exits, but I do not want to read an empty file.

As the other answers explain, a well behaving program flushes its internal file buffers before the process terminates normally. Afterwards the data may still linger in kernel or hardware buffers before it's written to persistent storage. However, the file system semantics of Linux guarantee that all processes see the content of files in the same way as the kernel does including internal buffers1.

This is typically implemented by having at most one in-kernel buffer per file object and to require all file access to go through this buffer.

  • If a process reads a file, the kernel will present the buffer content to the process, if the requested file part is currently in the buffer; if it is not, the kernel will fetch the data from the underlying storage medium and place it inside the buffer, then go back to the previous step.

  • If a process writes to a file, the data is first placed inside the in-kernel buffer for that file. Eventually the buffer content will be flushed to storage. In the mean time read access is satisfied from the same buffer (see above).


1 At least for regular files, directories and symbolic links. FIFOs and sockets are a different matter since their content is never stored persistently anyway. There are some special cases of regular files whose contents depend on who's asking; examples are files in procfs and sysfs (think /proc/self which is a symbolic link to the process ID of the process reading the symbolic link).

2
  • 2
    Strictly speaking, it's not the filesystem semantics of Linux that guarantee this, it's POSIX semantics that do. In particular, BSD behaves exactly the same, as does macOS, and even Windows (though this is one of the few cases where Windows follows POSIX semantics). This also assumes nobody's doing odd stuff with mmap() and O_DIRECT, which can lead to things being out-of-sync between the disk and the page cache (but that will resolve the moment that the process doing that exits). – Austin Hemmelgarn Jan 25 '18 at 20:03
  • 2
    @AustinHemmelgarn: Strictly speaking we're both right since Linux was designed with support for Unix (System V) applications in mind and later made to support POSIX which also bases many concepts on System V. – David Foerster Jan 25 '18 at 22:08
5

Assuming your command is executed by some program using the C runtime library, at some point it should invoke fclose to close the open file.

The man page for fclose C function says:

NOTES Note that fclose() only flushes the user space buffers provided by the C library. To ensure that the data is physically stored on disk the kernel buffers must be flushed too, for example, with sync(2) or fsync(2).

and the man page for fflush has the same note. The man page for close says:

A successful close does not guarantee that the data has been successfully saved to disk, as the kernel defers writes. It is not common for a file system to flush the buffers when the stream is closed. If you need to be sure that the data is physically stored use fsync(2). (It will depend on the disk hardware at this point.)

Note that the data is available to other processes even if it is not synced to the drive. Maybe that is already good enough for you.

If you are in doubt, write a test.

4
  • 2
    C or not, everything will / should use the close() syscall to close a file's descriptor. – Attie Jan 25 '18 at 13:00
  • @Attie: You don't need to close files before exiting (in hacky programs that don't check for errors); the kernel will clean them up, effectively calling close for you after your process dies. You do need to fclose any buffered stdio streams, though, or let libc do that for you with exit(3), as opposed to the exit system call directly. – Peter Cordes Jan 28 '18 at 10:41
  • If you are in doubt, write a test. This is bad advice for detecting race conditions. Testing on one kernel running on one piece of hardware might tell you that the race can't happen under the software conditions produced by your test on that system, or if it does it's too rare to detect. But it can't tell you whether that behaviour is supposed to be safe across all filesystems, kernels, and all hardware (e.g. PowerPC). i.e. you can't tell whether the guarantee you're depending on is an implementation detail or an intentional future-proof guarantee! (In this case it is.) – Peter Cordes Jan 28 '18 at 10:45
  • It depends on the situation. Some folk trying to get his shell script running might be helped by this advice. It was not intended as general solution for more advanced but less likely environments, e.g. a software engineer working on an OS kernel, some folk working on Intel's microcode update, or some gal doing work on some system for the ISS. – mvw Jan 28 '18 at 23:16
3

When I redirect a command's output to a file (e.g., echo Hello > file) will that file be guaranteed to have such data right after the command exits?

Yes. The shell opens the output-file, and echo outputs directly to that. After the command exits, it's done.

Or is there still a very small window between the command exits and data written to the file?

Whether the data is already on the media is another matter, which only matters if there is thereafter a hardware failure, or you inspect the live partition with some forensic software, bypassing the mounted filesystem.

I'd like to read the file right after the command exits, but I do not want to read an empty file.

Don't worry, the kernel only keeps one view of the file, independent of how often it is opened.

1
  • "the kernel only keeps one view of the file": not quite true for mmap(MAP_SHARED): stores into the mmaped region aren't coherent with reads of the file (by that thread or other processes). This is why msync(2) exists. At least that's what the man pages warn about; depending on the implementation, Linux may actually map physical pages from the pagecache, in which case I'd guess that it basically is coherent (modulo memory-ordering). Anyway, it still all happens before _exit(2). – Peter Cordes Jan 28 '18 at 10:55
2

As a general rule, any data owned by the kernel is maintained & cleaned up by the kernel, period. Such data includes data transferred to kernel memory by a system call such as write(2).

However, if your application (e.g. C library) performs buffering on top of this, then the kernel obviously has no idea and hence does not guarantee its clean-up.

Moreover, I don't believe there is any timing guarantee for the clean-up—it is, in general, performed on a "best-effort" (read: "when I have a sec") basis.

3
  • There is a guarantee that any cleanup / buffer-flushing will happen before a parent process's waitpid() returns, if the cleanup happens at all. i.e. other processes can't directly observe process termination to happen before any file modifications done by that process. (I said "directly" to rule out indirect observation through NFS file timestamps, because NFS caching isn't perfectly coherent between hosts.) – Peter Cordes Jan 28 '18 at 10:50
  • @PeterCordes: I suppose it depends what you mean by "clean-up" as opposed to "maintain". To me "maintain" is "provide a coherent view" (which does have the guarantee you mentioned) and "clean up" is "flush to disk" which I don't believe has a timing guarantee. – user541686 Jan 28 '18 at 11:24
  • Oh I see, you're answering the "flushed to disk" part of the question which is irrelevant to what later processes will see when reading the file. "clean up" in the sense of "make dirty i/o cache/buffer memory clean". Right, no timing guarantee unless you use fsync / fdatasync, although buffer write-back on Linux will start after /proc/sys/vm/dirty_writeback_centisecs hundredths of a second (if not delayed by other I/O traffic), and various other tunables in that procfs directory also affect things (e.g. how large to let buffers grow before doing any write-back). – Peter Cordes Jan 28 '18 at 11:56
2

Or is there still a very small window between the command exits and data written to the file?

No, there isn't.

I'd like to read the file right after the command exits, but I do not want to read an empty file.

You can read the final contents of the file right after the command exits, you will never be reading the empty file instead. (In C and C++, use the wait, waitpid, wait3 or wait4 system calls to wait for the program to exit, and only then read the file. If you are using a shell, another programming language or a library (e.g. the C library call system or the Java Process class), it probably uses one of these system calls already.)

As other answers and comments have pointed out, you may end up reading an empty file after the exit of the program if the program has exited without flushing its internal output buffers (e.g. because of _exit, abort or receiving a fatal signal, or because it's a Java program exiting normally). However there is nothing you can do about this at this point: the unflushed data is lost forever, additional waiting won't recover it.

0

Yes

Sorry for maybe adding another superfluous answer, but most seem to focus on the red herring of the title of the question. But as far as I can tell, the question is not about buffering at all, but this:

When I redirect a command's output to a file (e.g., echo Hello > file) will that file be guaranteed to have such data right after the command exits?

Yes, unconditionally. The usage of ">" that you are describing, along with "|" and "<", is the pipe-based processing model that the Unix and Linux world is heavily based on. You will find hundreds, if not thousands of scripts totally depending on this behaviour in every Linux installation.

It works as you want per design, and if there was even the slightest chance of a race condition, it would have been fixed probably decades ago.

1
  • This is superfluous, unfortunately. Only a couple of the answers mostly focus on the red-herring of committing data to non-volatile storage. See @pts's answer and several others for a clear description: file modification happens before exit, or not at all. – Peter Cordes Jan 28 '18 at 10:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.