Compressing a file in place - does "gzip -c file | dd of=file" really work?

Question

In the question How can I compress a file on Linux in-place, without using additional disk space?, one answer proposes to simply use

gzip -c file | dd of=file

I tried it (on Debian Linux), and it does appear to work. I don't quite understand why, however.

Doesn't dd truncate its output file before writing? Wouldn't this "pull out the rug" under gzip, thus taking away the data that gzip wants to read?

Or is there some race condition involved, meaning the command will usually work, but may sometimes fail? Or does it somehow depend on the block sizes which the commands use for I/O?

I know that a process that has opened a file can continue reading it even if another process deletes it (the file will be discarded once the process closes it). Is there a similar mechanism if a file is truncated while some process has it open?

Answer 1

Experiment shows that this does not work.

I created a 2-megabyte file from /dev/urandom, then tried the above command on it. Here are the results:

% ls -l
total 41008
-rw-r--r-- 1 kst kst 20971520 2012-01-18 03:47 file
-rw-r--r-- 1 kst kst 20971520 2012-01-18 02:48 orig
% gzip -c file | dd of=file
0+1 records in
0+1 records out
25 bytes (25 B) copied, 0.000118005 s, 212 kB/s
% ls -l
total 20508
-rw-r--r-- 1 kst kst       25 2012-01-18 03:47 file
-rw-r--r-- 1 kst kst 20971520 2012-01-18 02:48 orig
$ 

Obviously a 2-megabyte random file won't compress to 25 bytes, and in fact running gunzip on the compressed file yields an empty file.

I got similar results for a much smaller random file (100 bytes).

So what happened?

In this case, the dd command truncated file to zero bytes before starting to write to it; gzip started reading from the newly empty file and produced 25 bytes of output, which dd then appended to the empty file. (An empty file "compresses" to a non-zero size; it's theoretically impossible for any compressor to make all input smaller).

Other results may be possible, depending on the timing of the gzip, dd, and shell processes, all of which are running in parallel.

There's a race condition because one process, gzip, reads from file, while another parallel process, the shell, writes to it.

It should be possible to implement an in-place file compressor that reads and writes to the same file, using whatever internal buffering is necessary to avoid clobbering data. But I've never heard of anyone actually implementing that, probably because it usually isn't necessary and because if the compressor fails partway through, the file will be permanently corrupted.